Skip to main content

2 posts tagged with "IaC"

View All Tags

· 7 min read
George Fahmy

In this tutorial you'll use devx to generate Terraform IaC with security best-practices to run workloads on AWS.

infra

Things you get out of the box:

  • secrets management with SSM parameter store or secrets manager
  • least privilege ECS task IAM policies
  • buttoned up security groups
  • TLS configuration for application loadbalancer
  • task logging with cloudwatch
  • application loadbalancer healthchecks

Sections:

Prerequisites​

  • devx is installed on your machine Use homebrew

    brew tap stakpak/stakpak
    brew install devx

    Or

    download the latest binary

  • AWS cli configured on your machine

    You have to make sure your Terraform IAM role/user have enough permissions to manage ECS clusters, ALBs, Security Groups, SSM Parameters, VPCs, Subnets, IAM Roles, IAM Policies, etc...

    If you're just trying this out, make sure you're using an empty account and give Terraform Administrator Access, but in production you have to reduce the permission to just what you need.

  • AWS Route53 is managing your domain

  • Terraform and Git are installed on your machine

Building your infrastructure (Platform Engineers)​

This project will house the definitions of common infrastructure components like your VPC, and any configurations that you want to be re-usable by your developers.

1) Create a new project​

mkdir platform
cd platform
devx project init stakpak.dev/demo-ecs

update your module name

./cue.mod/module.cue
// Add your own module name
{
    module: "stakpak.dev/demo-ecs"
    cue: lang: "v0.6.0-alpha.1"
    deps: "github.com/stakpak/devx-catalog": {
        // You can optionally pin dependencies to a specific git tag or commit
        v: "v2.0.0-alpha.117"
    }
}

2) Create a VPC, Cluster, and ALB​

You first have to create a stack

touch stack.cue
./stack.cue
package main

import (
    "stakpak.dev/devx/v1"
    "stakpak.dev/devx/v1/traits"
)

stack: v1.#Stack & {
    components: {
        // Virtual private network
        network: {
            traits.#VPC
            vpc: {
                name: "tutorial"
                cidr: "10.0.0.0/16"
                subnets: {
                    private: ["10.0.1.0/24", "10.0.2.0/24"]
                    public: ["10.0.101.0/24", "10.0.102.0/24"]
                }
            }
        }

        // ECS cluster
        cluster: {
            traits.#ECS
            ecs: {
                name: "tutorial"
                capacityProviders: fargateSpot: enabled: true
                vpc: network.vpc
            }
        }

        // Application load balancer
        gateway: {
            traits.#Gateway
            gateway: {
                name:   "tutorial"
                public: true

                // Replace this with you own domains
                // You have to move your domain name servers to Route53 for this to work
                addresses: [
                    "tutorial-aws-ecs.stakpak.dev",
                ]

                // We want our gateway to listen on port 80 and 443
                listeners: {
                    http: {
                        port:     80
                        protocol: "HTTP"
                    }
                    https: {
                        port:     443
                        protocol: "HTTPS"
                    }
                }
            }
        }
    }
}

then tell devx how to build this stack for the prod environment

touch builder.cue
./builder.cue
package main

import (
    "stakpak.dev/devx/v2alpha1"
    tf "stakpak.dev/devx/v1/transformers/terraform"
    "stakpak.dev/devx/v1/transformers/terraform/aws"
)

builders: v2alpha1.#Environments & {
    prod: {
        // Set terraform output directory to "./deploy"
        drivers: terraform: output: dir: ["deploy"]

        components: {
            // Add this component to customize the generated Terraform AWS providers
            tfprovider: {
                // This reserved field contains all the low-level resources
                $resources: {
                    // The name of the resource doesn't matter
                    // We use the tf.#Terraform schema to make sure we define valid Terraform configurations
                    terraform: tf.#Terraform & {
                        provider: aws: region: "us-west-1"
                    }
                }
            }

            // The aws.#AddGateway transformer requires vpc data as input
            // We set the vpc here because we cannot reference external components from within a transformer
            network: _
            gateway: aws: vpc: network
        }

        flows: {
            // This flow expects a VPC trait and generates Terraform code to add an AWS VPC
            "terraform/add-vpc": pipeline: [aws.#AddVPC]

            // This flow expects an ECS trait and generates Terraform code to add an AWS ECS cluster
            "terraform/add-ec-cluster": pipeline: [aws.#AddECS & {
                aws: {
                    // Add your AWS region
                    region:  "us-west-1"

                    // Add your AWS account id
                    account: "000000000000"
                }
            }]

            // This flow expects an Gateway trait and generates Terraform code to add an AWS ALB
            "terraform/add-gateway": pipeline: [aws.#AddGateway & {
                // Allow the transformer to create Route53 DNS records
                createDNS: true

                // Allow the transformer to create TLS certificates
                createTLS: true
            }]
        }
    }
}
caution

Make sure to set your AWS account id and region.

3) Generate configurations​

devx build prod

Terraform resources generated for the network: cluster

Terraform resources generated for the cluster: cluster

Terraform resources generated for the gateway: cluster

4) Apply configurations​

cd deploy
terraform init
terraform plan

if all looks good

terraform apply --auto-approve
> Apply complete! Resources: 28 added, 0 changed, 0 destroyed.

5) Environment builders​

Define the builders that your developers will use to build their configurations.

cd ..
mkdir platform
touch platform/platform.cue
./platform/platform.cue
package platform

import (
    "stakpak.dev/devx/v2alpha1"
    "stakpak.dev/devx/v2alpha1/environments"
    "stakpak.dev/devx/v1/transformers/terraform/aws"
    infra "stakpak.dev/demo-ecs:main"
)

Builders: v2alpha1.#Environments & {
    prod: environments.#ECS & {
        drivers: terraform: output: dir: ["deploy"]
        config: {
            aws: {
                // Add your AWS region
                region:  "us-west-1"

                // Add your AWS account id
                account: "000000000000"
            }
            vpc: name: infra.stack.components.network.vpc.name
            ecs: {
                name:       infra.stack.components.cluster.ecs.name
                launchType: "FARGATE"
            }
            secrets: service: "ParameterStore"
            gateway: infra.stack.components.gateway.gateway
        }

        // Add this component to configure the generated Terraform AWS providers
        // We re-use the provider definition of our infra stack
        components: tfprovider: infra.builders.prod.components.tfprovider

        // Add an extra flow to auto-generate secrets in SSM parameter store
        // The aws.#AddSSMSecretParameter transformer use random_password to generate a 32-character string
        flows: "terraform/autogenerate-secrets": pipeline: [aws.#AddSSMSecretParameter]
    }
}
caution

Make sure to set your AWS account id and region.

6) Commit and push your code​

git init
cat > .gitignore <<EOL
cue.mod/pkg
deploy/.terraform
deploy/terraform.tfstate
EOL
git add .
git commit -m "init devx"
git remote add origin https://github.com/stakpak/devxtut-aws-ecs.git
git branch -M main
git push -u origin main

Deploying apps (Developers)​

1) Create a new project​

mkdir myapp
cd myapp
devx project init stakpak.dev/demo-ecs/apps/api

update your module name and add your platform repository as a dependency

./cue.mod/module.cue
// Add your own module name
{
    module: "stakpak.dev/demo-ecs/apps/api"
    cue: lang: "v0.6.0-alpha.1"
    deps: {
        "github.com/stakpak/devx-catalog": {
            // You can optionally pin dependencies to a specific git tag or commit
            v: "v2.0.0-alpha.117"
        }
        // Your platform repository
        "github.com/stakpak/devxtut-aws-ecs": {}
    }
}

2) Define what your service needs​

We create a stack for our service

touch stack.cue
./stack.cue
package main

import (
    "stakpak.dev/devx/v1"
    "stakpak.dev/devx/v1/traits"
    "stakpak.dev/demo-ecs/platform"
)

stack: v1.#Stack & {
    // Components can be workloads, resources, CI pipeline etc...
    components: {
        // Add secrets
        commonSecrets: {
            // The Secret trait let's you add references to your secrets
            traits.#Secret
            secrets: apisecret: name: "API_SECRET"
        }

        // Define your workload
        api: {
            // The Workload trait runs some containers
            traits.#Workload
            containers: default: {
                image: "hashicorp/http-echo:alpine"
                command: ["-listen=:8000", "-text=hello"]
                resources: requests: {
                    cpu:    "256m"
                    memory: "512M"
                }
                env: {
                    API_SECRET:     commonSecrets.secrets.apisecret
                    NAME:    "Devopzilla API"
                }
            }

            // The Exposable trait exposes ports on your containers
            traits.#Exposable
            endpoints: default: ports: [
                {
                    port:   8000
                },
            ]
        }

        // Expose your workloads through a public HTTPS ingress
        route: {
            // The HTTPRoute trait exposes some endpoints through your ingress gateway
            traits.#HTTPRoute
            http: {
                listener: "https"
                hostnames: [
                    "tutorial-aws-ecs.stakpak.dev",
                ]
                rules: [{
                    match: path: "/*"
                    backends: [
                        {
                            name:       api.appName
                            endpoint:   api.endpoints.default
                            containers: api.containers
                            port:       8000
                        },
                    ]
                }]
            }
        }

        // Redirect HTTP traffic to HTTPS
        redirect: {
            traits.#HTTPRoute
            http: {
                listener: "http"
                hostnames: route.http.hostnames
                rules: [{
                    match: path: "/*"
                    redirect: {
                        scheme: "https"
                        port:   443
                    }
                }]
            }
        }
    }
}

// Use the central platform builders
builders: platform.Builders

3) Generate configurations​

devx build prod

Terraform resources generated for the stack: cluster

4) Apply configurations​

cd deploy
terraform init
terraform plan

if all looks good

terraform apply --auto-approve
> Apply complete! Resources: 10 added, 0 changed, 0 destroyed.

5) Test the workload​

curl "https://tutorial-aws-ecs.stakpak.dev"
> hello

Clean up​

Destroy the workload​

cd myapp/deploy
terraform destroy --auto-approve
> Destroy complete! Resources: 10 destroyed.

Destroy the infrastructure​

cd platform/deploy
terraform destroy --auto-approve
> Destroy complete! Resources: 28 destroyed.

Why not just use Terraform modules?​

1) Stack definitions are technology-agnostic. You don't have to know much about ECS to deploy a workload, and if you swap transformers you will be able to deploy with Docker Compose, or even Kubernetes

2) You can package and re-use other types of configurations in devx modules e.g. CI pipelines, policies, workflows, shared variables, validation rules, and application configurations

3) Nested configurations like the task definition are validated much earlier at generation

· 2 min read
George Fahmy

Install devx​

Option 1: Homebrew​

brew tap stakpak/stakpak
brew install devx

Option 2: Binary​

Init the project​

mkdir myapp
cd myapp
devx project init

Update project dependencies​

devx project update

Generate example​

devx project gen

This will generate a sample DevX stack and builder. You create a stack to define your workload and what it needs to run.

stack.cue
package main

import (
    "stakpak.dev/devx/v1"
    "stakpak.dev/devx/v1/traits"
)

stack: v1.#Stack & {
    components: {
        cowsay: {
            traits.#Workload
            containers: default: {
                image: "docker/whalesay"
                command: ["cowsay"]
                args: ["Hello DevX!"]
            }
        }
    }
}

You create a builder for the dev environment to tell DevX how to generate configurations. This step will usually be performed by platform teams or developers wishing to extend the platform.

builder.cue
package main

import (
    "stakpak.dev/devx/v2alpha1"
    "stakpak.dev/devx/v2alpha1/environments"
)

builders: v2alpha1.#Environments & {
    dev: environments.#Compose
}

Build configurations for the dev environment​

devx build dev
🏗️  Loading stack...
👀 Validating stack...
🏭 Transforming stack 100% |████████████████████████| (1/1, 711 it/s)        
[compose] applied resources to "docker-compose.yml"
docker-compose.yml
version: "3"
volumes: {}
services:
  cowsay:
    image: docker/whalesay
    environment: {}
    depends_on: []
    command:
      - cowsay
      - Hello DevX!
    restart: always
    volumes: []

No we run the compose file

docker-compose up
[+] Running 1/0
 ⠿ Container compose-cowsay-1  Created                                                                                                  0.0s
Attaching to compose-cowsay-1
compose-cowsay-1  |  _____________ 
compose-cowsay-1  | < Hello DevX! >
compose-cowsay-1  |  ------------- 
compose-cowsay-1  |     \
compose-cowsay-1  |      \
compose-cowsay-1  |       \     
compose-cowsay-1  |                     ##        .            
compose-cowsay-1  |               ## ## ##       ==            
compose-cowsay-1  |            ## ## ## ##      ===            
compose-cowsay-1  |        /""""""""""""""""___/ ===        
compose-cowsay-1  |   ~~~ {~~ ~~~~ ~~~ ~~~~ ~~ ~ /  ===- ~~~   
compose-cowsay-1  |        \______ o          __/            
compose-cowsay-1  |         \    \        __/             
compose-cowsay-1  |           \____\______/   
compose-cowsay-1 exited with code 0