DEV Community: Lucas F. da Costa

Breaking Terraform files into composable layers

Lucas F. da Costa — Thu, 14 Sep 2023 23:20:10 +0000

Terraform allows you to spin up cloud infrastructure using a single command. Let's say you're trying to run Elasticsearch and Kibana within a Kubernetes cluster, for example.

For that, you could write a few .tf files and run terraform apply to provision a Kubernetes cluster and deploy a few pods to it.

Now, assume you want other instances of the Elastic stack that you can use for demos. In that case, you'll have to set up brand new Terraform workspaces and run terraform apply multiple times.

The problem with this approach is that it will cause you to replicate your entire infrastructure every time. Consequently, you'll have multiple Kubernetes clusters. Each cluster takes at least 15 minutes to spin up and costs $72 a month on AWS.

A much better alternative would be to reuse a single Kubernetes cluster and spin up multiple environments on top of it. Thus, you'd pay for a single cluster, and you don't have to wait for a brand new cluster every time you spawn a new environment.

That's why Layerform exists. Layerform allows engineers to break their Terraform files into composable layers. That way, teams can have a shared base layer for their Kubernetes cluster and multiple top layers with Elasticsearch, Kibana, and even other serverless components they need, like Lambdas, SQS queues, or load balancers.

In this post, I'll explain how layers work, how to break your Terraform files into composable layers and demonstrate a few use cases that layers enable. Those use cases include creating production-like development environments and setting up pull request preview links for any type of application.

As a note, Terraform workspaces are quite laborious to use. The official documentation even contains a warning about using them.
Important: Workspaces are not appropriate for system decomposition or deployments requiring separate credentials and access controls.

How layers work

In Layerform, Terraform files are divided into layers. Each layer definition specifies the files that belong to it and a list of dependencies.

Let's say you're trying to run Elasticsearch and Kibana within an EKS instance (AWS-managed k8s).

For that, you'd have two layers, one for eks, and another for the elastic_stack.

{
    "layers": [
        {
            "name": "eks",
            "files": ["layers/eks.tf", "layers/eks/**"]
        },
        {
            "name": "elastic_stack",
            "files": ["layers/elastic_stack.tf", "layers/elastic_stack/**"],
            "dependencies": ["eks"]
        }
    ]
}

In the eks layer definition, you specify which files belong to that layer. Given that eks does not depend on any other infrastructure, it won't list any other layers as dependencies. On the other hand, the elastic_stack layer's files depend on eks. Consequently, it will include eks as a dependency.

After defining layers this way, you can spin up an EKS cluster independently by running layerform spawn eks default.

Then, each engineer can create their own instances of the elastic_stack and reuse the same cluster. For that, they'd all run layerform spawn elastic_stack <name>, and each of these layers would look for an underlying eks layer with ID default.

➜ layerform list instances
INSTANCE NAME                LAYER NAME    DEPENDENCIES
default                      eks
first                        elastic_stack  eks=default
second                       elastic_stack  eks=default

If someone needs a new cluster, they can recreate the whole stack by passing the --base flag, as in layerform spawn elastic_stack third --base eks=another_eks.

➜ layerform list instances
INSTANCE NAME                LAYER NAME    DEPENDENCIES
default                      eks
another_eks                  eks
first                        elastic_stack  eks=default
second                       elastic_stack  eks=default
third                        elastic_stack  eks=another_eks

When done with their layers, engineers can kill their layer instances without damaging someone else's work.

In a way, Layerform's layers are similar to container layers. Some people refer to them as "Kustomize for Terraform."

Using layers to create development environments

For most teams, "staging" is a bottleneck. Usually, companies have many engineers, but there's only a single "staging" environment for everyone to use. Consequently, developers wanting to test changes in a production-like environment must queue.

Additionally, when systems are large or depend on serverless components, engineers cannot reliably run all their software on their machines. Therefore, they all point to the same staging environment and step on each others' toes.

Layers help teams solve this problem by enabling each engineer to spin up their own environment and share core pieces of infrastructure, as shown earlier.

Layerform also provides developers valuable features to develop applications locally while pointing them to their own "staging" environment's back-end.

Assume you cannot run Elasticsearch locally because the JVM needs too much memory, for example. In that case, you could use layerform output to get the address of your remote Elasticsearch instance.

➜ layerform output elastic_stack first

{
  "cluster_name": {
    "sensitive": false,
    "type": "string",
    "value": "demo-eks-post-example"
  },
  "elasticsearch_url": {
    "sensitive": false,
    "type": "string",
    "value": "https://elasticsearch-post-example.environment.ergomake.link"
  }
}

You could use that URL in your Kibana configuration file (config.yml). That way, you could point your local Kibana to the remote Elasticsearch as you develop.

elasticsearch.hosts:
    ["https://elasticsearch-post-example.environment.ergomake.link"]

In case you had multiple outputs you need to interpolate in your configuration file, as many applications have, you could even write a template file and tell Layerform to interpolate it for you.

Take the following template file for Kibana's config.yml, for example.

elasticsearch.hosts: ["{{elasticsearch_url.value}}"]

To render that file and replace {{ elasticsearch_url.value }} with the actual values for your particular layer, you could use the --template flag to output the rendered file to stdout.

➜ layerform output elastic_stack first --template config/kibana-template.yml

elasticsearch.hosts: ['http://elastic-stack-xehl-es.environment.ergomake.link']

Engineers could then pipe the rendered file to an actual config.yml and use it without knowing anything about layers or manually entering values.

➜ layerform output elastic_stack first --template config/kibana-template.yml > config.yml

After that, yarn start would cause Kibana to read config.yml and point to the remote Elasticsearch.

A note on collaborating with layers

Similarly to Terraform, Layerform may use a remote back-end to store states and layer definitions.

Once an engineer creates a layer instance, Layerform syncs it with a remote state file in S3. Then, when someone else uses layerform list instances, Layerform will fetch states from the remote back-end and list an up-to-date list of instances.

Assume engineer A spun up layer first. If engineer B wants to collaborate with engineer A, they will be able to see first in the list of layers because the list is stored remotely.

# Ran by engineer B
➜ layerform list instances

INSTANCE NAME                LAYER NAME     DEPENDENCIES
default                      eks
first                        elastic_stack  eks=default

Then, engineer B can also get the URL of the Elasticsearch instance in layer first by running layerform output.

We're still working on proper locking and collaboration mechanisms to keep states free of race conditions. We welcome pull requests for that in case you're interested in contributing.

Using layers to create pull request previews

Whenever engineers make changes and open a pull request, Vercel spins up their Next.js application and adds a preview link to the PR.

Most developers, designers, and product managers love Vercel's previews. The problem is that these previews do not work for teams with more complex set-ups. For example, you can't get full-stack previews if you have a serverless application or need a Kubernetes cluster.

Today, most teams end up building these preview environments in-house, using a bunch of ad-hoc scripts.

Layerform helps teams solve this problem by allowing them to create a separate preview layer. Then, engineers can call the Layerform CLI from their CI and spin up only the infrastructure they need.

Since Layerform uses Terraform files, it can spin up all types of infrastructure for these previews, including Kubernetes resources, Lambdas, SQS instances, and S3 buckets.

In the case of the Elastic stack, for example, engineers could create a GitHub action to layerform spawn only the Elastic stack layer on top of a shared preview cluster, making previews much less expensive.

name: Preview

on:
    pull_request:

jobs:
    preview:
        runs-on: ubuntu-latest

        steps:
            # + Checkout code, install deps, etc...
            - name: Install Layerform
              run: go install github.com/ergomake/layerform@main

            # Configure remote state — we'll improve this
            - name: Create Layerform config file
              run: |
                  mkdir -p ~/.layerform
                  echo "currentContext: demo" > ~/.layerform/config
                  echo "contexts:" >> ~/.layerform/config
                  echo "  demo:" >> ~/.layerform/config
                  echo "    type: s3" >> ~/.layerform/config
                  echo "    bucket: layerform-post-demo" >> ~/.layerform/config
                  echo "    region: us-west-1" >> ~/.layerform/config

            # Actually spawn the preview layer
            - id: layerform
              name: Spawn Elasticstack using Layerform
              run: |
                  layerform spawn elastic_stack ${{ github.event.pull_request.head.ref }}
              env:
                  AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
                  AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}

The workflow above will use the pull request's branch name as the layer instance's name.

Just like any other layer instance, it will appear when running layerform list, so you can connect to it from your terminal too, if you want.

Finally, once the spawn is done, you can write some code to call layerform output and add a comment with a preview link to the pull request.

# ...

# Get Kibana's URL from layerform output using jq
- id: layerform
  name: Spawn Elasticstack using Layerform
  run: |
      layerform spawn elastic_stack ${{ github.event.pull_request.head.ref }}
      kibana=$(layerform output elastic_stack ${{ github.event.pull_request.head.ref }} | jq -r .kibana_url.value)
      echo "kibana=$kibana" >> "$GITHUB_OUTPUT"
  env:
      AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
      AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}

# Add a comment to the pull request
- uses: actions/github-script@v6
  with:
      script: |
          github.rest.issues.createComment({
            issue_number: context.issue.number,
            owner: context.repo.owner,
            repo: context.repo.repo,
            body: '${{ steps.layerform.outputs.kibana }}'
          })

After that change, you'll see a preview link pointing to actual production-like infrastructure in every PR you open.

To destroy these layer instances, you should create a second workflow that's triggered when someone closes or merges a PR. That workflow should call layerform kill and pass it the layer's name, which, in this case, is the same as the branch's.

name: Destroy previews when PRs are closed

on:
    pull_request:
        types: [closed]

jobs:
    kill_preview:
        runs-on: ubuntu-latest
        steps:
            # + Checkout code, install deps, etc...
            - name: Install Layerform
              run: go install github.com/ergomake/layerform@main

            - name: Kill preview
              run: layerform kill elastic_stack ${{ github.event.pull_request.head.ref }}

Extra implementation notes

In these examples, I assume that users have deployed an nginx-ingress-controller to their cluster through the eks layer. This controller is responsible for creating an nlb and exposing Elasticsearch and Kibana to the internet through their ingresses.

In any case, many developers would probably want to keep their development environments within a VPN, which is also possible because Layerform provisions all the infrastructure in your cloud using plain Terraform files.

When it comes to deploying the actual pods to your cluster, you can use whatever you want. When I tested the examples myself, I specified raw Kubernetes resources through the kubernetes provider. Still, you could use Helm charts with the Helm provider. The only requirement for using layers is that your infra is in a Terraform file, not ad-hoc scripts.

Finally, another interesting detail is that engineers must provide layer definitions to the remote back-end with layerform configure before anyone can spawn instances.

If you've found this idea intriguing, please consider giving us a star on GitHub.

One staging for each engineer: introducing Layerform

Lucas F. da Costa — Tue, 29 Aug 2023 21:10:27 +0000

Layerform allows developers to spin up multiple instances of their applications while reusing core parts of their infrastructure.

That way, companies can give each engineer their own staging environment. Consequently, engineers can avoid bugs from getting into production and more easily show their work to others. If this sounds useful, consider giving us a star on GitHub.

Assume you're running multiple pods in a Kubernetes cluster, for example. In that case, you can use Layerform so that each developer has their own pods and namespaces on top of a shared Kubernetes cluster.

In this post, I'll explain how Layerform allows engineers to do that by manipulating Terraform state.

First, I'll explain how Terraform states work and the problems with how they're currently implemented. Then, I'll explain how you could manipulate states to create resource replicas and reuse infrastructure. Finally, I'll show how Layerform itself uses similar state management techniques to enable development platforms that allow for resources to be reused.

What is a Terraform state?

Every Terraform setup has two parts: .tf files describing the ideal state and .tfstate files describing the current state.

Whenever you call terraform apply, it will compare the desired state described in your .tf files with the current state in your .tfstate files. Then, it will apply the differences.

Assume you want to create an S3 bucket in your AWS account. For that, you'll declare the desired state in your main.tf file.

# main.tf

provider "aws" {
  region = "us-east-1"
}

resource "aws_s3_bucket" "my_bucket" {
  bucket = "my-unique-bucket-name-layerform"

  tags = {
    Name        = "foo"
  }
}

Then, when running terraform apply, Terraform will invoke the AWS provider, which uses the AWS CLI to talk to AWS and create your bucket.

After creating your bucket, Terraform will store information about the new bucket in a terraform.tfstate file. That file contains the actual state of your infrastructure.

{
  "version": 4,
  "terraform_version": "1.5.5",
  "serial": 1,
  "lineage": "ca8081999999999999999999999999999999",
  "outputs": {},
  "resources": [
    {
      "mode": "managed",
      "type": "aws_s3_bucket",
      "name": "my_bucket",
      "provider": "provider[\"registry.terraform.io/hashicorp/aws\"]",
      "instances": [
        {
          "schema_version": 0,
          "attributes": {
            "arn": "arn:aws:s3:::my-unique-bucket-name-layerform",
            "bucket": "my-unique-bucket-name-layerform",
            "tags": {
              "Name": "foo"
            },
            "tags_all": {
              "Name": "foo"
            },
            // ...
        }
      ]
    }
  ],
  "check_results": null
}

The next time you change your main.tf file, Terraform will compare the desired state in main.tf with the actual state in terraform.tfstate to determine which changes it must apply.

Let's say you change your bucket's tags and run terraform apply, for example.

# main.tf

# ...

resource "aws_s3_bucket" "my_bucket" {
  bucket = "my-unique-bucket-name-layerform"

  tags = {
    Name        = "Look I changed something"
  }
}

When you do that, Terraform will compare the new desired state with the actual state in terraform.tfstate. Then, it will see that the tags and tags_all properties are different and tell you it plans to update these fields.

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  ~ update in-place

Terraform will perform the following actions:

  # aws_s3_bucket.my_bucket will be updated in-place
  ~ resource "aws_s3_bucket" "my_bucket" {
        id                          = "my-unique-bucket-name-layerform"
      ~ tags                        = {
          ~ "Name" = "foo" -> "Look I changed something"
        }
      ~ tags_all                    = {
          ~ "Name" = "foo" -> "Look I changed something"
        }
        # (10 unchanged attributes hidden)

        # (3 unchanged blocks hidden)
    }

After confirming the apply, the terraform.tfstate file will now contain the updated tags.

{
  "version": 4,
  "terraform_version": "1.5.5",
  // ...
  "resources": [
    {
      "mode": "managed",
      "type": "aws_s3_bucket",
      "name": "my_bucket",
      "provider": "provider[\"registry.terraform.io/hashicorp/aws\"]",
      "instances": [
        {
          "schema_version": 0,
          "attributes": {
            // ...
            "tags": {
              "Name": "Look I changed something"
            },
            "tags_all": {
              "Name": "Look I changed something"
            },
            // ...
        }
      ]
    }
  ],
  "check_results": null
}

Note that .tfstate files are the only way for Terraform to know which resources exist. If you delete your .tfstate file and run terraform apply, Terraform will try to create your bucket again. If you proceed with the apply, it will throw an error because the bucket already exists.

Error: creating Amazon S3 (Simple Storage) Bucket (my-unique-bucket-name-layerform): bucket already exists

   with aws_s3_bucket.my_bucket,
   on main.tf line 7, in resource "aws_s3_bucket" "my_bucket":
    7: resource "aws_s3_bucket" "my_bucket" {

Manipulating state to create multiple buckets

Let's say a developer called Aaron wants to have his own bucket for testing purposes. For that, Aaron needs to duplicate the aws_s3_bucket resource and run terraform apply again.

# main.tf

# ...

resource "aws_s3_bucket" "my_bucket" {
  bucket = "my-unique-bucket-name-layerform"
}

resource "aws_s3_bucket" "aarons_bucket" {
  bucket = "aarons-unique-bucket-layerform"
}

The problem with this approach is that developers must duplicate resources to get their own instances of resources.

This problem compounds as you add resources that depend on one another.

For example, if these S3 buckets must contain an object, Aaron needs to duplicate the aws_s3_bucket and the aws_s3_object within.

# main.tf

resource "aws_s3_bucket" "my_bucket" {
  bucket = "my-unique-bucket-name-layerform"
}

resource "aws_s3_object" "my_configs" {
  bucket  = "my-unique-bucket-name-layerform"
  key     = "configs/.keep"
  content = "some configs"
}

resource "aws_s3_bucket" "my_bucket" {
  bucket = "aarons-unique-bucket-layerform"
}

resource "aws_s3_object" "my_configs" {
  bucket  = "aarons-unique-bucket-layerform"
  key     = "configs/.keep"
  content = "some configs"
}

As you can see, duplication quickly becomes unmanageable and error-prone.

Even if you use modules, there's only so much you can do until you're duplicating the module declarations themselves.

You could also try using the count and for_each meta-arguments, but there's only so much you can do until complexity hits you in the head. And let's be honest, if something is called a "meta-argument", it's probably not a good idea.

To avoid duplication, Aaron could tell Terraform to use his own empty .tfstate file when running terraform apply.

$ terraform apply -state="aaron.tfstate"

The problem with just using a new .tfstate file is that it will cause Terraform to throw an error because there's already a bucket whose name is my-unique-bucket-name-layerform.

To avoid conflicts, Aaron should create a random_id resource and use its random ID in the bucket's name.

# main.tf

# ...

resource "random_string" "prefix" {
  length  = 8
  upper   = false
  special = false
}

resource "aws_s3_bucket" "my_bucket" {
  bucket = "${random_string.prefix.id}-layerform-example"
}

Now, Aaron and his team can create as many buckets as they want by specifying a different .tfstate file every time. That way, everyone on the team can have their own buckets for testing and development purposes.

$ terraform apply -state=aaron.tfstate
$ terraform apply -state=dalton.tfstate
$ terraform apply -state=pete.tfstate
$ terraform apply -state=nicolas.tfstate

This approach will work because each .tfstate file will be empty, causing Terraform to generate a new ID every time. Consequently, each bucket will have a different name, and there won't be conflicts.

Besides the practical advantages of not having to modify .tf files every time someone needs a new bucket, this approach also completely detaches the concept of a desired state with the concept of an actual state.

In Terraform's design, a declaration of the desired infrastructure is tied to a single actual state. That's a bad abstraction. In the real world, various instances of the desired infrastructure may exist. Consequently, having various actual states should be possible, too — one for each infrastructure instance.

Baking reusability into state files

Buckets are cheap. Consequently, it's okay for each person on Aaron's team to have their buckets.

Now, let's imagine Aaron's system ran on top of a Kubernetes cluster. At the time of this writing, a managed Kubernetes cluster on AWS costs $0.10, which amounts to $72 a month.

If Aaron, Dalton, Pete, and Nicolas all use brand new EKS clusters for their testing and development environments, that will cost their company $288 a month, and all clusters will be subutilized, especially if they're provisioning large EC2 instances.

To save money, Aaron could share the Kubernetes cluster with his team and allow each colleague to deploy their testing pods to a new namespace. That way, they'd have a single cluster and pay only $72. Additionally, all pods could share the same EC2 instance, making it even cheaper to run these development environments.

For that, Aaron's first step would be to create an eks.tf file and declare an EKS cluster (Amazon's managed Kubernetes offering).

# eks.tf

module "eks" "shared_cluster" {
  source  = "terraform-aws-modules/eks/aws"
  cluster_name    = "dev-eks-base-layerform"

  # I have intentionally skipped the VPC and node groups
  # configurations to keep these examples short and sweet.
}

When Aaron applies this file using terraform apply -state=base.tfstate, Terraform will generate a base.tfstate file containing the cluster's data.

After that, Aaron will create a pods.tf file, which declares a namespace and pods within that namespace. To avoid conflicts, Aaron will use random_id to generate the namespace's name.

# pods.tf

resource "random_string" "namespace_suffix" {
  length  = 8
  special = false
}

resource "kubernetes_namespace" "user_ns" {
  metadata {
    name = "user-ns-${namespace_suffix}"
  }
}

resource "kubernetes_pod" "service_a" {
  metadata {
    name      = "service-a"
    namespace = kubernetes_namespace.user_ns.metadata[0].name
  }

  # ...
}

resource "kubernetes_pod" "service_b" {
  metadata {
    name      = "service-b"
    namespace = kubernetes_namespace.user_ns.metadata[0].name
  }

  # ...
}

Then, Aaron can take a snapshot of the old base.tfstate by copying it to base-snapshot.tfstate. Then, he can create his development pods using terraform apply -state=base.tfstate.

That way, Terraform will create his pods on top of the existing Kubernetes cluster without modifying base-snapshot.tfstate.

Now, let's say Dalton also needs his own development pods but doesn't want to spend another $72 (and an extra 20 minutes) to create a new cluster for himself.

For that, Dalton could ask Aaron for the base-snapshot.tfstate file. Then, he'd run terraform apply -state=base-snapshot.tfstate to create his own set of pods within a new namespace in the same cluster.

Pete and Nicolas could do the same. All they need to create their development pods in the same cluster is to ask Aaron for the base-snapshot.tfstate and use it when running terraform apply.

The problem with sharing state snapshots

The problem with sharing state snapshots is that they must be kept up-to-date across all machines. What happens if Nicolas wants to change the cluster's name and write it in French, for example?

In that case, he'd have to apply the change and send the new .tfstate file to Aaron, Dalton, and Pete to download. Otherwise, their terraform apply runs would start failing.

Also, managing these files would become increasingly complicated if they share a kafka instance within the cluster. In that case, everyone in Aaron's team would have to have a base-snapshot.tfstate and a kafka-snapshot.tfstate that they need to merge into a single file before running terraform apply.

Layerform solves that problem by encapsulating your Terraform files into what we call layer definitions.

In Aaron's case, if he were to use Layerform, he'd create one layer definition referencing his eks.tf file and another referencing pods.tf. These would be the eks and the pods layers, respectively.

For the pods layer definition, Aaron would say it depends on an instance of the eks layer.

Then, when running layerform spawn eks, Layerform would run the eks.tf file and associate the resulting .tfstate to the ID default.

After that, if Pete runs layerform spawn pods, Layerform will pull the state for the default instance of eks. Then, it will merge the eks.tf and pods.tf files and apply them using that base state.

Dalton and Nicolas could do the same. When they run layerform spawn pods, Layerform will look for the default state for eks and spin their layer instances on top of it.

Storing state snapshots

Similarly to Terraform, Layerform can use a cloud back-end to store the states for the different instances and associate them with their IDs.

The difference between Layerform and Terraform is that Layerform can pick and merge the different states based on their IDs.

Other things you can do with layers

By breaking down your infrastructure into layers, you can have each team care for a particular part of the infrastructure.

For example, the platform team could put together the configurations for a Kubernetes cluster and Kafka instance.

The other teams could then build their layers on top of the platform team's. That way, each team takes care of its own infra and may spin up multiple instances of it without relying on numerous "matching strings" coming from Terraform's data sources.

Besides encapsulation, layers are also helpful for cost control and chargebacks. By breaking down your infrastructure into layers, Layerform can automatically tag resources in each layer instance so you know which teams' layers spend the most.

Summarising layers

At the end of the day, we can summarise Layerform as an intelligent way to break down desired states and dissociate them from actual states. That way, multiple actual states can exist for a particular desired state definition.

In a way, layers and their instances are like classes and objects. The terraform files for each layer are a "class," and each instance is an "object" with that class' properties.

Questions?

If you've got questions, feel free to use this repository's issue tracker or book a few minutes with me.

If this post has been helpful, please consider starring the repo or sharing it on HackerNews and Reddit.