DEV Community: Chris ter Beke

Terraform with YAML: Part 2

Chris ter Beke — Wed, 26 Jul 2023 00:00:00 +0000

This post is the second in a series of three about supercharging your Terraform setup using YAML.

In part one of this series we learned how to use YAML to simplify the configuration of Terraform resources. We mainly focussed on reducing syntax overhead of the HCL language and making the configuration accessible to non-infra engineers.

In this second part we will dive into some more advanced techniques and patterns.

Dynamic blocks

A powerful feature of Terraform is dynamic blocks. They allow you to specify multiple nested blocks by looping over a set or map.

In the following example we add a lifecycle rule to a storage bucket that automatically deletes objects after 3 days. We also add a lifecycle rule to automatically abort an incomplete upload after 1 day.

resource "google_storage_bucket" "bucket" {
  name = "my-awesome-bucket"
  location = "EU"
  force_destroy = false

  lifecycle_rule {
    action {
      type = "Delete"
    }
    condition {
      age = 3
    }
  }

  lifecycle_rule {
    action {
      type = "AbortIncompleteMultipartUpload"
    }
    condition {
      age = 1
    }
  }
}

You can imagine that if we add even more lifecyle rules, the syntax of this resources becomes long and tedious to read. Luckily we have dynamic blocks to relief some of our pain.

In the following example we use a dynamic block with a local map to apply the same lifecycle rules:

locals {
    lifecycle_rules = {
        "Delete" = 3
        "AbortIncompleteMultipartUpload" = 1
    }
}

resource "google_storage_bucket" "bucket" {
  name = "my-awesome-bucket"
  location = "EU"
  force_destroy = false

  dynamic "lifecycle_rule" {
    for_each = local.lifecycle_rules

    content {
        action = {
            type = lifecycle_rule.key
        }
        condition {
            age = lifecycle_rule.value
        }
    }
  }
}

As you can see, the amount of boilerplate code is already significantly reduced. Now let’s apply our YAML magic to it and see what happens.

bucket:
  name: example-bucket-123
  location: EU
  force_destroy: true
  lifecycle_rules:
    Delete: 3
    AbortIncompleteMultipartUpload: 1


locals {
  config = yamldecode(file("config.yaml"))
}

resource "google_storage_bucket" "bucket" {
  name = config.bucket.name
  location = config.bucket.location
  force_destroy = config.bucket.force_destroy

  dynamic "lifecycle_rule" {
    for_each = config.bucket.lifecycle_rules

    content {
      action = {
        type = lifecycle_rule.key
      }
      condition {
        age = lifecycle_rule.value
      }
    }
  }
}

By specifying the actual rules in our YAML config file, it became very clear which rules we are enforcing on our bucket.

Multiple resource types

Now let’s see how we can define more than a single resource based on a YAML configuration file. Here is an example of this for storage bucket IAM members:

bucket:
  name: example-bucket-123
  location: EU
  force_destroy: true
  admins:
    - "group:storage-admins@company.com"
    - "user:john-break-glass@company.com"


locals {
  config = yamldecode(file("config.yaml"))
}

resource "google_storage_bucket" "bucket" {
  name = config.bucket.name
  location = config.bucket.location
  force_destroy = config.bucket.force_destroy
}

resource "google_storage_bucket_iam_member" "admins" {
  for_each = toset(config.bucket.admins)

  bucket = google_storage_bucket.bucket.name
  role = "roles/storage.admin"
  member = each.key
}

One pattern used here is to group configuration together in YAML and spread it out over multiple Terraform resources. This reduces the amount of locations in the code you need to touch in order to change your infrastructure.

Up next

Now we know the basics of YAML in Terraform, as well as some more advanced situation that it can be useful in. In the next and final part of this series, we will dive into templating and schema validation. We’ll also have a quick look at how to automate the injection of YAML config files using Terragrunt.

The post Terraform with YAML: Part 2 appeared first on Xebia.

Guarantee unique keys in Terraform

Chris ter Beke — Wed, 26 Jul 2023 00:00:00 +0000

When using Terraform to dynamically create resources based on lists of maps you probaby have run into this issue. Consider the following list of maps that represents IAM access on a generic cloud resource:

locals = {
    members = [
        {
            member = "contact@christerbeke.com"
            resource = "projects/12345"
            role = "roles/owner"
        },
        {
            member = "test@christerbeke.com"
            resource = "projects/12345"
            role = "roles/reader"
        }
    ]
}

If we want to iterate over this list to create a dynamic amount of resources (using for_each) we need to convert it to a map. However there is no way you can construct an map key from any of the separate attributes and guarantee uniqueness. So how can we solve this?

The trick is to create a combination of all the attributes. But simply concatenating all the attributes in a string results in very long keys. To solve this, and get predictable key lengths, we can use an md5 hash:

locals = {
    unique_members = { for key, member in local.members : md5("${member.member}/${member.resource}/${member.role}") => member }
}

This results in the following data:

{
    "0f334c4500b1faab57203343199d5c86" = {
        member = "contact@christerbeke.com"
        resource = "projects/12345"
        role = "roles/owner"
    },
    "c02f629ef8bf2b413a203c4dcafa60c1" = {
        member = "test@christerbeke.com"
        resource = "projects/12345"
        role = "roles/reader"
    }
}

Now we can use it in our for_each iterator:

resource "iam_member" "default" {
    for_each = local.unique_members

    member = each.value.member
    resource = each.value.resource
    role = each.value.role
}

Now you know a simple trick to convert a list of maps into an iterable map with unique keys.

As a bonus you now get alerted about duplicate list entries, as they would result in dulicate map keys causing Terraform to throw an error!

The post Guarantee unique keys in Terraform appeared first on Xebia.

Terraform with YAML: Part 1

Chris ter Beke — Tue, 04 Apr 2023 00:00:00 +0000

This post is the first in a series of three about supercharging your Terraform setup using YAML.

Terraform is one of the most common tools to provision infrastructure from code or configuration. However it’s using a custom language called HCL (Hashicorp Configuration Language). In this blog post we will explore how we can replace as much HCL code as possible with YAML and what the benefits are of doing so.

Why YAML?

One of the best properties of YAML in my opinion is the absence of syntax overhead. It allows you to consicely write down parameters and values. Let’s look at a comparison of some HCL code and YAML where we configure some Google Pub/Sub topics and subscriptions:

locals {
  config = {
    topics = [
      {
        name = "my-topic"
        labels = {
          environment = "prod"
        }
        subscriptions = [
          {
            name = "my-subscription"
            push_endpoint = "https://example.com/push"
          }
        ]
      }
    ]
  }
}


topics:
  - name: my-topic
    labels:
        environment: prod
    subscriptions:
      - name: my-subription

As you can see, the difference in number of lines is quite large. Of course this will change once we add some HCL code to import the YAML configuration, but it quickly adds up when your infrastructure grows.

Loading and converting the YAML file to HCL is very easy. You can do it in one line even using the yamldecode and file functions:

locals {
  config = yamldecode(file("config.yaml"))
}

The result is an HCL represenation of the same data as shown in the earlier example.

For this particular example, the total number of lines of code using plain HCL is 18, of which 9 are purely syntax. The total number of lines using YAML, including the loading and parsing of the file, is 9. That’s a 50% reduction!

For more information about YAML decoding in Terraform, check the official documentation.

Another benefit of YAML over HCL is familiarity. Many engineers that do not work on infrastructure are not familiar with the HCL syntax and it’s quirks. YAML on the other hand is so simple and widely used that almost every engineer has used it in their career at some point. This means that if your repository contains YAML for infrastructure configuration, other types of engineers can easily adjust the configuration and deploy it (preferrably using a CI/CD pipeline and proper code review). This provides a self-sufficient environment for application or data teams that work on top of the base infrastructure.

A simple example

Let’s build a fully working example:

project:
  id: my-project-id
  region: europe-west4
bucket:
  name: example-bucket-123
  location: EU
  force_destroy: true


terraform {
  required_providers {
    google = {
      source = "hashicorp/google"
      version = "4.47.0"
    }
  }
}

locals {
  config = yamldecode(file("config.yaml"))
}

provider "google" {
  project = config.project.id
  region = config.project.region
}

resource "google_storage_bucket" "bucket" {
  name = config.bucket.name
  location = config.bucket.location
  force_destroy = config.bucket.force_destroy
}

In this example, we create and configure a Cloud Storage bucket. We use two separate root objects (project and bucket) to keep the config tidy and readable.

Using loops

Often we want to configure multiple resources, for example different storage buckets for different applications. Let’s adjust the example above to use a for_each loop:

project:
  id: my-project-id
  region: europe-west4
buckets:
  - name: example-bucket-123
    location: EU
    force_destroy: true
  - name: example-bucket-456
    location: US
    force_destroy: false


terraform {
  required_providers {
    google = {
      source = "hashicorp/google"
      version = "4.47.0"
    }
  }
}

locals {
  config = yamldecode(file("config.yaml"))
}

provider "google" {
  project = config.project.id
  region = config.project.region
}

resource "google_storage_bucket" "bucket" {
  for_each = toset(config.buckets)

  name = each.value.name
  location = each.value.location
  force_destroy = each.value.force_destroy
}

As you can see, with minimal extra code, we can now provision as many buckets as we want.

Up next

Now we have a basic understanding of the benefits of using YAML configuration files in your Terraform code. In the next post in this series we will dive into more advanced topics, like how to deal with nested loops, creating multiple resource types from a single YAML configuration, and dynamic variable injection and templating. As a bonus we will look into validating YAML files using a schema to get early feedback on the configuration without having to run a Terraform plan.

The post Terraform with YAML: Part 1 appeared first on Xebia.

Creating a non-classic Google Cloud Global Load Balancer with Terraform

Chris ter Beke — Fri, 16 Sep 2022 07:30:00 +0000

The Google Cloud Terraform Provider has resources to configure a Global External HTTP(S) Load Balancer. By default however this creates a classic load balancer, not a new one. For new features like traffic management you cannot use the classic load balancer, so you definitely want to use the new one.

The Google and Terraform documentation is not clear about how to do this properly. The name classic does not even appear once on the documentation pages for the relevant resources.

A typical Global Load Balancing stack looks like this:

resource "google_compute_global_address" "default" {
    ...
}

resource "google_compute_backend_service" "default" {
    ...
}

resource "google_compute_url_map" "default" {
    ...
}

resource "google_compute_target_http_proxy" "default" {
    ...
}

resource "google_compute_global_forwarding_rule" "default" {
    ...
}

In this stack, google_compute_backend_service is the load balancing back-end, and google_compute_global_forwarding_rule is the front-end.

In order to use a new load balancer, both the back-end and front-end need to have their load_balancing_scheme configured:

resource "google_compute_backend_service" "default" {
    ...
    load_balancing_scheme = "EXTERNAL_MANAGED"
}

resource "google_compute_global_forwarding_rule" "default" {
    ...
    load_balancing_scheme = "EXTERNAL_MANAGED"
}

Conclusion

Now you know how to create a non-classic Global Load Balancer in Google Cloud using Terraform. The configuration is simple, but hard to find based on the available documentation.

The post Creating a non-classic Google Cloud Global Load Balancer with Terraform appeared first on Xebia.

A declarative approach for Dataflow Flex Templates

Chris ter Beke — Thu, 28 Jul 2022 09:00:00 +0000

Google Cloud offers a managed Apache Beam solution called Dataflow. Since some time now Dataflow has a feature called Flex Templates. Flex Templates use Docker containers instead of Dataflow’s custom templates. The benefit is that Docker is a known standard and the container can run in different environments. However, a custom metadata JSON file is still needed to point to the Docker image in your registry.

Both the CLI and the Terraform approach require you to push the Docker image to a registry.

Using the gcloud CLI

To generate and upload the JSON file you can run gcloud dataflow flex-template build. The input for this command is a bit of JSON that defines the pipeline name and parameters:

{
    "name": "My Apache Beam pipeline",
    "parameters": [
        {
            "name": "output_table",
            "label": "BigQuery output table name.",
            "helpText": "Name of the BigQuery output table name.",
            "regexes": ["([^:]+:)?[^.]+[.].+"]
        }
    ]
}

Let’s see what is in Cloud Storage after running the command:

{
    "image": "eu-docker.pkg.dev/my-gcp-project-id/dataflow-templates/example:latest",
    "sdkInfo": {
        "language": "PYTHON"
    },
    "metadata": {
        "name": "My Apache Beam pipeline",
        "parameters": [
            {
                "name": "output_table",
                "label": "BigQuery output table name.",
                "helpText": "Name of the BigQuery output table name.",
                "regexes": ["([^:]+:)?[^.]+[.].+"]
            }
        ]
    }
}

The command only adds the Docker image location and some Beam SDK before uploading it to Cloud Storage.

Using Terraform

While this works fine, it goes against the declarative approach of Terraform and other infrastructure as code tools.

Let’s see what it takes to generate and manage this metadata JSON file in Terraform:

resource "google_storage_bucket_object" "flex_template_metadata" {
    bucket = "my-unique-bucket"
    name = "dataflow-templates/example/metadata.json"
    content_type = "application/json"

    content = jsonencode({
        image = "eu-docker.pkg.dev/my-gcp-project-id/dataflow-templates/example:latest"
        sdkInfo = {
            language = "PYTHON"
        }
        metadata = {
            name = "My Apache Beam pipeline"
            parameters = [
                {
                    name = "output_table"
                    label = "BigQuery output table name."
                    helpText = "Name of the BigQuery output table name.",
                    regexes = ["([^:]+:)?[^.]+[.].+"]
                }
            ]
        }
    })
}

We can reference the storage file path in our Flex Template job:

resource "google_dataflow_flex_template_job" "flex_template_job" {
    provider = google-beta

    name = "example_pipeline"
    region = "europe-west4"
    container_spec_gcs_path = "gs://${google_storage_bucket_object.flex_template_metadata.bucket}/${google_storage_bucket_object.flex_template_metadata.name}"

    parameters = {
        output_table = "my-gcp-project-id/example_dataset/example_table"
    }
}

Run terraform apply to create both the template file and the Dataflow job.

Updating the Dataflow job

We have one issue remaining. A change in the template data does not trigger an update of the Dataflow job. For this to work, we need an attribute on the Dataflow job resource to change. We can do this by including an MD5 hash of the file contents in the storage path:

locals {
    template_content = jsonencode({
        image = "eu-docker.pkg.dev/my-gcp-project-id/dataflow-templates/example:latest"
        sdkInfo = {
            language = "PYTHON"
        }
        metadata = {
            name = "My Apache Beam pipeline"
            parameters = [
                {
                    name = "output_table"
                    label = "BigQuery output table name."
                    helpText = "Name of the BigQuery output table name.",
                    regexes = ["([^:]+:)?[^.]+[.].+"]
                }
            ]
        }
    })
    template_gcs_path = "dataflow-templates/example/${base64encode(md5(local.template_content))}/metadata.json"
}

resource "google_storage_bucket_object" "flex_template_metadata" {
    bucket = "my-unique-bucket"
    name = local.template_gcs_path
    ...
}

A change in the template data will trigger a change in the MD5 hash. This will trigger a change in the template storage path that we also reference in the Dataflow job resource. Running terraform apply now correctly updates both the JSON data in storage as well as the Dataflow flex job. If you are using a batch mode it will create a new job instance.

Conclusion

Dataflow Flex Templates allow you to use Docker images Dataflow jobs. A gcloud CLI command is required to build and upload some JSON metadata. We can replicate this behavior using Terraform code. This allows for a 100% declarative infrastructure-as-code solution.

For a full code example, check my Dataflow Flex Terraform module on GitHub.

The post A declarative approach for Dataflow Flex Templates appeared first on Xebia.

A minimal setup for a high availability service using Cloud Run

Chris ter Beke — Tue, 11 Jan 2022 14:16:13 +0000

In this blog post, I will explain what is needed to set up a web service that runs in multiple GCP regions.

The main reasons to deploy your service in more than one region are:

Handle single-region failures so that your application is highly available.
Route traffic to the nearest region so your users experience faster loading times.

Create Cloud Run deployments

A Cloud Run service only lives in a single region, so for a multi-region setup we will need to deploy the same container in multiple regions.

Luckily using a Terraform for_each loop, this does not add too much additional configuration:

locals {
  locations = ["europe-west4", "europe-west1"]
}

resource "google_cloud_run_service" "service" {
  for_each = toset(local.locations)

  name = "service-${each.key}"
  location = each.key

  ...
}

I recommend to use the name of the region in the name of the Cloud Run service so you can easily find them and guarantee uniqueness.

We use local.locations to define the regions we want to deploy in so we can re-use that configuration in other resources.

Set up load balancing ingress

By default, Cloud Run gives a service a publicly available .run.app URL.

However this points to a single Cloud Run service, and for a multi-region set we will need multiple services.

To do this, we will need to create a Global Load Balancer that uses Serverless Network Endpoint Groups (NEGs) as backend.

These NEGs then route the traffic to the Cloud Run instances.

Let’s set up the needed resource for our ingress stack:

resource "google_compute_global_address" "ip" {
  name = "service-ip"
}

resource "google_compute_region_network_endpoint_group" "neg" {
  for_each = toset(local.locations)

  name = "neg-${each.key}"
  network_endpoint_type = "SERVERLESS"
  region = each.key

  cloud_run {
    service = google_cloud_run_service.service[each.key].name
  }
}

resource "google_compute_backend_service" "backend" {
  name = "backend"
  protocol = "HTTP"

  dynamic "backend" {
    for_each = toset(local.locations)

    content {
      group = google_compute_region_network_endpoint_group.neg[backend.key].id
    }
  }
}

resource "google_compute_url_map" "url_map" {
  name = "url-map"
  default_service = google_compute_backend_service.backend.id
}

resource "google_compute_target_http_proxy" "http_proxy" {
  name = "http-proxy"
  url_map = google_compute_url_map.url_map.id
}

resource "google_compute_global_forwarding_rule" "frontend" {
  name = "frontend"
  target = google_compute_target_http_proxy.http_proxy.id
  port_range = "80"
  ip_address = google_compute_global_address.ip.address
}

Notice how we are re-using local.locations to create the regional resources.

No one can call our service yet though, because we need to tell GCP that this is a public service that can be invoked by everyone:

data "google_iam_policy" "noauth" {
  binding {
    role = "roles/run.invoker"
    members = ["allUsers"]
  }
}

resource "google_cloud_run_service_iam_policy" "noauth" {
  for_each = toset(local.locations)

  service = google_cloud_run_service.service[each.key].name
  location = google_cloud_run_service.service[each.key].location
  policy_data = data.google_iam_policy.noauth.policy_data
}

Deploy and call the service

Let’s add an output for the static IP address so we know what to call after deployment:

output "static_ip" {
  value = google_compute_global_address.ip.address
}

Now run terraform apply to deploy everything and validate that it returns the “Hello World” container (for example using curl $(terraform output --raw static_ip)).

The Google Cloud Console also gives a nice visual overview of how the requests are routed:

Now you know how to deploy Google Cloud Run services in multiple regions. Give it a try with PrivateBin!

Bonus: enable Cloud CDN for even faster loading times

To prevent static assets from being served from your container, you can enable Cloud CDN to automatically serve these from Cloud Storage edge locations instead of the container itself.

Cloud CDN will automatically detect which routes are static resources, but you can manually override this configuration as well.

Simply add the enable_cnd flag to the backend service resource:

resource "google_compute_backend_service" "backend" {
  name = "backend"
  protocol = "HTTP"
  enable_cdn = true

  ...
}

Conclusion

By default, a single Cloud Run service can only be deployed in one region.

By using a global load balancer, we can deploy a Cloud Run service in multiple regions to bring high availability and low latency.

The for_each loop feature of Terraform makes this very easy to set up.

The post A minimal setup for a high availability service using Cloud Run appeared first on Xebia.