DEV Community: Fábio Matavelli

Using Terraform setproduct function

Fábio Matavelli — Fri, 18 Mar 2022 17:55:48 +0000

In 2019 one of my contributions for Terraform were published in the 0.12 version, the setproduct function. It added the capability to calculate the cartesian product of multiple sets of strings.

Cartesian product

I will try not to be too theoretical in this one, but the cartesian product is a mathematical function that, given two sets, calculates the sets of all ordered pairs, AxB = {(a, b),...}, where a is in A and b is in B.
To give you an example, we can use a deck of cards, where you have a rank of a 13-element set and a suit of a four-element set.

A = {A, K, Q, J, 10, 9, 8, 7, 6, 5, 4, 3, 2}
B = {♠, ♥, ♦, ♣}

AxB = {(A, ♠), (A, ♥), (A, ♦), (A, ♣), (K, ♠), ..., (3, ♣), (2, ♠), (2, ♥), (2, ♦), (2, ♣)}

You can find more information about cartesian product on Wikipedia.

Why that?

In 2018, I was working on a project that was using AWS as a cloud provider, and Terraform was responsible to manage the infrastructure as a code (IaC) for AWS. We needed to create multiple queues in SQS with the same purpose, receive data from an ERP to be processed by a lambda consumer and integrate it with an e-commerce platform.

The main idea was to create a queue for every ERP module that needed to have the message processed by this lambda consumer. We had some modules in the ERP at the beginning of the project, but the idea was to expand it to many more. Products, SKUs, stock, customers, orders, and more, needed to be integrated with the e-commerce platform.

Since the project wasn't that big in the beginning, we were using the same AWS account for the three environments (what I don't recommend), development, staging, and production. Said that we need to create the queues for every stage:

A = {product, sku, stock, customer, order}
B = {dev, stage, prod}

AxB = {(product, dev), (product, stage), (product, prod), (sku, dev)... (order, dev), (order, stage), (order, prod)}

Fifteen queues needed to be created and managed by Terraform, and we had the idea to expand it to more modules in the future.

The setproduct function

Terraform didn't have an easy way to do that, so I had the idea to create an MR adding the setproduct function to solve this problem.
Setproduct is easy to use, and you can pass more than 2 sets to it (AxBxCxN...), and it will return all the possible combinations of the sets.

So we had the problem, and now we have the solution, how to use it?

Simple:

> setproduct(["product", "sku", "stock", "customer", "order"], ["dev", "stage" , "prod"])

[
  [
    "product",
    "dev"
  ],
  [
    "product",
    "stage"
  ],
  [
    "product",
    "prod"
  ],
  [
    "sku",
    "prod"
  ],
  ...
  [
    "order",
    "dev"
  ],
  [
    "order",
    "stage"
  ],
  [
    "order",
    "prod"
  ],
]

With that, now we can create the SQS queues:

locals {
  queues = setproduct(["product", "sku", "stock", "customer", "order"], ["dev", "stage" , "prod"])
}

resource "aws_sqs_queue" "queue" {
  for_each = {
    for q in local.queues : "${q[0]}-${q[1]}" => {
      module = q[0]
      stage = q[1]
    }
  }

  name = "${each.value.module}-${each.value.stage}"

  tags = {
    Module = each.value.module
    Stage = each.value.stage
  }
}

Running the terraform plan, we have:

Refreshing Terraform state in-memory prior to plan...
The refreshed state will be used to calculate this plan, but will not be
persisted to local or remote state storage.


------------------------------------------------------------------------

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # aws_sqs_queue.queue["customer-dev"] will be created
  + resource "aws_sqs_queue" "queue" {
      + arn                               = (known after apply)
      + content_based_deduplication       = false
      + delay_seconds                     = 0
      + fifo_queue                        = false
      + id                                = (known after apply)
      + kms_data_key_reuse_period_seconds = (known after apply)
      + max_message_size                  = 262144
      + message_retention_seconds         = 345600
      + name                              = "customer-dev"
      + policy                            = (known after apply)
      + receive_wait_time_seconds         = 0
      + tags                              = {
          + "Module" = "customer"
          + "Stage"  = "dev"
        }
      + visibility_timeout_seconds        = 30
    }

  # aws_sqs_queue.queue["customer-prod"] will be created
  + resource "aws_sqs_queue" "queue" {
      + arn                               = (known after apply)
      + content_based_deduplication       = false
      + delay_seconds                     = 0
      + fifo_queue                        = false
      + id                                = (known after apply)
      + kms_data_key_reuse_period_seconds = (known after apply)
      + max_message_size                  = 262144
      + message_retention_seconds         = 345600
      + name                              = "customer-prod"
      + policy                            = (known after apply)
      + receive_wait_time_seconds         = 0
      + tags                              = {
          + "Module" = "customer"
          + "Stage"  = "prod"
        }
      + visibility_timeout_seconds        = 30
    }

...

  # aws_sqs_queue.queue["stock-stage"] will be created
  + resource "aws_sqs_queue" "queue" {
      + arn                               = (known after apply)
      + content_based_deduplication       = false
      + delay_seconds                     = 0
      + fifo_queue                        = false
      + id                                = (known after apply)
      + kms_data_key_reuse_period_seconds = (known after apply)
      + max_message_size                  = 262144
      + message_retention_seconds         = 345600
      + name                              = "stock-stage"
      + policy                            = (known after apply)
      + receive_wait_time_seconds         = 0
      + tags                              = {
          + "Module" = "stock"
          + "Stage"  = "stage"
        }
      + visibility_timeout_seconds        = 30
    }

Easy no? You have many other possibilities of use of the setproduct function.

Ok Fábio, but if I have more than 2 sets that need to have a combination? No problem, the function resolves it for you:

> setproduct(["A", "B", "C"], ["1", "2", "3"], ["ABC", "XYZ"], ["123", "000"])

[
  [
    "A",
    "1",
    "ABC",
    "123",
  ],
  [
    "A",
    "1",
    "ABC",
    "000",
  ],
  [
    "A",
    "1",
    "XYZ",
    "123",
  ],
  [
    "A",
    "1",
    "XYZ",
    "000",
  ],
  [
    "A",
    "2",
    "ABC",
    "123",
  ],
  ...
  [
    "C",
    "3",
    "ABC",
    "123",
  ],
  [
    "C",
    "3",
    "ABC",
    "000",
  ],
  [
    "C",
    "3",
    "XYZ",
    "123",
  ],
  [
    "C",
    "3",
    "XYZ",
    "000",
  ],
]

> length(setproduct(["A", "B", "C"], ["1", "2", "3"], ["ABC", "XYZ"], ["123", "000"]))

36

Easy! Feel free to comment on it and give your suggestion.

If you want more information, you can check the Terraform manual for setproduct function.

Setup Gitlab CI with Terraform

Fábio Matavelli — Mon, 02 Aug 2021 09:24:44 +0000

Gitlab team is doing such a great job on their CI/CD pipelines. In this post, I will show you how to use its power to deploy infrastructure as code (IaC). For this, we are going to use Terraform, which is a tool for building, changing, and managing infrastructure in a safe, repeatable way.

Gitlab CI

I imagine that you already have an account on Gitlab (if not just go to gitlab.com and create one) or a Gitlab CE/EE installation.

So let's understand how the pipeline is configured.

First, you will need to create a file on the root of your repository called .gitlab-ci.yml. You can change the name of the file on the repository configuration, but for now, let's keep how it is.

Stages

Let's configure the stages of the pipeline. In this example, I will only define the steps for the Terraform, or Infrastructure As Code (IaC). You can define as many as other stages that you want to.

stages:
  - iac_validate
  - iac_plan
  - iac_apply

The stages will always run in sequence.

iac_validate

We will use this stage to validate the Terraform configuration. It will only validate the syntax and the inner reference of resources in the configuration.

iac_plan

In this stage, Terraform will run the plan of the configuration. It will check all the resources of the configuration against the state and it will show you what will be created, changed or destroyed.

iac_apply

This stage will, as the name says, apply the previous plan generated in the pipeline.

Default configuration

For this example, we will use the Terraform official Docker image to run the pipeline. So, let's define it as a default image in the Gitlab CI file:

default:
  image:
    name: hashicorp/terraform:latest
    entrypoint:
      - /usr/bin/env
      - "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
  before_script:
    - terraform init
  cache:
    key: terraform
    paths:
      - .terraform

We have defined the entrypoint for the image because Terraform uses terraform binary as entrypoint. As we are running this in the Gitlab CI pipeline, we need to change it to /usr/bin/env in order to job execute otherwise it will fail. In Docker, the entrypoint defines the program that will be executed in the Docker container. You can find more information here. If you are running Gitlab Runner as shell you can remove the image definition.

Another thing that we have defined in the default configuration is the before_script. The before_script will run before every job unless you define it in the job.

Last but not least, the cache. We defined the .terraform directory to be cached. This directory is created automatically by Terraform on every execution.

iac_validate

As I mentioned before, this stage will only validate the syntax and the inner reference of the resources specified in the configuration.

terraform_validate:
  stage: iac_validate
  script:
    - terraform validate
  except:
    refs:
      - master

stage specifies the stage that this job will run. You can define multiple jobs to run in the same stage
script receives all the commands that will be executed in the job. If any command fails, the job will fail
except tells the job to not run on the keys, in this case on the ref master (the branch master)

We will only execute it in the feature or fix branches, not on the master branch. I am assuming that you don't commit to master, right? ;-)
So it will execute and if there is any error, it will output to you.

For example, I have defined a wrong argument in the Terraform null_resource, take a look at the error that terraform validate thrown to me:

$ terraform validate
Error: Unsupported argument

  on example.tf line 2, in resource "null_resource" "example":
   2:   my_argument = "NOK"

An argument named "my_argument" is not expected here.

If everything is ok, it will output the following message to you:

$ terraform validate
Success! The configuration is valid.

iac_plan

Here Terraform will compare your configuration with the state and will show you all the resources that need to be created, changed or destroyed.

terraform_plan:
  stage: iac_plan
  script: 
    - terraform plan --out plan
  only:
    refs:
      - master
  artifacts:
    paths:
      - plan

Now we have two new keys defined in the terraform_plan job, that is only and artifacts.

only means that this job will only run in those keys, in this case in ref master (the master branch).
artifacts will retain the file that will be created by the terraform plan, with the parameter --out. In this case, the plan file. We will use this file in the next job to apply the infrastructure.

Again, using the last example from the terraform validate but, this time, with a valid configuration, let's see what the plan will output to us:

$ terraform plan --out plan
Refreshing Terraform state in-memory prior to plan...
The refreshed state will be used to calculate this plan, but will not be
persisted to local or remote state storage.


------------------------------------------------------------------------

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # null_resource.example will be created
  + resource "null_resource" "example" {
      + id       = (known after apply)
      + triggers = {
          + "my_variable" = "OK"
        }
    }

Plan: 1 to add, 0 to change, 0 to destroy.

------------------------------------------------------------------------

This plan was saved to: plan

To perform exactly these actions, run the following command to apply:
    terraform apply "plan"

I will not explain in this article the terraform state, it will be a subject for a future post.

If there was nothing to create, change or destroy, terraform will output it to you:

$ terraform plan --out plan
Refreshing Terraform state in-memory prior to plan...
The refreshed state will be used to calculate this plan, but will not be
persisted to local or remote state storage.


------------------------------------------------------------------------

No changes. Infrastructure is up-to-date.

This means that Terraform did not detect any differences between your
configuration and real physical resources that exist. As a result, no
actions need to be performed.

Let's move to our final job.

iac_apply

And here is the final job that will apply the infrastructure.

terraform_apply:
  stage: iac_apply
  script:
    - terraform apply --auto-approve plan
  when: manual
  allow_failure: false
  only:
    refs:
      - master

So, new keys here. Let me explain the when and allow_failure keys defined in here:

when defines when the job will be run. In this case, the job will run manually
allow_failure this specifies if the job can fail or not

This job will apply your infrastructure using the plan that was generated in the last job (terraform_plan). Take a look at what Terraform will output to you when you execute this job:

$ terraform apply --auto-aprove plan
null_resource.example: Creating...
null_resource.example: Creation complete after 0s [id=5363055958456141136]

Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

The state of your infrastructure has been saved to the path
below. This state is required to modify and destroy your
infrastructure, so keep it safe. To inspect the complete state
use the `terraform show` command.

State path: terraform.tfstate

We used the --auto-approve parameter so you don't have to type (well you can't type actually) yes to apply the configuration.

Resume

We reached the end of this post, let's see how our final .gitlab-ci.yml file is:

stages:
  - iac_validate
  - iac_plan
  - iac_apply

default:
  image:
    name: hashicorp/terraform:latest
    entrypoint:
      - /usr/bin/env
      - "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
  before_script:
    - terraform init
  cache:
    key: terraform
    paths:
      - .terraform

terraform_validate:
  stage: iac_validate
  script:
    - terraform validate
  except:
    refs:
      - master

terraform_plan:
  stage: iac_plan
  script: 
    - terraform plan --out plan
  only:
    refs:
      - master
  artifacts:
    paths:
      - plan

terraform_apply:
  stage: iac_apply
  script:
    - terraform apply --auto-approve plan
  when: manual
  allow_failure: false
  only:
    refs:
      - master

There were few things that I didn't cover in this post but I will do in future ones, so keep visiting me! :-)
I hope you have enjoyed this article. Please leave your comment so I can improve future posts. Enjoy it!