In 2019 one of my contributions for Terraform were published in the 0.12 version, the setproduct function. It added the capability to calculate the cartesian product of multiple sets of strings.
Cartesian product
I will try not to be too theoretical in this one, but the cartesian product is a mathematical function that, given two sets, calculates the sets of all ordered pairs, AxB = {(a, b),...}, where a is in A and b is in B.
To give you an example, we can use a deck of cards, where you have a rank of a 13-element set and a suit of a four-element set.
A = {A, K, Q, J, 10, 9, 8, 7, 6, 5, 4, 3, 2}
B = {♠, ♥, ♦, ♣}
AxB = {(A, ♠), (A, ♥), (A, ♦), (A, ♣), (K, ♠), ..., (3, ♣), (2, ♠), (2, ♥), (2, ♦), (2, ♣)}
You can find more information about cartesian product on Wikipedia.
Why that?
In 2018, I was working on a project that was using AWS as a cloud provider, and Terraform was responsible to manage the infrastructure as a code (IaC) for AWS. We needed to create multiple queues in SQS with the same purpose, receive data from an ERP to be processed by a lambda consumer and integrate it with an e-commerce platform.
The main idea was to create a queue for every ERP module that needed to have the message processed by this lambda consumer. We had some modules in the ERP at the beginning of the project, but the idea was to expand it to many more. Products, SKUs, stock, customers, orders, and more, needed to be integrated with the e-commerce platform.
Since the project wasn't that big in the beginning, we were using the same AWS account for the three environments (what I don't recommend), development, staging, and production. Said that we need to create the queues for every stage:
A = {product, sku, stock, customer, order}
B = {dev, stage, prod}
AxB = {(product, dev), (product, stage), (product, prod), (sku, dev)... (order, dev), (order, stage), (order, prod)}
Fifteen queues needed to be created and managed by Terraform, and we had the idea to expand it to more modules in the future.
The setproduct function
Terraform didn't have an easy way to do that, so I had the idea to create an MR adding the setproduct function to solve this problem.
Setproduct is easy to use, and you can pass more than 2 sets to it (AxBxCxN...), and it will return all the possible combinations of the sets.
So we had the problem, and now we have the solution, how to use it?
Simple:
> setproduct(["product", "sku", "stock", "customer", "order"], ["dev", "stage" , "prod"])
[
[
"product",
"dev"
],
[
"product",
"stage"
],
[
"product",
"prod"
],
[
"sku",
"prod"
],
...
[
"order",
"dev"
],
[
"order",
"stage"
],
[
"order",
"prod"
],
]
With that, now we can create the SQS queues:
locals {
queues = setproduct(["product", "sku", "stock", "customer", "order"], ["dev", "stage" , "prod"])
}
resource "aws_sqs_queue" "queue" {
for_each = {
for q in local.queues : "${q[0]}-${q[1]}" => {
module = q[0]
stage = q[1]
}
}
name = "${each.value.module}-${each.value.stage}"
tags = {
Module = each.value.module
Stage = each.value.stage
}
}
Running the terraform plan, we have:
Refreshing Terraform state in-memory prior to plan...
The refreshed state will be used to calculate this plan, but will not be
persisted to local or remote state storage.
------------------------------------------------------------------------
An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
+ create
Terraform will perform the following actions:
# aws_sqs_queue.queue["customer-dev"] will be created
+ resource "aws_sqs_queue" "queue" {
+ arn = (known after apply)
+ content_based_deduplication = false
+ delay_seconds = 0
+ fifo_queue = false
+ id = (known after apply)
+ kms_data_key_reuse_period_seconds = (known after apply)
+ max_message_size = 262144
+ message_retention_seconds = 345600
+ name = "customer-dev"
+ policy = (known after apply)
+ receive_wait_time_seconds = 0
+ tags = {
+ "Module" = "customer"
+ "Stage" = "dev"
}
+ visibility_timeout_seconds = 30
}
# aws_sqs_queue.queue["customer-prod"] will be created
+ resource "aws_sqs_queue" "queue" {
+ arn = (known after apply)
+ content_based_deduplication = false
+ delay_seconds = 0
+ fifo_queue = false
+ id = (known after apply)
+ kms_data_key_reuse_period_seconds = (known after apply)
+ max_message_size = 262144
+ message_retention_seconds = 345600
+ name = "customer-prod"
+ policy = (known after apply)
+ receive_wait_time_seconds = 0
+ tags = {
+ "Module" = "customer"
+ "Stage" = "prod"
}
+ visibility_timeout_seconds = 30
}
...
# aws_sqs_queue.queue["stock-stage"] will be created
+ resource "aws_sqs_queue" "queue" {
+ arn = (known after apply)
+ content_based_deduplication = false
+ delay_seconds = 0
+ fifo_queue = false
+ id = (known after apply)
+ kms_data_key_reuse_period_seconds = (known after apply)
+ max_message_size = 262144
+ message_retention_seconds = 345600
+ name = "stock-stage"
+ policy = (known after apply)
+ receive_wait_time_seconds = 0
+ tags = {
+ "Module" = "stock"
+ "Stage" = "stage"
}
+ visibility_timeout_seconds = 30
}
Easy no? You have many other possibilities of use of the setproduct function.
Ok Fábio, but if I have more than 2 sets that need to have a combination? No problem, the function resolves it for you:
> setproduct(["A", "B", "C"], ["1", "2", "3"], ["ABC", "XYZ"], ["123", "000"])
[
[
"A",
"1",
"ABC",
"123",
],
[
"A",
"1",
"ABC",
"000",
],
[
"A",
"1",
"XYZ",
"123",
],
[
"A",
"1",
"XYZ",
"000",
],
[
"A",
"2",
"ABC",
"123",
],
...
[
"C",
"3",
"ABC",
"123",
],
[
"C",
"3",
"ABC",
"000",
],
[
"C",
"3",
"XYZ",
"123",
],
[
"C",
"3",
"XYZ",
"000",
],
]
> length(setproduct(["A", "B", "C"], ["1", "2", "3"], ["ABC", "XYZ"], ["123", "000"]))
36
Easy! Feel free to comment on it and give your suggestion.
If you want more information, you can check the Terraform manual for setproduct function.
Oldest comments (0)