DEV Community: Darren Horwitz

pattern(composition) : terraform nested for loop

Darren Horwitz — Sat, 25 Mar 2023 14:05:49 +0000

Recently I have discovered a pattern that has emerged to become extremely handy when wanting to build out resources that are composed together, typically a one-to-many relationship.

One-to-Many

Now in terraform/cloud land the above image one can express the above image with something as simple as a security-group. The security group can be expressed as the "parent" and a security-group rule can would be expressed as a child of security-group. Thus, the security group is a composition of many security-group rules.

Foo Bar Code




locals {
  map = {
    "foo" = ["child1", "child4"]
    "bar" = ["child2", "child5"]
    "baz" = ["child3", "child6"]
  }

  out = merge([for parentKey, parentValue in local.map : {
    for child in parentValue : "${parentKey}/${child}" => {
      value = child
    }
    }
  ]...)
}


output "out" {
  value = local.out
}

The above code loops through each key-value pair in the map and then it loops through each item in the array (the value of the respective pair).

The for expression in the array will create a map based on the key and item . This is responsible for grouping the maps by the keys of the initial map.

The next for expression in the map is responsible for the creation of the key - values based on the respective value (array of strings ). The key will get namespaced by the respective key (e.g. "foo") and the respective value of the element in the array (e.g. "child1").

Resulting in the composed key : "foo/child1"

The result of the two for expressions before the merge looks like such:




  array = [
    {
      "foo/child1" = {
        value = "child1"
      },
      "foo/child4" = {
        value = "child4"
      }
    },
    {
      "bar/child2" = {
        value = "child2"
      },
      "bar/child5" = {
        value = "child5"
      }
    },
    {
      "baz/child3" = {
        value = "child3"
      },
      "baz/child6" = {
        value = "child6"
      }
    },
  ]

The ellipsis at the end of the array is similar to the spread operator in Javascript , similarly in golang it's a vararg .

What it does is spread the three maps as args in the merge function like such merge({...},{...},{...}).

From there the merge function handles the rest by merge the 3 maps into a single map. Just be careful with this as you need unique keys , otherwise the duplicated key will get overridden with the last respective value.



console output
  + out = {
      + "bar/child2" = {
          + value = "child2"
        }
      + "bar/child5" = {
          + value = "child5"
        }
      + "baz/child3" = {
          + value = "child3"
        }
      + "baz/child6" = {
          + value = "child6"
        }
      + "foo/child1" = {
          + value = "child1"
        }
      + "foo/child4" = {
          + value = "child4"
        }
    }

Example

So obviously the above pattern is pretty nifty, let's find an example of how this pattern can become useful.

Lets say , you need to create a bunch of vpc-endpoints and you want each endpoint to have its own security-group because you need to control what can access these endpoints.



terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 4.0"
    }
  }
}



locals {
  ## variables
  default_vpc_id = "vpc-id"
  subnet_ids     = ["subnet-1-id", "subnet-2-id"]
  endpoint_services = {
    "s3" = {
      service_name        = "com.amazonaws.eu-west-1.s3",
      private_dns_enabled = true,
      vpce_name_tag       = "s3-interface",

      security_group_ingress_rules = {
        "team-z-vpc-443" = {
          port        = 443,
          cidr_blocks = ["10.0.0.0/24"],
          protocol    = "tcp",
          description = "allow https vpc traffic in ",
        },
        "onprem-443" = {
          port        = 443,
          cidr_blocks = ["192.168.0.0/16"],
          protocol    = "tcp",
          description = "allow https onprem traffic in ",
        },
      },
    },
    "ddb" = {
      service_name        = "com.amazonaws.eu-west-1.dynamodb",
      private_dns_enabled = true,
      vpce_name_tag       = "ddb-interface",

      security_group_ingress_rules = {
        "team-a-vpc-443" = {
          port        = 443,
          cidr_blocks = ["10.0.1.0/24"],
          protocol    = "tcp",
          description = "allow https vpc traffic in ",
        },
      },
    }
  }


  ## nested for loop for sg-rules
  ## not that sg_key_ref is used to reference the sg which is created based on the key-values of local.endpoint_services
  endpoint_sg_ingress_rules = merge([for k, v in local.endpoint_services : {
    for ingressKey, ingressValue in v.security_group_ingress_rules : "${k}/${ingressKey}" => merge({ sg_key_ref = k }, ingressValue)
  }]...)
}

resource "aws_vpc_endpoint" "endpoint_services" {
  for_each = local.endpoint_services

  vpc_id            = local.default_vpc_id
  subnet_ids        = local.subnet_ids
  service_name      = each.value.service_name
  vpc_endpoint_type = "Interface"
  security_group_ids = [
    aws_security_group.this[each.key].id,
  ]
  private_dns_enabled = each.value.private_dns_enabled
  tags = {
    Name = each.value.vpce_name_tag
  }

}



resource "aws_security_group" "this" {
  for_each    = local.endpoint_services
  vpc_id      = local.default_vpc_id
  name        = "${each.value.vpce_name_tag}-sg"
  description = "sg for endpoint service: ${each.value.service_name}"

  tags = {
    Name = "${each.value.vpce_name_tag}-sg"
  }
}

resource "aws_security_group_rule" "ingress" {
  for_each          = local.endpoint_sg_ingress_rules
  type              = "ingress"
  from_port         = each.value.port
  to_port           = each.value.port
  protocol          = each.value.protocol
  cidr_blocks       = each.value.cidr_blocks
  ## referencing the parent ( security group) from which the rule is composed into
  security_group_id = aws_security_group.this[each.value.sg_key_ref].id
}

The cool thing is: when you need to create another endpoint , all you do is add another object into the map that meets the interface thats defined by the variable(yes I didn't create a variable for this example).

Alternatives/Anti-pattern

I think the alternative to this pattern is to rather create a module for the logic that needs to composed and a for_each on the module by passing the respective properties/input variables into the module. The module would be responsible for handling the logic.



module "endpoints" {
  for_each = local.endpoint_services
  source   = "../"

  default_vpc_id = "vpc-id"
  subnet_ids     = ["subnet-1-id", "subnet-2-id"]

  service_name        = each.value.service_name
  private_dns_enabled = each.value.private_dns_enabled
  vpce_name_tag       = each.value.vpce_name_tag

  security_group_ingress_rules = each.value.security_group_ingress_rules
}

Thanks for taking the time read this article , hopefully it will become useful for you. Lastly, comment below how you would implement this sort of pattern !

advanced terraform conditionals

Darren Horwitz — Sun, 15 Jan 2023 18:23:41 +0000

Recently, I had found a new sort of pattern of being able to work around the fact that terraform does not have any method of creating your own functions in configuration files, which resulted in this pattern that I am about to present to you.

Disclaimer, this is a bit overkill normal configurations and this came from the need in our team to create a module that is composed of submodules(which need to be conditionally deployed based on the variable inputs of the parent module). Now I know this post is not about why you should or shouldn't compose submodules into a parent module, that's up to you and your team . And obviously there are tradeoffs for that sort of decision.

Anyway.

Simple conditional deployments

You may skip to 'conditional deployments using objects' if you already know this stuff.

Typically when you want to conditionally deploy a resource/module, you would make use of a count in the resource/module.

such as below:

variable "create_vpc" {
 type = bool
 value = false
}

resource "aws_vpc" "main" {
  count = var.create_vpc ? 1 : 0
  cidr_block = "10.0.0.0/16"
}

This will create the vpc , if the create_vpc variable is true (by creating one instance of the aws_vpc.main resource) ; else it will not create the vpc resource i.e. 0.

There are other ways of utilising this pattern such as doing a count using the length of some string variable, a list, or using a for_each for a map/set .

length of string


variable "some_string" {
 type = string
 value = ""
}
resource "aws_vpc" "main" {
  count = length(var.some_string) !=0 ? 1 : 0
  cidr_block = "10.0.0.0/16"
}

length of list


variable "some_list" {
 type = list(any)
 value = []
}
resource "aws_vpc" "main" {
  count = length(var.some_list)
  cidr_block = "10.0.0.0/16"
}

this one can potentially create n vpcs depending on how many elements are provided in the list.

using a map


variable "some_map" {
 type = map(any)
 value = {}
}
resource "aws_vpc" "main" {
  for_each = var.some_map
  cidr_block = "10.0.0.0/16"
}

this one can potentially create n vpcs depending on how many key pairs are provided in the map.

conditional deployments using objects and locals

Now we are going to step it up a notch by using nested turnery statements with objects.

nested twice

So say you have some object like such:

variable "foo" {
  type = object({
    bar = optional(bool, true)
  })
  default = null
}

resource "aws_vpc" "single" {
  count = var.foo != null ? 1 : 0
  cidr_block = "10.0.0.0/16"
}

resource "aws_vpc" "nested" {
  count = var.foo != null ? var.foo.bar ? 1 : 0 : 0
  cidr_block = "10.0.0.0/16"
}

In this example, if foo is not being passed a value then it will result in null. Which means, both the aws_vpc.nested and aws_vpc.single resources will not get deployed as the variable (foo) is null.

If we pass an object , aws_vpc.single will get deployed as it not null.

This next explanation of the aws_vpc.nested resource will assume you know how optional functions work but if not I'll explain by means of a short example:

if you passed an empty object to the variable "foo" as such foo = {} , this would result property bar in foo taking the default value of true like such: 'foo = { bar = true}' . You can override this default value , when passing in a object with the respective properties. The docs probably do a better explanation that this so you can read it over here : https://developer.hashicorp.com/terraform/language/expressions/type-constraints#optional-object-type-attributes

So given that foo is not null then we can do an additional turnery expression on a property of the object, in this case bar.

What the count statement is saying , given that foo is not null and bar is true , then create one instance of this vpc resource.

you can view it like such

resource "aws_vpc" "nested" {
  count = var.foo != null ? 
                     var.foo.bar ? 1 : 0 
                  : 0
  cidr_block = "10.0.0.0/16"
}

*I know this is not legit terraform syntax

or like this


function deployVpc(foo){
  if (foo != null){
    if(foo.bar){
      return 1
    }else{
      return 0
    }
  }else{ 
    return 0
  {
}

The reason we can't use && logic expressions ( var.foo != null && var.foo.bar ? 1 : 0 ) is because it will fail at deployment time because you can't access a property on null variable. This is, of course, if the foo variable is null.

nested three times

the same sort of logic can apply to below


variable "foo" {
  type = object({
    bar = optional(object({
      baz= optional(bool,true)
    }),null)
  })
  default = null
}

resource "aws_vpc" "nested" {
  count = var.foo != null ? var.foo.bar != null ? var.foo.bar.baz ? 1 : 0 : 0 : 0
  cidr_block = "10.0.0.0/16"
}

Now doing this is overkill, and you should only resort to it as an anti pattern. Rather, keep the conditional logic as simple as possible.

the actual advanced implementation

So, the real reason why we have this sort of pattern is because our team typically works with a multi account setup in aws particularly. So we need to deploy a whole bunch of baseline resources that should be part of each account by default.

We found this pattern emerging with our network baseline implementation of the baseline module where we have transit gateway present and it is used for spoke vpcs' in another account and spoke vpcs' that may be present in the networking account.

So how this pattern would look like in practice would be as follows:


variable "network" {
  type = object({
    tgw = optional(object({
      create_tgw                   = bool
      exisiting_transit_gateway_id = optional(string, "")
    }))

    vpc = optional(object({
      cidr             = string
      azs              = list(string)
      tgw_subnet_cidrs = list(string)
    }))
  })
}

locals {
  deploy_vpc   = var.network != null ? var.network.vpc != null ? true : false : false
  tgw_not_null = var.network != null ? var.network.tgw != null ? true : false : false
  deploy_tgw   = local.tgw_not_null ? var.network.tgw.create_tgw ? true : false : false
}

resource "aws_ec2_transit_gateway" "this" {
  count = local.deploy_tgw ? 1 : 0
  # ... other vars

}
resource "aws_ec2_transit_gateway_vpc_attachment" "this" {
  count              = local.deploy_vpc && local.tgw_not_null ? 1 : 0
  subnet_ids         = [for subnet in aws_subnet.tgw : subnet.id]
  transit_gateway_id = local.deploy_tgw ? aws_ec2_transit_gateway.this[0].id : var.network.tgw.exisiting_transit_gateway_id
  vpc_id             = aws_vpc.this[0].id
}
resource "aws_subnet" "tgw" {
  count      = local.deploy_vpc && local.tgw_not_null ? length(var.network.vpc.azs) : 0
  cidr_block = var.network.vpc.tgw_subnet_cidrs[count.index]
}

resource "aws_vpc" "this" {
  count = local.deploy_vpc ? 1 : 0
  # ... other vars
}

Setting aside implementation of the above resources and the fact that there are plenty of resources that missing to make this complete.

This will deploy a vpc based on the vpc property in the network variable. Similarly, a transit gateway will be deployed if it is present and create_tgw is true.

Inside of the transit gateway attachment resource, one can see that its only dependent on the fact that a vpc is being deployed and that the tgw object in the network object is not null. This means that, one is either passing in an existing tgw id or using the one that has been created.

If the tgw object is null and the vpc object is not null then only a standalone vpc will be deployed based on the current logic.

Pseudo Functions

I Like to think of locals as pseudo functions because they return a value, based on the variables/resources that have been configured.

You can see how nifty they are when trying to separate messy logic that can make your file extend horizontally, which in turn makes it less readable.

I think a good use case of using the pseudo functions are for scenarios that are quite deterministic i.e. the true/false scenarios, but they can be used to further separate the logic complex turnery statements. Or even build up some strings or separate for logic.

Closing thoughts

Try to avoid this sort complexity, but if you do need something to do a bit of heavy lifting for you: look to use 'psuedo functions' to make your code a little bit more readable.

I hope you enjoyed reading this post as much as I did writing it.

Cheers.

Aws Resource Tags

Darren Horwitz — Fri, 24 Jun 2022 08:53:44 +0000

Anyone would like to share their thoughts on some resource tagging strategies and commonly used tags?

As well as the average amount of tags used on resources ?

Thanks a lot 🙏