Terraform: Template file with loops for Dynamic Resource Management

Template files combined with loops provide powerful way to handle repetitive configurations dynamically, reducing complexity, errors and improving consistency. This blog is will explain how to use Template file with loops.

The Use Case

Imagine you need to manage infrastructure for an application that quite often requires a new S3 bucket for various purposes such as archive data, backups, different reports, etc. Each bucket requires specific permissions, and instead of manually creating IAM policies for each bucket, you can automate this via Terraform Template file and loops.

The Template File

The IAM policy template defines permissions for S3 buckets and includes a loop to handle multiple actions. Here’s the iam_policy.yaml.tpl template file:

Version: "2012-10-17"
Statement:
  - Effect: "Allow"
    Action:
      %{ for action in actions ~}
      - ${action}
      %{ endfor ~}
    Resource:
      - "arn:aws:s3:::${bucket}"
      - "arn:aws:s3:::${bucket}/*"

What this does

As you can see we use for loop to iterate over a list of actions (e.g., s3:GetObject, s3:PutObject, s3:ListBucket) to populate the IAM policy dynamically.

Additionally, for the bucket, we need to introduce a variable to insert the bucket's name into the ARN format later when calling the Template file.

This Template ensures that each policy is tailored to the specific actions and bucket provided during deployment.

Terraform Code

On of the ways to forward values to a Template file is to use locals as it is easier for you to create a more complex data structure than file expects.

locals {
  buckets = {
    "my-app-dev-archive"   = ["s3:PutObject", "s3:GetObject"],
    "my-app-dev-logs"   = ["s3:PutObject", "s3:GetObject"],
    "my-app-dev-backup" = ["s3:ListBucket", "s3:GetObject"],
    "my-app-dev-reports" = ["s3:DeleteObject"]
  }
}

Here we created buckets map to store bucket information.

• Map Keys: bucket name
• Map Values: IAM policy actions

Dynamically create IAM policies

resource "aws_iam_policy" "bucket_policies" {
  for_each = local.buckets

  name   = "policy-${each.key}"
  policy = templatefile("${path.module}/iam_policy.yaml.tpl", {
    bucket  = each.key,
    actions = each.value
  })
}

As you noticed we use for_each to iterate over buckets map.
The name of each policy will have a key from the map which is the bucket name. The value for the policy itself will be passed using the Template file function which accepts the path to the Template file as the first argument and inputs that the file expects as the second argument.
In the Template file as previously stated we have two variables:
${action} and ${bucket} means we need to pass both values as the second argument in the Template file method. From the buckets map we get the key which is the name of the bucket and we get the value for each key to pass the actions.

Dynamically generated output

Since in the buckets map we got four buckets, Terraform will generate four configurations for IAM policies. For instance, here are two examples:

Version: "2012-10-17"
Statement:
  - Effect: "Allow"
    Action:
      - "s3:PutObject"
      - "s3:GetObject"
    Resource:
      - "arn:aws:s3:::my-app-dev-archive"
      - "arn:aws:s3:::my-app-dev-archive/*"

Version: "2012-10-17"
Statement:
  - Effect: "Allow"
    Action:
      - "s3:PutObject"
      - "s3:GetObject"
    Resource:
      - "arn:aws:s3:::my-app-dev-logs"
      - "arn:aws:s3:::my-app-dev-logs/*"

Similar policies will be generated for rest of them or any new key value pair we add to buckets map.

Terraform plan would look something like this:

# aws_iam_policy.bucket_policies["my-app-dev-logs"] will be created
  + resource "aws_iam_policy" "bucket_policies" {
      + id     = (known after apply)
      + name   = "policy-my-app-dev-logs"
      + policy = jsonencode(
            {
              "Version" : "2012-10-17",
              "Statement" : [
                {
                  "Effect" : "Allow",
                  "Action" : [
                    "s3:PutObject",
                    "s3:GetObject"
                  ],
                  "Resource" : [
                    "arn:aws:s3:::my-app-dev-logs",
                    "arn:aws:s3:::my-app-dev-logs/*"
                  ]
                }
              ]
            }
        )
    }

  # aws_iam_policy.bucket_policies["my-app-dev-backup"] will be created
  + resource "aws_iam_policy" "bucket_policies" {
      + id     = (known after apply)
      + name   = "policy-my-app-dev-backup"
      + policy = jsonencode(
            {
              "Version" : "2012-10-17",
              "Statement" : [
                {
                  "Effect" : "Allow",
                  "Action" : [
                    "s3:ListBucket",
                    "s3:GetObject"
                  ],
                  "Resource" : [
                    "arn:aws:s3:::my-app-dev-backup",
                    "arn:aws:s3:::my-app-dev-backup/*"
                  ]
                }
              ]
            }
        )
    }

Benefits of Using Template Files with Loops

• Dynamic Configurations: Template files allow you to define reusable structures that adapt to input variables, reducing redundancy.
• Scalability: Adding new buckets or updating actions is as simple as modifying the local.buckets map.
• Consistency: Ensures that all policies follow the same structure and format, minimizing errors.
• Time-Saving: Automates repetitive tasks, saving significant time during deployments.

Some other examples where this approach could be used

• Generating Security Group rules
• Generating DNS records for multiple domains
• Generate EC2 User Data scripts
• ...

Conclusion

By combining Terraform template files with loops, you can dynamically generate configurations specialized to your infrastructure needs. However this approach is best suited for scenarios where complex, repetitive configurations need to be created. For simpler use cases like creating multiple resources with minor variations, using Terraform's built-in variables and loops without Template files is usually sufficient.