Terraform For_Each Meta-Argument 101

• What is Terraform for_each?
• How to Use For Each in Terraform?
  • Basic Syntax
  • How For Each Works
  • Key Points to Remember
• Example 1: Using For Each with a Set of Strings to Create EC2 Instances in Terraform
  • Defining Instance Names
  • Creating EC2 Instances with For Each
  • Accessing the Created Instances
• Example 2: Using For Each with a List of Strings to Create EC2 Instances in Terraform
  • How For Each Works with a List and Why We Use toset()
• Example 3: Using For Each with a Map of Strings to Create EC2 Instances in Terraform
  • How For Each Works with a Map
  • Using for Expressions in Outputs
• Example 4: Using For Each with a Map of Objects to Create EC2 Instances in Terraform
  • How For Each Works with a Map of Objects
• Example 5: Using For Each with a Data Source to Create EC2 Instances in Terraform
  • How For Each Works with a Data Source
  • Why This Approach is Effective
• Example 5: Using For Each with a Module to Create EC2 Instances in Terraform
  • How For Each Works with a Module
  • Advantages of Using For Each with Modules
• Understanding Terraform For Each Indexing Challenges
  • The Key Based Indexing Difference
  • The Impact of Modifying Keys
  • Best Practices for Handling For Each Indexing
  • The Takeaway: Embrace For Each with Care

In Terraform, scaling your infrastructure efficiently often means dealing with sets of resources that have unique characteristics, even if they belong to the same group. While the count meta-argument is useful for creating multiple similar resources, it falls short when you need more nuanced control over individual elements.

This is where the for_each meta-argument shines! Unlike count, which relies solely on indexes, for_each offers the flexibility to manage resources based on keys from a map or values from a set, giving you unparalleled control over each instance, allowing you to precisely configure resources without the hassle of repetitive code.

In this blog, we’ll explore how for_each can elevate your Terraform projects, making them more dynamic, adaptable, and responsive to change. Ready to unlock the next level of Terraform mastery? Let’s dive in!

What is Terraform for_each?

Terraform's for_each meta-argument is a flexible way to create multiple instances of a resource, data source, or module by iterating over a map or set of strings.

Unlike count, which relies on numeric indexing, for_each allows you to define unique configurations for each instance using the each object (each.key and each.value).

This makes it ideal for managing resources that share similarities but need individual customization, such as setting different CIDR ranges for subnets, all within a single resource block.

How to Use For Each in Terraform?

The for_each meta-argument in Terraform allows you to create multiple instances of a resource, data source, or module with great flexibility. Let's break down the general syntax and understand how to use it effectively.

Basic Syntax

resource "<resource_type>" "<resource_name>" {
  for_each = var.instances

  // Other resource attributes

  tags = {
    Name = each.<value/key>
  }
}

• <resource_type>: This specifies the type of Terraform resource, such as aws_vpc, aws_instance, etc.
• <resource_name>: A user-defined name used to reference this resource within your Terraform configuration.
• for_each: This is assigned a variable (e.g., var.instances), which can be a list, set, or map that dictates how many instances to create.

How For Each Works

1. Variable Source: The for_each meta-argument is assigned to a variable like var.instances. This variable can be a set of strings, list, or map.

• If it’s a set of strings, you’ll use each.value.
• If it’s a map, you can access both each.key and each.value.

1. The each Object: Within the resource block, Terraform automatically provides the each object, enabling you to reference the data for each instance created:
   • each.value: Represents the value of the current element in the iteration.
   • each.key: Represents the key if the for_each variable is a map.

Key Points to Remember

• each.value and each.key both are same for Sets, but generally use each.value as industry standard.
• Maps give you access to both each.key and each.value.

Note:
When you run the terraform plan command, it will show the resources Terraform plans to create based on the for_each logic. Feel free to apply these changes using terraform apply to see the for_each magic in action!

Example 1: Using For Each with a Set of Strings to Create EC2 Instances in Terraform

In this section, we'll explore how to create multiple AWS EC2 instances using Terraform's for_each meta-argument with a set of strings. This approach allows us to manage multiple resources efficiently and dynamically.

Defining Instance Names

We start by defining a variable instance_names, which contains a set of strings representing the names for each EC2 instance:

variable "instance_names" {
  description = "Set of names for each EC2 instance"
  type        = set(string)
  default     = ["ubuntu_server_1", "ubuntu_server_2", "ubuntu_server_3"]
}

The default value specifies that we want to create three instances with the names ubuntu_server_1, ubuntu_server_2, and ubuntu_server_3.

Creating EC2 Instances with For Each

The for_each meta-argument is used in the aws_instance resource to iterate over each item in the instance_names set, creating a separate EC2 instance for each name:

resource "aws_instance" "ubuntu_instance" {
   for_each         = var.instance_names

   instance_type = "t2.micro"
   ami           = "ami-0a0e5d9c7acc336f1"

   tags = {
    Name = each.value
   }
}

• for_each = var.instance_names: Instructs Terraform to loop through each item in instance_names.
• each.value: Represents the current name from the set during each iteration, ensuring each instance gets a unique Name tag based on the set item.

This configuration results in three EC2 instances, each tagged with its respective name: ubuntu_server_1, ubuntu_server_2, and ubuntu_server_3.

Accessing the Created Instances

To retrieve the IDs of these instances, we can use Terraform's output values. Here’s how you can access the ID of a specific instance and all instance IDs:

output "ubuntu_server_1_id" {
  value = aws_instance.ubuntu_instance["ubuntu_server_1"].id
}

output "all_ubuntu_servers_ids" {
  value = aws_instance.ubuntu_instance[*].id
}

• aws_instance.ubuntu_instance["ubuntu_server_1"].id: Fetches the ID of the ubuntu_server_1 instance.
• aws_instance.ubuntu_instance[*].id: Collects the IDs of all created instances in a set.

This setup provides a clear, scalable way to manage and access multiple resources, ensuring efficient and organized infrastructure provisioning with Terraform.

Example 2: Using For Each with a List of Strings to Create EC2 Instances in Terraform

In this example, we’ll see how to use for_each with a list of strings to create multiple EC2 instances, focusing on how for_each works with lists and why we use the toset() function.

variable "instance_names" {
  description = "List of names for each EC2 instance"
  type        = list(string)
  default     = ["ubuntu_server_1", "ubuntu_server_2", "ubuntu_server_3"]
}

resource "aws_instance" "ubuntu_instance" {
   for_each         = toset(var.instance_names)

   instance_type = "t2.micro"
   ami           = "ami-0a0e5d9c7acc336f1"

   tags = {
    Name = each.value
   }
}

output "ubuntu_server_1_id" {
  value = aws_instance.ubuntu_instance["ubuntu_server_1"].id
}

output "all_ubuntu_servers_ids" {
  value = aws_instance.ubuntu_instance[*].id
}

How For Each Works with a List and Why We Use toset()

The for_each argument expects a set-like collection of unique values to iterate over. However, lists in Terraform may contain duplicate values, which for_each does not support directly. Therefore, we convert the list to a set using toset() to ensure that each item is unique and treated correctly.

By using for_each = toset(var.instance_names), Terraform iterates over each unique name in instance_names, creating an EC2 instance for each item. each.value represents the current item in the iteration, allowing us to assign it as the Name tag for each instance.

Example 3: Using For Each with a Map of Strings to Create EC2 Instances in Terraform

In this example, we’ll show how to use for_each with a map of strings to create multiple EC2 instances. We will also dive deep into how the for_each statement works with a map and explain the for expression used in the output values.

variable "instance_ami" {
  description = "Map of AMI for each EC2 instance"
  type        = map(string)
  default = {
    "ubuntu_server" = "ubuntu_ami_id",
    "redhat_server" = "redhat_ami_id"
  }
}

resource "aws_instance" "linux_server" {
   for_each         = var.instance_ami

   instance_type = "t2.micro"
   ami           = each.value

   tags = {
    Name = each.key
   }
}

output "ubuntu_server_id" {
  value = [for instance_key, instance in aws_instance.linux_server : instance.id if instance_key == "ubuntu_server"]
}

output "all_linux_servers_ids" {
  value = [for instance in aws_instance.linux_server : instance.id]
}

How For Each Works with a Map

When we use for_each with a map, Terraform treats each key-value pair as an individual item to process:

• for_each = var.instance_ami: This allows Terraform to iterate over each key-value pair in the instance_ami map.
• each.key: Represents the current key in the map (e.g., ubuntu_server or redhat_server).
• each.value: Corresponds to the AMI ID associated with that key.

By setting for_each to this map, Terraform creates an instance for each key-value pair in the map, with the Name tag being set to the key and the ami attribute to the corresponding AMI ID.

Using for Expressions in Outputs

The for expression is a powerful way to build dynamic values based on existing data. Let’s break down how it works in our outputs:

1. First Output (Ubuntu Server ID)

   output "ubuntu_server_id" {
     value = [for instance_key, instance in aws_instance.linux_server : instance.id if instance_key == "ubuntu_server"]
   }

• for instance_key, instance in aws_instance.linux_server: Iterates over the linux_server instances, extracting both the key (instance_key) and the instance object (instance).
• : instance.id: This specifies that we want the id of the instance as the output value.
• if instance_key == "ubuntu_server": The if clause filters the instances, so only the ID of ubuntu_server is selected. Since the result is still a list (even with one item), this maintains the consistent format.

1. Second Output (All Linux Server IDs)

   output "all_linux_servers_ids" {
     value = [for instance in aws_instance.linux_server : instance.id]
   }

• This for expression iterates over all instances created by for_each, collecting each instance's ID into a list.

The use of for expressions here makes our output more flexible, adaptable, and concise. This way, we can extract exactly what we need based on certain conditions or patterns, making it a powerful tool in Terraform configurations.

Example 4: Using For Each with a Map of Objects to Create EC2 Instances in Terraform

In this example, we will use for_each with a map of objects to create multiple EC2 instances. We'll focus on how for_each works with a map of objects and how to access each object's properties within the resource configuration.

variable "instance_details" {
  description = "Map of objects containing details for each EC2 instance"
  type = map(object({
    instance_type = string
    ami           = string
  }))
  default = {
    "ubuntu_server" = {
      instance_type = "t2.micro"
      ami           = "ami-0a0e5d9c7acc336f1"
    }
    "redhat_server" = {
      instance_type = "t2.medium"
      ami           = "ami-0b69ea66ff7391e80"
    }
  }
}

resource "aws_instance" "linux_server" {
   for_each = var.instance_details

   instance_type = each.value.instance_type
   ami           = each.value.ami

   tags = {
    Name = each.key
   }
}

output "ubuntu_server_id" {
  value = [for instance_key, instance in aws_instance.linux_server : instance.id if instance_key == "ubuntu_server"]
}

output "all_linux_servers_ids" {
  value = [for instance in aws_instance.linux_server : instance.id]
}

How For Each Works with a Map of Objects

When using for_each with a map of objects, Terraform iterates over each key-value pair in the map. In this case:

• for_each = var.instance_details: Sets up the iteration over the instance_details variable, where each item is a key-value pair from the map.
• each.key: Represents the current key in the iteration (e.g., ubuntu_server or redhat_server).
• each.value: Contains the object associated with the current key, which in this case includes instance_type and ami.

For each iteration, Terraform creates a new aws_instance resource, setting the instance_type and ami properties using each.value.instance_type and each.value.ami, respectively. The tags block uses each.key to assign the instance name.

This approach provides a flexible way to manage more complex configurations by combining multiple attributes into a single map of objects, allowing you to handle more detailed setups for each resource without duplicating code.

Example 5: Using For Each with a Data Source to Create EC2 Instances in Terraform

In this example, we'll use a variable to define the AMI filters, making the configuration more flexible and easier to manage. We'll focus on explaining how for_each works with a data source and why this approach is highly effective for dynamic resource creation.

variable "ami_filters" {
  description = "Map of AMI filters for different Linux distributions"
  type = map(string)
  default = {
    "ubuntu_server" = "ubuntu/images/hvm-ssd/ubuntu-focal-20.04-amd64-server-*"
    "redhat_server" = "amzn2-ami-hvm-*-x86_64-gp2"
  }
}

data "aws_ami" "latest_linux_amis" {
  for_each = var.ami_filters

  most_recent = true
  owners      = ["099720109477", "137112412989"] # Owner IDs for Canonical and Amazon Linux
  filter {
    name   = "name"
    values = [each.value]
  }
}

resource "aws_instance" "linux_server" {
   for_each = data.aws_ami.latest_linux_amis

   instance_type = "t2.micro"
   ami           = each.value.id

   tags = {
    Name = each.key
   }
}

output "ubuntu_server_id" {
  value = [for instance_key, instance in aws_instance.linux_server : instance.id if instance_key == "ubuntu_server"]
}

output "all_linux_servers_ids" {
  value = [for instance in aws_instance.linux_server : instance.id]
}

How For Each Works with a Data Source

The for_each meta-argument is incredibly powerful when working with data sources, as it allows us to loop through each item in a collection and create resources dynamically based on the data retrieved. Let’s break this down step-by-step in an easy-to-understand way:

1. Using for_each in the Data Source:
   • In the data "aws_ami" "latest_linux_amis" block, we use for_each = var.ami_filters. This line tells Terraform to loop through the ami_filters variable, which is a map containing two entries: ubuntu_server and redhat_server.
   • For each entry in the map, Terraform will execute the data source block independently. This means it will search for the latest AMI based on the filter values provided (each.value).

Example:

• In the first iteration, each.key is "ubuntu_server" and each.value is "ubuntu/images/hvm-ssd/ubuntu-focal-20.04-amd64-server-*".
• In the second iteration, each.key is "redhat_server" and each.value is "amzn2-ami-hvm-*-x86_64-gp2".

This process ensures that we retrieve the most up-to-date AMI IDs for both the Ubuntu and RedHat servers.

1. Using for_each in the Resource Block:
   • In the resource "aws_instance" "linux_server" block, for_each = data.aws_ami.latest_linux_amis means Terraform will create one EC2 instance for each entry found by the data source.
   • each.key represents the unique name of the server ("ubuntu_server" or "redhat_server"), while each.value.id fetches the actual AMI ID for that specific server.

This results in two distinct EC2 instances, each tagged appropriately:

• The ubuntu_server instance with the AMI ID found for Ubuntu
• The redhat_server instance with the AMI ID found for RedHat

Why This Approach is Effective

• Dynamic Resource Creation: You don't have to manually define each resource. Instead, Terraform dynamically creates resources based on the data retrieved.
• Flexibility: By using variables and data sources, you can easily adapt the configuration to changes without modifying your core resource blocks.

Example 5: Using For Each with a Module to Create EC2 Instances in Terraform

In this example, we'll demonstrate how to use for_each with a module to create multiple EC2 instances dynamically based on the data passed to the module. This setup is both flexible and powerful, making it easy to scale your infrastructure. We'll focus on explaining how for_each works with a module, using a variable for better reusability and readability.

# ── main.tf ──────────────────────────────────────────────────────────────────────

# Define a variable for instance information
variable "instance_info" {
  description = "Map of instance names and their respective AMI filter names"
  type        = map(string)
  default     = {
    "ubuntu_server" = "ubuntu/images/hvm-ssd/ubuntu-focal-20.04-amd64-server-*"
    "redhat_server" = "amzn2-ami-hvm-*-x86_64-gp2"
  }
}

module "linux_servers" {
  source = "./modules/aws_instance"

  for_each = var.instance_info

  ami_filter_name = each.value
  instance_name   = each.key
  instance_type   = "t2.micro"
  ami_owner       = each.key == "ubuntu_server" ? "099720109477" : "137112412989" # Canonical & Amazon owner IDs
}

output "ubuntu_server_id" {
  value = [for instance_key, instance in module.linux_servers : instance.instance_id if instance_key == "ubuntu_server"]
}

output "all_linux_servers_ids" {
  value = [for instance in module.linux_servers : instance.instance_id]
}

# ── modules/aws_instance/main.tf ─────────────────────────────────────────────────
variable "ami_filter_name" {
  description = "AMI filter name for the instance"
  type        = string
}

variable "instance_name" {
  description = "Name tag for the EC2 instance"
  type        = string
}

variable "instance_type" {
  description = "Type of EC2 instance"
  type        = string
}

variable "ami_owner" {
  description = "Owner ID for the AMI"
  type        = string
}

data "aws_ami" "latest_linux_ami" {
  most_recent = true
  owners      = [var.ami_owner]
  filter {
    name   = "name"
    values = [var.ami_filter_name]
  }
}

resource "aws_instance" "linux_instance" {
  instance_type = var.instance_type
  ami           = data.aws_ami.latest_linux_ami.id

  tags = {
    Name = var.instance_name
  }
}

output "instance_id" {
  value = aws_instance.linux_instance.id
}

How For Each Works with a Module

When using for_each with a module, you gain the ability to create multiple instances of that module with different configurations based on a given set of inputs. Let's break down how this process works in an easy-to-understand manner:

1. Defining for_each in the Module Block:
   • In the main.tf file, the for_each argument in the module "linux_servers" block takes the instance_info variable as input.
   • This variable is a map where each key represents a unique server name ("ubuntu_server" and "redhat_server"), and each value specifies the AMI filter pattern for that server.

As a result, for_each instructs Terraform to create separate instances of the module for each entry in the instance_info map.

1. Passing Data to the Module Using for_each:

   • For each key-value pair in instance_info, the each.key and each.value references pass the necessary data to the module variables (ami_filter_name, instance_name, instance_type, and ami_owner).
   • For example:
     • For the first iteration: each.key = "ubuntu_server" and each.value = "ubuntu/images/hvm-ssd/ubuntu-focal-20.04-amd64-server-*".
     • The module will then create an instance using this information, specifically an Ubuntu server.

2. Handling Outputs with for_each Using for Expressions:

   • The output block ubuntu_server_id uses a for expression to filter out and retrieve the ID of the instance created for the "ubuntu_server" key.
   • The all_linux_servers_ids output collects all instance IDs generated by the module using another for expression, ensuring you have a complete list of all instances.

Advantages of Using For Each with Modules

• Scalability: The for_each approach lets you create as many instances as needed by simply modifying the input variable, without duplicating any module code.
• Flexibility: By using for_each with a map, you can create uniquely configured instances in one go, making your Terraform configurations both efficient and adaptable.

By leveraging for_each with modules, you achieve a powerful combination of modularity and dynamic resource creation. This approach allows you to maintain clean, reusable Terraform code, even when managing complex infrastructure setups.

Understanding Terraform For Each Indexing Challenges

Terraform's for_each meta-argument is a powerful tool for dynamically managing infrastructure, enabling us to iterate over a map or set of values effortlessly. This flexibility allows us to efficiently manage complex configurations with minimal duplication. However, there are some nuances and potential pitfalls, particularly with indexing, that we should be aware of.

The Key Based Indexing Difference

Unlike the traditional count meta-argument, where resources are indexed numerically (e.g., resource.example[0], resource.example[1]), for_each relies on unique keys from the map or set you provide. This key-based indexing introduces challenges in scenarios where sequential operations or precise ordering are necessary, as resources are now referenced by keys rather than predictable numerical indices.

For example, instead of referencing my_resource[0], you would access my_resource["key_name"]. While this approach is flexible, it can make certain tasks more complex, particularly when dealing with resources that depend on a specific order or sequence.

The Impact of Modifying Keys

Another consideration with for_each is how changes to the map or set can trigger unintended resource modifications. When you add, remove, or change keys in the map used by for_each, Terraform may perceive these changes as entirely new resources, resulting in resource destruction and re-creation. This behavior can be surprising, especially in a production environment, potentially causing downtime or unintended disruptions.

Best Practices for Handling For Each Indexing

• Use Stable and Predictable Keys: Choose keys that are unlikely to change over time, ensuring that your resources remain consistent and avoiding unnecessary re-creations.
• Be Cautious with Ordering Requirements: If the sequence of resource creation or updates is crucial, consider whether for_each is the right choice, as it doesn’t guarantee any specific ordering.
• Plan for Changes: Always run terraform plan before applying changes to understand the potential impact, particularly when modifying the map or set used with for_each.

The Takeaway: Embrace For Each with Care

While for_each brings tremendous flexibility and scalability to Terraform configurations, it's crucial to be mindful of its unique indexing behavior. By understanding these nuances, you can design more robust and resilient infrastructure code that leverages the power of for_each while avoiding common pitfalls.