Refactor Terraform code with Moved Blocks - a new way without manually modifying the state

Most software and IT infrastructure projects which have been deployed to production have to deal with requirement changes during their lifetime. User expectations change, new use cases appear, traffic patterns are different than expected or new technology becomes available. Refactoring of existing code (application code as well as infrastructure-as-code) has always been an important task but also one of the major pain points in IT. A good support of refactoring tools and patterns can make a difference for a framework like Terraform compared with its competitors.

Setting the stage

Terraform by HashiCorp -- one of the major players in the
infrastructure-as-code framework world - has been around since 2014. It has been used to setup a lot of small, medium, and large projects all over the world. It provides a rich feature set to define infrastructure in a concise manner. One of its strengths is the way to create identical/similar resources using either the meta-argument count or the newer version for_each.

count makes it very easy to define identical resources like shown in the listing below which defines a very basic setup for 3 EC2 instances running on AWS:

locals {
  server_names = ["webserver1", "webserver2", "webserver3"]
}

resource "aws_instance" "web" {
  count = length(local.server_names)

  ami                       = "ami-0a1ee2fb28fe05df3"
  instance_type             = "t3.micro"

  tags = {
    Name = local.server_names[count.index]
  }
} 

Terraform stores references to resources created by using the count meta-argument in its internal state in an array using an index-based approach.

This works fine if a single instance must not be replaced or deleted. Such an action will affect all resources which are located on a higher index in the array due to the nature Terraform manages its state.

Trying to remove "webserver2" in the example above

locals {
  server_names = ["webserver1", "webserver2", "webserver3"]
}
...

will result in the destruction of the EC2 instance tagged "webserver3" and a renaming of the previous named "webserver2" instance into "webserver3". The result does not correspond to the expressed intention.

Version 0.12.6 of Terraform introduced the for_each meta-argument - a more flexible way to create identical/similar resources.

resource "aws_instance" "web" {
  for_each = toset(local.server_names)

  ami                         = "ami-0a1ee2fb28fe05df3"
  instance_type               = "t3.micro"

  tags = {
    Name = each.value 
  }
} 

The Terraform state references the resources no longer based on an index but by using a key-based approach. It is now possible to address a single resource without affecting others.

The removal of "webserver2" can now be performed successfully without affecting other resources.

Due to the greater flexibility of for_each it might be helpful or even required to refactor existing code (migrate from count to for_each). This has been possible in the past by manipulating the Terraform state directly using the terraform state mv CLI command. However, all manual state manipulations are brittle and prone to errors which make them as a kind of last resort.

From imperative to explicit

HashiCorp introduced an improved refactoring experience with version 1.1 of Terraform: the moved block syntax which allows to express refactoring steps in code instead of using an imperative attempt via CLI.

The moved block allows to specify the old and new reference of a resource like shown in the following example which has been rewritten to use for_each instead of count:

locals {
  server_names = ["webserver1", "webserver2", "webserver3"]
}

moved {
  from = aws_instance.web[0]
  to   = aws_instance.web["webserver1"]
}

moved {
  from = aws_instance.web[1]
  to   = aws_instance.web["webserver2"]
}

moved {
  from = aws_instance.web[2]
  to   = aws_instance.web["webserver3"]
}

resource "aws_instance" "web" {
  for_each = toset(local.server_names)

  ami           = "ami-0a1ee2fb28fe05df3"
  instance_type = "t3.micro"

  tags = {
    Name = each.value
  }
} 

A following terraform plan/apply reveals that no instance will be destroyed or modified in any way but only moved in the state from its old reference to its new one created by the way for_each works. No need for any manual state manipulation anymore but everything can be done securely using Terraforms native way to work.

moved blocks cannot only be applied to refactor count into
for_each syntax but also be used to rename resources, to move resources into modules and so on. Not everything is possible using the new language element, but many (not extremely complex) refactoring tasks can benefit from using it. Terraforms documentation contains different examples and use cases with further details.

Wrap-up

moved blocks have made refactoring existing Terraform projects easier and safer to perform. No manual steps are required any longer for many use cases even though terraform state mv is still there to solve problems which cannot be tackled by using the new element. It is helpful to have tooling/framework elements like this on at hand.

Depending on the type and size of the project (internal project or public module) it might make sense respectively it is even recommended by HashiCorp not to delete the blocks after having applied the changes. Not everyone using the module might already have fetched the latest version. Apart from avoiding trouble for users it might be helpful to document any significant changes on the project structure for later reviews. A short well written and dated comment combined with the moved block syntax might answer your question or the one of a colleague six months down the road.