How to Migrate Terraform State Between Different Backends

The Terraform state file holds important information about your infrastructure, so it should be treated carefully when you need to move it between different backends. In this article, we will look at some common scenarios when you might want to migrate your state files, how to achieve it safely using Terraform commands, and the popular tool Terragrunt. 

If your state file is held locally, as a best practice, you'll typically want to migrate it to a remote backend in a storage account held in the cloud. That's why we'll take a look at how to move your local state file into Azure or AWS. The reverse is sometimes true if you want to take a state file from the remote end and use it locally for testing, so that process is covered before looking at two important options for use with the terraform init command for state management, terraform init -reconfigure and terraform init -migrate-state.

Why might you need to migrate your Terraform state?

There are several reasons why you might need to migrate your Terraform state:

1. Moving to the cloud
2. Switching cloud providers
3. Centralizing state management
4. Cost optimization

Moving to the cloud

 As a best practice, if you've been developing your infrastructure locally with Terraform, you should migrate it to a cloud environment. This would necessitate migrating your Terraform state to a backend that works in the cloud. 

Transitioning from local state storage to remote backends offers improved security and scalability. It ensures that sensitive infrastructure details are not stored locally and can be accessed securely by authorized users. Holding your state file remotely also facilitates team collaboration. It allows multiple team members to work concurrently on the same infrastructure, avoiding conflicts and ensuring consistency. 

Storing Terraform state remotely also provides better disaster recovery options. In case of local state loss (e.g., due to hardware failure), having a remote copy ensures that infrastructure configurations can be restored easily.

Switching cloud providers

Terraform can be used to manage infrastructure across multiple cloud providers. If you decide to switch providers, you'll need to migrate your Terraform state to the new provider's backend.

Centralizing state management

Local state files, while convenient for development, can become cumbersome to manage as your infrastructure grows. Migrating to a remote backend like Terraform Cloud, Terraform Enterprise, HashiCorp Consul or Spacelift allows centralized state management and collaboration.

Cost optimization

Some cloud providers might charge for storage space. Migrating your Terraform state to a more cost-effective backend solution could be a way to optimize your infrastructure costs.

Read more: Terraform Backends - Local and Remote Explained

How to migrate Terraform between different backends

Here are three different ways of migrating your Terraform state between different backends.

1. Using standard Terraform commands

terraform init -migrate-state is a built-in command used to migrate the Terraform state file from one backend to another. This allows you to change the location where Terraform stores and retrieves the state data.

To migrate the Terraform state using -the migrate-state flag, follow the steps below:

1. Set up backend configuration.
2. Update the access control on your backend.
3. Run terraform init.
4. Confirm the changes.

Note that if your local state file was previously locked, you might need to use terraform state lock -force to remove the lock before migration.

Before using this command, you will need to update your configuration file main.tffor the new backend. This involves creating the necessary resources depending on the chosen backend:

• Cloud Storage (S3, Azure Blob Storage, etc.): Create a bucket and configure access permissions. (See sections below for instructions on how to migrate state to Azure / AWS storage.
• Remote Services (Terraform Cloud, Terraform Enterprise): Set up authentication details and configure the remote service within Terraform configuration.

Using the terraform init command with the -migrate-state flag, offers several advantages:

• Safer migration: Terraform performs validation and checks during migration, reducing the risk of errors.
• Handles backend changes: It can handle migration between different backend types (e.g., local to S3) by automatically configuring the new backend based on your configuration file.
• Workspace management: It seamlessly handles migrating multiple workspaces if your Terraform configuration uses them.

2. Using the Terraform state pull and push commands

terraform state pull and terraform state push can be used to migrate state, but it is not the recommended best practice.

With the benefits of using terraform init -migrate-state in mind, you should back up your state file before using this method if you have a particular reason to do so. 

The steps to migrating Terraform state with pull and push commands are as follow:

1. Export state: Use terraform state mv to export your current state to a local file (e.g., terraform.tfstate).
2. Configure new backend: Set up the configuration for the new backend as mentioned before.
3. Pull state (optional): If the new backend supports pulling the existing state, use terraform state pull path/to/state.tfstate to retrieve the state from a remote location (consult the specific backend documentation for details).
4. Push state: Use terraform state push path/to/state.tfstate to push the exported state file (or the pulled state) to the new backend.

3. Using Terragrunt as a migration tool

Terragrunt is a popular wrapper around Terraform that adds extra features and simplifies the management of Terraform configurations. 

Terragrunt isn't specifically designed as a migration tool for Terraform state. However, Terragrunt encourages the use of remote state backends like Terraform Cloud or backend configurations defined in Terragrunt files, eliminating the need to manage local state files. 

To set up a Terragrunt remote backend:

1. Define the backend configuration in your Terragrunt configuration file (terragrunt.hcl). This configuration specifies where Terragrunt should store the Terraform state.

terraform {
  backend "s3" {
    bucket         = "your-bucket-name"
    key            = "path/to/terraform.tfstate"
    region         = "your-region"
    dynamodb_table = "terraform-state-lock"
  }
}

1. If your existing Terraform configurations are not already using Terragrunt, you can start by converting them to use Terragrunt. This involves creating Terragrunt configuration files (terragrunt.hcl) for each module and refactoring your Terraform configurations to follow Terragrunt's directory structure.
2. Modify the backend configuration in your Terragrunt configuration files to point to the new backend where you want to migrate the state. This might involve changing the backend type (e.g., from local to remote), updating credentials, bucket names, etc.
3. Initialize Terragrunt: Run terragrunt init to initialize Terragrunt and ensure that the new backend configuration is applied.

To learn more about Terragrunt, check out Terragrunt Tutorial -- Getting Started & Examples.

💡 You might also like:

• How to Enforce Policy as Code in Terraform
• Terraform on AWS – Deploying AWS Resources
• 12 Terraform Security Best Practices

Example: Migrating local state to Azure Storage Account remote backend

Now, let's look at some practical examples. Below are the steps needed to migrate your local Terraform state to an Azure Storage Account remote backend.

1. Create your storage account in Azure and create a container or desired folder structure within it to hold the state files.
2. Configure appropriate access control (ACL) on your Azure Storage Account and container to restrict access to authorized users.
3. Configure the Terraform backend block (usually in your *main.tf *file).

terraform {
  backend "azurerm" {
    resource_group_name = "your-resource-group"  # Replace with your resource group name
    storage_account_name = "your-storage-account"  # Replace with your storage account name
    container_name = "terraform-state"  # Replace with your desired container name
    key = "terraform.tfstate"  # Optional: Specify the filename within the container (defaults to 'terraform.tfstate')
  }
}

1. Run the following command:

terraform init -migrate-state

Terraform will detect the Azure Storage backend configuration and prompt you to confirm the migration. Enter "yes" to proceed. This will initiate the migration process, uploading your local state file to the specified location within your Azure Storage. 

If your Terraform configuration uses workspaces, Terraform will prompt you about migrating all workspaces.

Example: Migrating local state to Amazon S3

In this example, we will show how to migrate the local Terraform state to an AWS S3. (Read more about best practices around managing Terraform S3 backend.)

1. Create your Amazon S3 storage bucket in AWS to hold the state files.
2. Ensure your user account has S3 bucket creation permissions. Consider setting appropriate access permissions for your S3 bucket to ensure the Terraform state file is only accessible to authorized users.
3. Configure the backend in your configuration files (usually in your *main.tf *file).

terraform {
  backend "s3" {
    bucket = "your-bucket-name"
    key    = "environment/terraform.tfstate"
    region = "us-east-1"  # Replace with your desired region
  }
}

1. Run the following command:

terraform init -migrate-state

Terraform will detect the configured S3 backend and prompt you to confirm the migration. Type "yes" to proceed. This will initiate the migration process, uploading your local state file to the specified location within your S3 bucket. 

If your Terraform configuration uses workspaces, Terraform will prompt you about migrating all workspaces.

Migrating the remote backend back to the local machine

As mentioned previously, migrating your Terraform state from a remote backend to your local machine is possible, but it's generally not recommended.

1. Use the terraform state pull command to export the state from the remote backend to a local file.

 The specific command will vary depending on your backend type. Consult the documentation for your remote backend service for the exact syntax.

For example, in Terraform Cloud, you might use a command like:

terraform state pull remote-state.tfstate

Replace remote-state.tfstate with your desired filename for the local state file.

1. While Terraform can automatically use a local backend if no other backend is configured, you can explicitly define it in your main.tf file for clarity.

terraform {
  backend "local" {}
}

1. Initialize with local backend:

terraform init

Terraform init reconfigure vs migrate-state

The terraform init command has two relevant options when changing the backend configuration: -reconfigure and -migrate-state.

The terraform init -migrate-state command can be used to migrate your state file safely. The terraform init -reconfigure command, however, should be used only if your backend configuration file is corrupt or if you want to start fresh with a new backend entirely.

The -reconfigure flag completely disregards any existing backend configuration in your Terraform project directory. It discards any information about your existing state and requires a fresh initialization (downloading modules, etc.) Proceed with caution!

Managing Terraform state with Spacelift

Spacelift can optionally manage the Terraform state for you, offering a backend synchronized with the rest of the platform to maximize convenience and security. You also get the ability to import your state during stack creation, which is very useful for engineers who are migrating their old configurations and states to Spacelift.

Behind the scenes, Spacelift uses Amazon S3 and is storing all the data in Ireland. It's super simple to have Spacelift manage the state for you, as this behavior is achieved by default without you needing to do anything. You can read more about how it actually works here.

At the same time, it's protected against accidental or malicious access as Spacelift is able to map state access and changes to legitimate Spacelift runs, automatically blocking all unauthorized traffic.

If you are interested in learning more about Spacelift, create a free account today or book a demo with one of our engineers.

Key points

Your state file is precious as it acts like a map that keeps track of the infrastructure you've created and how it relates to your Terraform configuration files. As best practice, this should be held remotely, in cloud storage or a cloud service. If your state file is held locally, you'll want to use the terraform init -migrate-state command to safely move it to a remote location.

Written by Jack Roper