Rolling back your Terraform Remote State in S3

While I hope this is a rare occurrence, I've run into this a few times where I had to rollback Terraform state to a previous version in AWS S3. Why not just keep it in the newer version? Well, some of our CI/CD pipelines used an older version of Terraform (say 0.12), and while none of the resource states actually changed, our code would no longer run after a developer ran something locally on their computer with a newer version of Terraform (say 0.14).

So! Here's a simple example on how to rollback state.

Note: you must have versioning enabled on the S3 bucket. If it isn't, PLEASE do that for all places that Terraform State is stored :p

Setup

I'm going to set up this up completely fresh with a new remote state object in S3 with the following Terraform code:

terraform {
  backend "s3" {
    bucket  = "shannon-terraform"
    key     = "dev.to-example/terraform.tfstate"
    region  = "us-west-2"
    profile = "shannon"
  }

  required_providers {
    null = {
      source  = "hashicorp/null"
      version = "~> 3.1"
    }
  }
}

resource "null_resource" "fake1" {
  count = 1
}

So, I terraform apply -auto-approve'd this. Next, I'm ticking up the count to 2 and applying that as well. Thus, I have two different versions of Terraform State saved in that bucket path.

Now, we should be able to see these with the S3 api: aws s3api list-object-versions --bucket shannon-terraform --prefix dev.to-example/terraform.tfstate --output json --profile shannon | jq -r ".Versions[].LastModified"

In my case, I see two modifications:

2021-06-23T22:17:02+00:00
2021-06-23T22:16:28+00:00

Next up, I'm gonna grab a copy of my previous Terraform State locally in my current working directory: aws s3api get-object --bucket shannon-terraform --key dev.to-example/terraform.tfstate --version-id [VersionID] [local_copy_name] --profile shannon If you're wondering where the VersionID comes from, it should be in the output of the above list-object-versions command, but it won't show up without removing the jq parsing.

Finally, let's push up that local backup into the S3 path: aws s3 cp ./[local_copy_name] s3://shannon-terraform/dev.to-example/terraform.tfstate --profile shannon. Re-run the above command for listing objects, and you should now see another version!

Finally, you can confirm the update with terraform state list. In my case, I now only see one null_resource again after rolling the state back to before creating a second one.

Please be careful in the use cases here! Getting Terraform State out of sync with actual infrastructure can be disastrous. This should be a niche case for ever using this.

Comments

[TARUN-777]

How just updating the state file with previous version ensures that resources are decreased from 2 to 1 ? To apply there should be a corresponding change in tf file as well ?