Avoid these mistakes when using Terraform with S3 backend

#s3 #aws #terraform #infrastructure

Using an S3 backend for Terraform is a popular choice because it's cheap, reliable and widely understood. However, it's pretty easy to configure the bare minimum and move on, forgetting three small things that, in the long run, will very likely make someone's life miserable.

Table for locking

Without a lock table, two engineers (or two CI/CD pipelines) can run terraform apply at the same time, both reading the same state, and then both writing conflicting results back. It can corrupt your state file silently, leaving your real infrastructure, and the recorded state, permanently out of sync and you rethinking your life choices.

Adding a DynamoDB lock table

resource "aws_dynamodb_table" "tf_lock" {
  name         = "terraform-state-lock"
  billing_mode = "PAY_PER_REQUEST"
  hash_key     = "LockID"

  attribute {
    name = "LockID"
    type = "S"
  }
}

terraform {
  backend "s3" {
    bucket         = "my-tf-state"
    key            = "terraform.tfstate"
    region         = "us-east-1"
    dynamodb_table = "terraform-state-lock"
    encrypt        = true
  }
}

Versioning S3 bucket

Your state file is not code, it's a live record of everything Terraform believes about your infrastructure (that's a beautiful quote). Without versioning, a bad apply, a manual edit or an accidental terraform state rm overwrites the state with no way back. Enabling versioning keeps a full history of every state revision, turning recovery from a crisis into a five-minute rollback instead of a life of suffering.

Enabling versioning on the state bucket

resource "aws_s3_bucket" "tf_state" {
  bucket = "my-tf-state"
}

resource "aws_s3_bucket_versioning" "tf_state" {
  bucket = aws_s3_bucket.tf_state.id

  versioning_configuration {
    status = "Enabled"
  }
}

resource "aws_s3_bucket_lifecycle_configuration" "tf_state" {
  bucket = aws_s3_bucket.tf_state.id

  rule {
    id     = "expire-old-versions"
    status = "Enabled"
    noncurrent_version_expiration {
      noncurrent_days = 90
    }
  }
}

It's worth having a lifecycle rule to keep storage costs in check, in cases where there are a lot of changes.

Enabling cross-region bucket replication

While it's rare for an AWS region to go down, when it does, your S3 bucket goes with it. If your state file is only in one region, you can't run Terraform at all during the outage, even to fix the very infrastructure that's affected. Cross-region replication keeps a live copy of every state version in a second region so your operations don't have a single geographic point of failure. Also, for the paranoid ones, if you want to add to the already humongous durability of S3 objects, having another region is a way to do that.

Replicating state to a second region

resource "aws_s3_bucket" "tf_state_replica" {
  bucket   = "my-tf-state-replica"
  provider = aws.us_west_2 # Secondary region
}

resource "aws_s3_bucket_versioning" "tf_state_replica" {
  bucket   = aws_s3_bucket.tf_state_replica.id
  provider = aws.us_west_2

  versioning_configuration {
    status = "Enabled"
  }
}

resource "aws_s3_bucket_replication_configuration" "tf_state" {
  bucket = aws_s3_bucket.tf_state.id
  role   = aws_iam_role.replication.arn

  rule {
    id     = "replicate-state"
    status = "Enabled"
    destination {
      bucket             = aws_s3_bucket.tf_state_replica.arn
      storage_class      = "STANDARD"
      replica_kms_key_id = aws_kms_key.replica.arn
    }
  }
}

Replication requires versioning enabled on both source and destination buckets.

Wrap-up

These three settings are pretty quick to set up and they help mitigating a class of incidents that make software engineers wake up at 3 am. So, ensure you're locking, versioning and replicating your state - sleep tight.

This article was created with the help of AI because, while I'm good at being a software engineer (I hope so), I suck at being a writer. On the other hand, the silly jokes and wordplay are completely mine because I'm really good at those. Also, the content comes from things I've done or learned, has been verified and is edited to be straight to the point, since AI loves to stall, but I hate it.

DEV Community