Speeding Up Your Terraform Projects with S3-backend Remote States

Update 8/6/2021: Abstracted out the Resource property in the s3_policy.json file, since the script below handles replacing that. 🙂

Ever wonder if there's a quicker way to initialize your infrastructure with Terraform on a remote state? My go-to is the S3 backend, but manually setting that up can take more time versus making a script to do the work for us.

Assuming you already have your AWS credentials setup already, we need to create two files. The first file is our S3 policy to let Terraform view the items, and to get and update our Terraform state.

Our S3 policy, s3_policy.json:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "s3:ListBucket",
      "Resource": "RESOURCE",
      "Principal": {
        "AWS": "ARN"
      }
    },
    {
      "Effect": "Allow",
      "Action": [
        "s3:GetObject",
        "s3:PutObject"
      ],
      "Resource": "RESOURCE/KEY",
      "Principal": {
        "AWS": "ARN"
      }
    }
  ]
}

So, if you're wondering about the ARN, RESOURCE, and KEY, that's okay. Our script will handle those by replacing those via sed magic, so when you change the bucket name in the bash script, you don't need to update the policy file. 😀

The second file is our actual bash script, let's say tf-setup.sh (remember to chmod +x this file so you can execute it!):

#!/bin/bash

# Create S3 Bucket
MY_ARN=$(aws iam get-user --query User.Arn --output text 2>/dev/null)
aws s3 mb "s3://tf-remote-bucket" --region "us-east-1"
sed -e "s/RESOURCE/arn:aws:s3:::tf-remote-bucket/g" -e "s/KEY/terraform.tfstate/g" -e "s|ARN|${MY_ARN}|g" "$(dirname "$0")/s3_policy.json" > new-policy.json
aws s3api put-bucket-policy --bucket "tf-remote-bucket" --policy file://new-policy.json
aws s3api put-bucket-versioning --bucket "tf-remote-bucket" --versioning-configuration Status=Enabled
rm new-policy.json

# Create DynamoDB Table
aws dynamodb create-table \
  --table-name "tf-lock-table" \
  --attribute-definitions AttributeName=LockID,AttributeType=S \
  --key-schema AttributeName=LockID,KeyType=HASH \
  --provisioned-throughput ReadCapacityUnits=1,WriteCapacityUnits=1 \
  --region "us-east-1"

Now, let's configure our main.tf:

terraform {
  required_version = ">= 1.0.0"
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 3.29"
    }
  }

  backend "s3" {
    encrypt        = true
    bucket         = "tf-remote-bucket"
    dynamodb_table = "tf-lock-table"
    region         = "us-east-1"
    key            = "terraform.tfstate"
  }
}

provider "aws" {
  region = "us-east-1"
}

A terraform init outputs:

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.

You should be all set to run terraform plan and terraform apply! I hope this comes as a useful tool in your toolbox and/or pipeline. If you have any questions or run into any problems, please let me know in the comments!