Husein Bajrektarevic

Posted on Jun 21, 2023

Configuring Terraform backend with AWS S3 and DynamoDB state locking

#terraform #aws #dynamodb #devops

In this blog post I have explained how to create a remote Terraform backend using Amazon S3 and DynamoDB services with state locking. Prerequisites are installed and configured AWS CLI and Terraform with some code editor like VS Code. For better understanding of importance to create Terraform backend with S3, first it’s necessary to understand what is state file, backend and what type of backend is a standard S3 backend, besides that what is state file lock and what purpose does it have within a teamwork - which we will see later in the demo part.

The Terraform state file represents a crucial part of Terraform infrastructure that contains information about managed infrastructure and its configuration. When we initially create our infrastructure, the state file is saved locally by default as a terraform.tfstate file in JSON format. Besides local backend, we also have standard backend which essentially is a remote backend on AWS S3 service or some other cloud provider and it’s always defined inside of Terraform backend configuration code block - which we will write later.

Why is the standard backend (like AWS S3) so important? It’s important because of team collaboration, security reasons etc., but again it’s not a complete solution, because when two or more team members work on infrastructure simultaneously, they could have certain issues with resource modification or creation because of unpredictable situations like executing other processes before the state file has been finalized. I will include more illustrative information about this issue later in the post.

To avoid this issue, Terraform can lock our state (State file locking) to prevent other team members from overwriting our infrastructure by using the same but modified state file simultaneously. In other words - changes that other team members make will be applied in certain order which we will see later with Acquiring state lock and Releasing state lock prompts while working on our infrastructure. In this case, state file is locked which provides us with protection of our state file from potential changes that could be made by other team members at the same time. In AWS we can use locking feature via DynamoDB table, which we will see in action later.

Terraform backend infrastructure diagram in AWS

The practical part of this blog post has 3 components:

First we will create simple infrastructure using the EC2 Terraform Instance module.
After that we will create a backend with Terraform backend code block which will build S3 bucket in our AWS account and save the state file here.
In the third part we will write code for DynamoDB state locking via Terraform and see how it looks in our AWS account.

DEMO 1 - creating infrastructure using EC2 Terraform instance module.

Before we start, we should validate if AWS CLI & Terraform are installed by using following commands:

$ aws --version
$ terraform --version

In the previous, introduction part, I added links that point toward official documentation that we can use to install and configure both AWS CLI and Terraform. After we confirmed that we have these two tools installed we can start creating our basic infrastructure. First we will create config.tf file in VS Code and write the following code:

terraform {
 required_providers {
   aws = {
   source = "hashicorp/aws"
   version = "5.4.0"
   }
 }
}

provider "aws" {
 region = "eu-central-1"
}

After that we will create variables.tf file where we will write the following variables:

variable "stage_name" {
 description = "The stage name for the environment"
 type        = string
 default     = "production"
}

variable "instance_name" {
 description = "Instance name"
 type        = string
 default     = "web_server"
}

variable "instance_type" {
 description = "EC2 instance type"
 type        = string
 default     = "t2.micro"
}

variable "key_name" {
 description = "The name of the SSH key pair"
 type        = string
 default     = "ec2-key"
}

variable "subnet_id" {
 description = "Subnet ID"
 type        = string
 default = "subnet-xxxxxxxxx" # write your subnet ID
}

After that we will create main.tf file where we will write code for our infrastructure resources, we will also create two resources for demo purposes - AWS EC2 instance as a web server with one security group:

module "ec2_instance" {
 source  = "terraform-aws-modules/ec2-instance/aws"

 name = "instance-${var.instance_name}"

 instance_type          = var.instance_type
 key_name               = var.key_name
 monitoring             = true
 vpc_security_group_ids = [aws_security_group.web_server_sg.id]
 subnet_id              = var.subnet_id

 tags = {
   Name      = var.instance_name
   Stage     = var.stage_name
   ManagedBy = "Terraform"
 }
}

resource "aws_security_group" "web_server_sg" {
 name        = "web-server-sg"
 description = "Security group for our web server"

 ingress {
   from_port   = 22
   to_port     = 22
   protocol    = "tcp"
   cidr_blocks = ["0.0.0.0/0"]
 }

 ingress {
   from_port   = 80
   to_port     = 80
   protocol    = "tcp"
   cidr_blocks = ["0.0.0.0/0"]
 }

 egress {
   from_port   = 0
   to_port     = 0
   protocol    = "-1"
   cidr_blocks = ["0.0.0.0/0"]
 }
}

The next step is to run the following commands in our terminal:

$ terraform init - where we can see local backend initializing.
$ terraform plan - where we can see the infrastructure plan.
$ terraform apply - where we build the planned infrastructure, we should use yes value when prompted, to execute the command and create resources.

After the command has been executed, besides our .tf files Terraform has created terraform.tfstate file, and if we execute $ cat terraform.tfstate we can see the content of our terraform.tfstate file:

This screenshot contains only part of the file where we can see JSON code that provides us with version of terraform.tfstate file including many other information like terraform version that was used to generate this file and very important part that contains resources with attributes (like arn), type, name and other specific resource values.

After the terraform.tfstate file has been generated, we are ready to start with the practical part for our backend creation.

DEMO 2 - Creating Terraform backend with AWS S3 service

When we built our infrastructure, Terraform has created a state file that is saved locally on our machine. In official documentation we can see code which we can write to build our S3 bucket on AWS using Terraform, and then we will migrate our state file to that bucket.

Before we proceed with practical part, I think it’s very important to add few paraphrased notes related to state file from the book Terraform: Up and Running, 3rd Edition:

If we use Terraform for our personal projects, it’s OK to use terraform.tfstate file locally on our machine, but if we want to use Terraform within our team on the real project we can encounter several issues with the approach above: our team members will not be able to access state file because our local machine is not a shared location, there would be no locking option and we could not isolate state files.
We can save our Terraform code on VCS (ie. Github), but we should not save our state file in the same way. Why? Because of the fact that code inside our Terraform state file is saved in textual format and can contain confidential information. For example, if we are using aws_db_instance resource to create our database, Terraform will save database username and password in state file which will be in textual format and it will upload it on GitHub which can represent security risk later.
Instead of using VCS for remote backend, the best practice is to use Terraform code block for remote backend, in this case we will use this code block later for creating Amazon S3 bucket.
Remote backend will provide us with a locking feature that is useful when several team members are working on a project and running the $ terraform apply command. In that case the first team member will have a lock on their side and will be able to apply the changes while other team members will be prompted to wait for a lock to be released. This does not mean that they will not be able to work, it only means that their changes will be applied in some order.
Besides this we should mention encryption in transit and how much of benefits we have when we are using Terraform with AWS S3 service which is Amazon’s managed file store and in the book above it’s mentioned as the recommended way for remote backend. You can read more about this in chapter 3 - How to Manage Terraform State.

Now we should continue with our practical part, first we will create a backend.tf file which we will use to write our code related to bucket and key parameters. We should also add region parameters but we already included them previously in our code.

resource "aws_s3_bucket" "terraform-backend-bucket" {
 bucket = "backend-bucket-iac-terraform"


 tags = {
   Name        = "tf-bucket"
   Environment = "dev"
 }
}

When it comes to naming convention, we can name our S3 bucket like backend-bucket-iac-terraform, and you should provide it with some unique name. According to the Terraform: Up and Running, 3rd Edition it’s recommended to add few layers of protection for our S3 bucket that will enable S3 versioning, the default encryption, and block all public access to S3 bucket:

resource "aws_kms_key" "mykey" {
 description = "This key is used to encrypt bucket objects"
 deletion_window_in_days = 10
}


resource "aws_s3_bucket_server_side_encryption_configuration" "default" {
 bucket = aws_s3_bucket.terraform-backend-bucket.id


 rule {
   apply_server_side_encryption_by_default {
     kms_master_key_id = aws_kms_key.mykey.arn
     sse_algorithm     = "aws:kms"
   }
 }
}


resource "aws_s3_bucket_versioning" "versioning_bucket_enabled" {
 bucket = aws_s3_bucket.terraform-backend-bucket.id
 versioning_configuration {
   status = "Enabled"
 }
}


resource "aws_s3_bucket_public_access_block" "public_access" {
  bucket                  = aws_s3_bucket.aws_s3_bucket.terraform-backend-bucket.id
  block_public_acls       = true
  block_public_policy     = true
  ignore_public_acls      = true
  restrict_public_buckets = true
}

After we wrote our code in the backend.tf file, we should run the $ terraform plan command to see what will be built from our infrastructure, in the output we should see that Terraform will create an S3 bucket with default encryption and enabled versioning. After that we will apply the $ terraform apply command to apply all changes.

We can see that the command has been executed successfully and we can confirm in our AWS S3 service dashboard that S3 bucket has been created.

However, our S3 bucket is empty and now it’s time to write terraform backend configuration code block in our backend.tf file with values that we used when we were creating the S3 bucket:

terraform {
   backend "s3" {
       bucket = "backend-bucket-iac-terraform"
       key = "terraform.tfstate"
       region = "eu-central-1"
   }
}

After we wrote this code block, to store our state file in S3 bucket we have to run the $ terraform init command to initialize our terraform project again. We will get a prompt this time with a question if we want to copy our state to the new “S3” backend, we should write the yes value.

After the command is executed successfully when we open our S3 bucket we can confirm that the terraform state file is placed here. At the same time if we open our terraform.tfstate locally on our machine, we will see that the file is empty.

At this moment we have successfully built our Terraform backend where instead of using our terraform.tfstate file that we have locally, we are using terraform.tfstate file remotely from our S3 bucket, and now it’s time to initialize DynamoDB state locking.

DEMO 3 - DynamoDB state locking

After we created our S3 bucket and migrated our state file we should create DynamoDB state locking by creating a table for Terraform state locking with a simple hash LockID key and one string attribute. We can write code for this in our backend.tf file.

resource "aws_dynamodb_table" "terraform_locks" {
   name = "terraform-state-locking"
   billing_mode = "PAY_PER_REQUEST"
   hash_key = "LockID"

   attribute {
       name = "LockID"
       type = "S"
   }
}

After that we will run $ terraform apply command for Terraform to create our table within the DynamoDB service, and we will get the following output on the screenshot below.

Now we will update our backend code block within our backend.tf file with the name of the recently created database.

terraform {
  backend "s3" {
      bucket = "backend-bucket-iac-terraform"
      key = "terraform.tfstate"
      region = "eu-central-1"

      dynamodb_table = "terraform-state-locking"
      encrypt        = true
  }
}

After that we will run $ terraform init -reconfigure to reconfigure our state locking.

In the screenshot below we can see our table inside of DynamoDB service.

With this backend configuration, Terraform will automatically pull the latest changes on the state file from S3 bucket and later by running the $ terraform apply command it will update and set the latest state file changes on the S3 bucket. We can test this by using output variables that we will write in our variables.tf file.

output "s3_bucket_arn" {
value       = aws_s3_bucket.terraform-backend-bucket.arn
description = "The ARN of the S3 bucket"
}

output "dynamodb_table_name" {
value       = aws_dynamodb_table.terraform_locks.name
description = "The name of the DynamoDB table"
}

These variables will print out the Amazon Resource Name (arn) of our S3 bucket and the name of our DynamoDB table. After we run the $ terraform apply command, we can notice Acquiring state lock (in the beginning) and Releasing state lock (at the end) prompts.

We can also see that we had an entry inside of our DynamoDB table with a corresponding Digest value.

Digest value is a hash value of our terraform.tfstate file, besides integrity check - if the content of our file has been changed or modified, Digest hash code will also change which tells us that new change has been made. We can test that if we comment out some existing resource inside of our code, change, modify or add a new one and then run the $ terraform apply command. In the screenshot below we can see Info that is providing information about state infrastructure.

When changes have been applied successfully, we can see a new Digest hash value.

Conclusion:

The goal of this blog post was to explain a way to create a remote Terraform backend by using S3 and DynamoDB with state locking. For better understanding of how important it is to create Terraform backend with S3, I explained what is a state file, backend and what type of backend is a standard S3 backend, what is a state file lock and what kind of purpose does it have when it comes to teamwork and collaboration.

Besides theory, in the practical part I explained how to create infrastructure by using the code from the Terraform: Up and Running book and official documentation. I tried to provide some simple and practical insights and keep the state file topic in the scope of the article, without addressing some other important concepts such as state file isolation and how to practically apply terraform_remote_state, which probably will be explained in some of my future posts.

To avoid unnecessary costs we can clean our resources by using the $ terraform destroy command - just remember to avoid using this command within the production environment.

Literature: