CI/CD Pipeline for Terraform Workflow Using Amazon CodeCatalyst

Terraform workflow can be ran using several methods. One of them is running Terraform workflow inside CI/CD pipeline. Running Terraform workflow inside CI/CD pipeline can have several benefits, such as: automate creation or provision resources, simplify collaboration between engineers/developers, etc.

Table of Contents

1. Scenario
2. Prerequisites
3. Configuration Steps
   • Directory Structure
   • Terraform Backend Configurations
   • CodeCatalyst Workflow File Configurations
   • Example of Terraform Workflow Process in CodeCatalyst CI/CD
4. Summary

Scenario

In this blog, I will explain how to run Terraform workflow inside CI/CD pipeline which in this blog is Amazon CodeCatalyst. GitHub is used as source code repository and has been connected with CodeCatalyst. Next in CodeCatalyst pipeline/workflow, We will provision resources via Terraform. In this blog, Terraform is using S3 backend. Here is the topology used in this blog:

This blog is using reference from tutorial “Bootstrapping your Terraform automation with Amazon CodeCatalyst” written by @Cobus Bernard with some adjustment in configuration.

If You want to know more details about CodeCatalyst, please check CodeCatalyst documentation.

Prerequisites

• Before starting configuration, if you have not got AWS Builder ID account, please sign up first
• After sign up, go to Amazon CodeCatalyst web console to start project and configuration. Apply initial configuration that needed by CodeCatalyst such as:
  • Creating space
  • Connect CodeCatalyst space with AWS account
  • Create and connect IAM role that will be used for running CodeCatalyst workflow/pipeline
  • Connect GitHub source code repository
  • Define environment that will be used by workflow/pipeline
  • Please check my blog “Build and Release Container Image to Amazon Elastic Container Registry (ECR) via Amazon CodeCatalyst” to find reference for configuration above. Please check section Amazon CodeCatalyst Pipeline/Workflow Configuration
• Prepare S3 bucket and DynamoDB lock table (optional) that will be used by Terraform backend

Configuration Steps

Directory Structure

Below is directory structure used in this blog:

.
├── .codecatalyst/
│   └── workflows/
│       └── tf-sbx2-vpc-apse1.yml
└── sandbox2/
    ├── vpc/
    │   └── ap-southeast-1/
    │       ├── main.tf
    │       ├── resources.tf
    │       └── variables.tf
    └── tf-backend/
        ├── main.tf
        ├── resources.tf
        └── variables.tf

Terraform Backend Configurations

Configuration start with provisioning resources that will be needed by Terraform backend such as S3 bucket and DynamoDB table. Beside that, We also need to prepare IAM role that will be needed to provision resources from CodeCatalyst workflow/pipeline.

1. At first, We haven’t had any resource for storing Terraform state in S3, so that we need to run and store terraform state on our local, for example: laptop. Later, We will move that state from local to S3. We start by defining configuration in sandbox2/tf-backend/main.tf
   

   terraform {
     required_providers {
       aws = {
         source  = "hashicorp/aws"
         version = "~> 5.11.0"
       }
     }
   
     required_version = ">= 1.3.0"
   }
   
   provider "aws" {
     region = var.aws_region
   }
   

2. Define variable region in file sandbox2/tf-backend/variables.tf
   

   variable "aws_region" {
     type        = string
     description = "AWS Region"
     default     = "ap-southeast-1"
   }
   

3. Define resources S3, DynamoDB and IAM that needed by CodeCatalyst in file sandbox2/tf-backend/resources.tf. In this blog, IAM role CodeCatalyst has administrator access just for demo purposes. Please bear in mind to always use least privilege method on your production to prevent any security breaches
   

   ######
   # S3 #
   ######
   
   resource "aws_s3_bucket" "my_sandbox2_tfbucket" {
     bucket = "my-sandbox2-tfbucket"
   
     lifecycle {
       prevent_destroy = true
     }
   }
   
   resource "aws_s3_bucket_versioning" "my_sandbox2_tfbucket" {
     bucket = aws_s3_bucket.my_sandbox2_tfbucket.id
   
     versioning_configuration {
       status = "Enabled"
     }
   }
   
   resource "aws_s3_bucket_server_side_encryption_configuration" "my_sandbox2_tfbucket" {
     bucket = aws_s3_bucket.my_sandbox2_tfbucket.id
   
     rule {
       apply_server_side_encryption_by_default {
         sse_algorithm = "AES256"
       }
     }
   }
   
   resource "aws_s3_bucket_public_access_block" "my_sandbox2_tfbucket" {
     bucket                  = aws_s3_bucket.my_sandbox2_tfbucket.id
     block_public_acls       = true
     block_public_policy     = true
     ignore_public_acls      = true
     restrict_public_buckets = true
   }
   
   ############
   # DynamoDB #
   ############
   
   resource "aws_dynamodb_table" "my_sandbox2_tflocks" {
     name         = "my-sandbox2-tflocks"
     billing_mode = "PAY_PER_REQUEST"
     hash_key     = "LockID"
   
     attribute {
       name = "LockID"
       type = "S"
     }
   }
   
   #######
   # IAM #
   #######
   
   data "aws_iam_policy_document" "codecatalyst_assume_role_policy" {
     statement {
       actions = ["sts:AssumeRole"]
       principals {
         type = "Service"
         identifiers = [
           "codecatalyst.amazonaws.com",
           "codecatalyst-runner.amazonaws.com"
         ]
       }
     }
   }
   
   resource "aws_iam_role" "codecatalyst_admin" {
     name               = "codecatalyst-admin"
     assume_role_policy = data.aws_iam_policy_document.codecatalyst_assume_role_policy.json
   }
   
   resource "aws_iam_role_policy_attachment" "codecatalyst_admin" {
     role       = aws_iam_role.codecatalyst_admin.name
     policy_arn = "arn:aws:iam::aws:policy/AdministratorAccess"
   }
   

4. Refer AWS keys that needed for running terraform workflow commands, for example by exporting environment variables as explained in this document

5. Run terraform cli commands that needed for provisioning resources
   

   terraform init
   terraform plan
   terraform apply
   

6. Verify terraform apply is successful and resources successfully provisioned

7. Back to file sandbox2/tf-backend/main.tf, add additional configurations related to S3 backend
   

   ###########################
   # Terraform Configuration #
   ###########################
   
   terraform {
     backend "s3" {
       bucket         = "my-sandbox2-tfbucket"
       key            = "path/to/terraform.tfstate"
       region         = "ap-southeast-1"
       dynamodb_table = "my-sandbox2-tflocks"
       encrypt        = true
     }
   }
   

8. Run terraform init command again. Terraform will detect changes related to S3 backend and will move terraform state to S3 bucket that has been defined. Choose yes if asked for input
   

   terraform init
   
   Initializing the backend...
   Do you want to copy existing state to the new backend?
     Pre-existing state was found while migrating the previous "local" backend to the
     newly configured "s3" backend. No existing state was found in the newly
     configured "s3" backend. Do you want to copy this state to the new "s3"
     backend? Enter "yes" to copy and "no" to start with an empty state.
   
     Enter a value: yes
   
   Releasing state lock. This may take a few moments...
   
   Successfully configured the backend "s3"! Terraform will automatically
   use this backend unless the backend configuration changes.
   

9. Go to Amazon Web Services (AWS) web console, go to S3 service menu. Verify terraform.tfstate file has been copied to defined S3 bucket/path

CodeCatalyst Workflow File Configurations

After configurations for terraform backend has been finished as described above, next we continue configure CodeCatalyst workflow/pipeline.

Before starting workflow configuration, please make sure to configure all configurations that needed by CodeCatalyst as mentioned in Prerequisites.

As mentioned in document Build, test, and deploy with workflows in CodeCatalyst, CodeCatalyst use workflow definition file as reference workflow/pipeline configuration and this file is saved in directory ~/.codecatalyst/workflows/.

In this blog, I will create basic resources that needed by VPC on AWS. As mentioned on Directory Structure above, pipeline will be ran when there is changes inside directory sandbox2/vpc/ap-southeast-1.

Below is full content of workflow file codecatalyst/workflows/tf-sbx2-vpc-apse1.yml that used as example in this blog.

Name: tf-sbx2-vpc-apse1
SchemaVersion: "1.0"

Triggers:
  - Type: PUSH
    Branches:
      - master
    FilesChanged:
      - sandbox2\/vpc\/ap-southeast-1\/.*

Actions:
  terraform-plan:
    Identifier: aws/build@v1
    Inputs:
      Sources:
        - WorkflowSource
    Environment:
      Name: sandbox
      Connections:
        - Name: lanandra-sandbox
          Role: codecatalyst-admin
    Configuration:
      Container:
        Registry: DockerHub
        Image: hashicorp/terraform:1.5.7
      Steps:
      - Run: cd sandbox2/vpc/ap-southeast-1
      - Run: terraform fmt -check -no-color
      - Run: terraform init -no-color
      - Run: terraform validate -no-color
      - Run: terraform plan -no-color -input=false
    Compute:
      Type: EC2
  wait-period:
    DependsOn:
      - terraform-plan
    Identifier: aws/build@v1
    Inputs:
      Sources:
        - WorkflowSource
    Environment:
      Name: sandbox
      Connections:
        - Name: lanandra-sandbox
          Role: codecatalyst-admin
    Configuration:
      Steps:
      - Run: echo "Please wait for a while before terraform apply"
      - Run: echo "If you wish to cancel terraform apply, please stop this run"
      - Run: sleep 60
    Compute:
      Type: EC2
  terraform-apply:
    DependsOn:
      - wait-period
    Identifier: aws/build@v1
    Inputs:
      Sources:
        - WorkflowSource
    Environment:
      Name: sandbox
      Connections:
        - Name: lanandra-sandbox
          Role: codecatalyst-admin
    Configuration:
      Container:
        Registry: DockerHub
        Image: hashicorp/terraform:1.5.7
      Steps:
      - Run: cd sandbox2/vpc/ap-southeast-1
      - Run: terraform init -no-color
      - Run: terraform apply -auto-approve -no-color -input=false
    Compute:
      Type: EC2

Next, I will explain more detail each sections that defined on workflow file above.

1. Define workflow name. Then define how workflow will be ran. In this blog, workflow will be ran when there is a push to master branch but only ran when changes is under directory sandbox2/vpc/ap-southeast-1/ (regular expression or regex is used to detect changes)
   

   Triggers:
     - Type: PUSH
       Branches:
         - master
       FilesChanged:
         - sandbox2\/vpc\/ap-southeast-1\/.*
   

2. Define action. Action will be divided into 3 sections. The first one is used to run workflow terraform plan. In this section, action uses identifier aws/build@v1. Then action will be ran on sandbox environment that has been connected with AWS account and also has IAM role associated with that account. In Configuration section, has been defined that action will be ran on top of container using specific version of terraform public image that pulled from DockerHub. Also has defined working directory and several terraform commands which include terraform plan. Lastly, define compute type which is EC2
   

   Actions:
     terraform-plan:
       Identifier: aws/build@v1
       Inputs:
         Sources:
           - WorkflowSource
       Environment:
         Name: sandbox
         Connections:
           - Name: lanandra-sandbox
             Role: codecatalyst-admin
       Configuration:
         Container:
           Registry: DockerHub
           Image: hashicorp/terraform:1.5.7
         Steps:
         - Run: cd sandbox2/vpc/ap-southeast-1
         - Run: terraform fmt -check -no-color
         - Run: terraform init -no-color
         - Run: terraform validate -no-color
         - Run: terraform plan -no-color -input=false
       Compute:
         Type: EC2
   

3. Next action that defined is called wait-period. This action is defined so that there is an interlude between terraform plan and terraform apply. In case there is something that unwanted in terraform plan, so that we have a time to not continue to next action or process which is terraform apply. This action is workaround because by the time this blog is released, there is no out of the box solution provided by CodeCatalyst, for example manual approval. So in case there is something unwanted, we can abort the workflow. In this action, I created a logic to pause the workflow for 60 seconds and this action is depends on previous action which is terraform plan
   

   wait-period:
       DependsOn:
         - terraform-plan
       Identifier: aws/build@v1
       Inputs:
         Sources:
           - WorkflowSource
       Environment:
         Name: sandbox
         Connections:
           - Name: lanandra-sandbox
             Role: codecatalyst-admin
       Configuration:
         Steps:
         - Run: echo "Please wait for a while before terraform apply"
         - Run: echo "If you wish to cancel terraform apply, please stop this run"
         - Run: sleep 60
       Compute:
         Type: EC2
   

4. Next action that defined is action to run terraform apply. This action more or less similar with action terraform-plan, the difference is just terraform command that declared on configuration. In this section, action will run commands that needed for terraform apply. This action is depends on previous action which is wait-period
   

   terraform-apply:
       DependsOn:
         - wait-period
       Identifier: aws/build@v1
       Inputs:
         Sources:
           - WorkflowSource
       Environment:
         Name: sandbox
         Connections:
           - Name: lanandra-sandbox
             Role: codecatalyst-admin
       Configuration:
         Container:
           Registry: DockerHub
           Image: hashicorp/terraform:1.5.7
         Steps:
         - Run: cd sandbox2/vpc/ap-southeast-1
         - Run: terraform init -no-color
         - Run: terraform apply -auto-approve -no-color -input=false
       Compute:
         Type: EC2
   

Example of Terraform Workflow Process in CodeCatalyst CI/CD

After configure CodeCatalyst workflow file, next I will create example of Terraform codes. In this blog, I will create VPC resources using public module terraform-aws-modules/vpc/aws as reference. Here is the initial configuration/codes that defined:

sandbox2/vpc/ap-souhteast-1/main.tf

###########################
# Terraform Configuration #
###########################

terraform {
  backend "s3" {
    bucket         = "my-sandbox-tfbucket"
    key            = "path/to/terraform.tfstate"
    region         = "ap-southeast-1"
    dynamodb_table = "my-sandbox-tflocks"
    encrypt        = true
  }
}

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.11.0"
    }
  }

  required_version = ">= 1.3.0"
}

provider "aws" {
  region = var.aws_region
}

sandbox2/vpc/ap-souhteast-1/variables.tf

variable "aws_region" {
  type        = string
  description = "AWS Region"
  default     = "ap-southeast-1"
}

variable "env_name" {
  type    = string
  default = "sbx2"
}

sandbox2/vpc/ap-souhteast-1/resources.tf

#######
# VPC #
#######


data "aws_availability_zones" "available" {}

locals {
  azs = slice(data.aws_availability_zones.available.names, 0, 3)

  tags = {
    "myTag:environment" = "sbx2"
    "myTag:managedBy"   = "terraform"
  }
}

module "vpc" {
  source  = "terraform-aws-modules/vpc/aws"
  version = "5.1.2"

  name = var.env_name
  cidr = "10.255.8.0/21"

  azs             = local.azs
  private_subnets = ["10.255.8.0/24", "10.255.9.0/24", "10.255.10.0/24"]
  public_subnets  = ["10.255.15.0/24", "10.255.14.0/24", "10.255.13.0/24"]

  enable_nat_gateway = false
  single_nat_gateway = true

  tags = local.tags
}

1. To conduct testing of Terraform workflow in CI/CD, I will change 1 line of code inside file sandbox2/vpc/ap-souhteast-1/resources.tf. I will activate NAT gateway with changing this line of code that current value is false
   

   enable_nat_gateway = true
   

2. As defined in workflow file, all changes inside directory sandbox/vpc/ap-southeast-1 will trigger workflow run. To verify, go to CodeCatalyst web console, go to space and project where codes resided. Then go to CI/CD menu, choose Workflows. Go to tab Runs, verify there is new run that currently running.

   

3. Go to run details by clicking name/id of run. Then We will be redirected to overview page of run. Click action name terraform-plan, on the right will be showed tray box that display details configuration/steps of action. For example We want to see details of terraform plan, go to tab log, then click step terraform plan -no-color -input-false. Verify action succeeded

   

4. Verify terraform plan show expected result. Terraform will create resources releated to NAT gateway as mentioned in point number 1

   

5. As defined in workflow file, after terraform-plan action ran successfully, next workflow will proceed next action which is wait-period to give some interlude whether terraform plan will be executed to next action which is terraform-apply or apply will be aborted. If plan has met expectation and We want to proceed to terraform apply, no action needed, just wait 60 seconds and then verify action succeeded

   

6. But, if want to abort workflow and don’t want continue to terraform apply, click stop button when action still in progress

   

7. Let’s say We want to continue to terraform apply process, then terraform-apply action will be ran. We can see the details of the action inside tray box similar with previous actions. To verify resources has been provisioned via terraform apply, click step terraform apply -auto-approve -no-color -input=false

   

8. Verify terraform apply has been expected and action is succeeded

   

9. Go to VPC service page in AWS web console to verify NAT gateway has been provisioned successfully

   

Summary

We have reached the last section of this blog. Here are some key takeaways that can be summarized:

• Amazon CodeCatalyst can act as alternative for CI/CD engine/tools that can be used to run Terraform workflow
• Amazon CodeCatalyst use IAM role to interact with AWS services. By using this method, Terraform doesn’t need to inject static credentials such as AWS Access Key and AWS Secret Key into the pipeline. This can’t help prevent security breaches
• Amazon CodeCatalyst can give seamless experience if We want to deploy application to AWS environments
• But some workflow or logic haven’t been provided by CodeCatalyst by the time this blog is released. Maybe there will be some improvement in the future

Please check out my GitHub repository to see source code example that used in this blog:

• Terraform backend configuration: tf backend
• Amazon CodeCatalyst workflow file: codecatalyst workflow
• Example of Terraform AWS VPC configuration: tf aws vpc

Please comment if you have any suggestions, critiques, or thoughts.

Hope this article will benefit you. Thank you!