Setup CICD pipeline for Terraform using GitHub Actions

In the tutorial, I will walk you through how to setup a CICD pipeline using GitHub Actions to deploy AWS resources using Terraform.
The tutorial only focuses on the GitHub Actions part. I assume that you already have some experience on Terraform, GitHub Actions and want to know how to deploy Terraform projects to AWS using Github Actions. I will use a simple terraform project that only deploy a SNS topic to AWS for demo purpose. However, there is a batch of AWS resources and modules defined in a real terraform project with more complex terraform configuration. To deploy a larger terraform project, more complex GitHub Actions workflows are necessary. There will be another tutorial to introduce how to organize and deploy a larger Terraform project automatically using GitHub Actions workflows.
You can find the demo source code from Source Code

Preprequisties

1. VSCode or other IDE as you prefer
2. AWS account

Code Structure

Here is the main code structure which includes GitHub Actions workflows, Terraform configuration infrastructure (S3 Bucket and DynamoDB table), Terraform definitions and settings.

├── .github
│   └── workflows            # several workflows for different purposes
│       ├── tf-deploy-to-dev.yml
│       ├── tf-deploy-to-prod.yml
│       ├── tf-destroy.yml
│       └── tf-unit-tests.yml
├── cloudformation
│   └── infrastructure.yaml # the cloudformation for terrform state bucket and lock table
├── main.tf                 # Terraform configuration and resources 
├── outputs.tf
├── variables.tf
├── settings                # Multiple environments settings
│   ├── dev
│   │   ├── backend.conf
│   │   └── variables.tfvars
│   └── prod
│       ├── backend.conf
│       └── variables.tfvars

GitOps for Multiple Environments

From the code structure above, I enabled multiple environment deployment, dev for development environment, and prod for production environment. Meanwhile, I created the specific Github Actions deployment workflow for each environment, and two branches for the corresponding environment. I'm going to implement the GitOps scenario as below.

GitHub Repo Branches

There are two branches in the GitHub repository. The main branch is the default and release branch, which targets the production environment. The dev branch is the development branch, which targets the development environment. When you start to implement a new feature, you should create a feature branch from the main branch and name it as, for example, feature/add-a-new-page-on-ui. Then, you code and test on the local environment. You create a PR to merge code change from feature/add-a-new-page-on-ui to dev branch. After merging, deploy code on dev branch to development environment and validate the change. If everything works as expected, you can merge the change from dev to main branch, and wait for the release to production. This is the common lifecycle for the software development.
One of the biggest benefits is any change has to be validated in the development environment before moving to the production environment. Besides, you get a dedicated source code branch for each environment. The dev branch can be changed frequently, and release to production environment may happen weekly or bi-weekly.

Terraform Settings

There are environment specific Terraform backend configurations and variables under settings directory. In the GitHub Actions workflow, I use -backend-config to specify backend configuration.

terraform init -reconfigure -backend-config=settings/dev/backend.conf

For plan command, use -var-file to specify Terraform variables.

terraform plan -var-file=settings/dev/variables.tfvars

GitHub Environment

GitHub has an environment concept for deployment. Environments are used to describe a general deployment target like production, staging, or development. You can configure environments with protection rules and secrets. When a workflow job references an environment, the job won't start until all of the environment's protection rules pass. A job also can not access secrets that are defined in an environment until all the deployment protection rules pass.

In the demo. I created two environments, one named development, another named production via GitHub console as above. In each environment, there are two environmental secrets and one environmental variable, which are injected into workflow as below in env block. Then these environmental variables can be accessible within the entire workflows. You are also allowed to define env at job level as needed for security.

env:
  AWS_ACCESS_KEY_ID: "${{ secrets.AWS_ACCESS_KEY_ID }}"
  AWS_SECRET_ACCESS_KEY: "${{ secrets.AWS_SECRET_ACCESS_KEY }}"
  AWS_REGION: "${{ vars.AWS_REGION }}"

GitHub Actions Workflows

There are four workflows under .github/workflows directory. You can get the purpose from the names. I created specific workflow for deployment to the target environment. One for development, another for production. A destroy workflow to remove Terraform resources for the given environment. A unit test workflow that will be triggered automatically on each push action.

Workflow: tf-deploy-to-dev and tf-deploy-to-prod

Let's go through the deployment workflow from top to bottom.

1. name

The name of the workflow, which will be displayed on the list of All workflows under Actions tab.

2. on

A workflow run is triggered for any workflows that have on: values that match the triggering event. Some events also require the workflow file to be present on the default branch of the repository in order to run. The script below means the workflow run is triggered when:

1. There is a push event on the dev branch, or
2. There is a pull_request event on the dev branch, or
3. There is a manual trigger event (workflow_dispatch) on the dev branch

on:
  push:
    branches:
      - dev
  pull_request:
    branches:
      - dev
  workflow_dispatch:
    branches:
      - dev

4. permissions

The permissions you need to grant to the workflow so that it can take specific action. For example, if you want to write output to your PR, you have to grant write access to pull-requests actions.
Find the details from Permissions and Define Access.

permissions:
  contents: read
  pull-requests: write

5. env

Define AWS authentication related environment variables here. Two are secret , one is variable. These environment variables are saved in Github.

6. jobs

A workflow run is made up of one or more jobs. I created two jobs with multiple steps in each. terraform-plan is used to create a plan, save it to artifact. terraform-apply is used to download the plan from artifact, and apply the plan to environment.

In terraform-plan job, there are multiple steps. Each step fulfills a particular task. For example, Checkout step pulls source code into job runner workspace.

For prod deployment workflow, the only difference is the workflow can only be triggered manually from GitHub console.

Workflow: tf-destroy

I created a dedicated workflow for destroying Terraform resources. This is optional, but should be useful when there is something wrong with Terraform resources, and you have to destroy resources from the environment to redeploy/recreate the infrastructure manually via workflow.

Workflow: tf-unit-test

Optional, but it should be better to keep your code high quality and system availability.

Summary

In this tutorial, I walked you through how to create a multiple environments Terraform project using GitHub Actions workflow. You can take it as an example, and build your project with more complex and advanced features. If you are newbie to GitHub Actions (well I'm a newbie as well), the official documents are the best reference to learn GitHub Actions.

That's all.

I'm always looking forward to any comments and suggestions. Thank you. Happy learning!

Comments

[Lukas Krimphove]

Great tutorial! GitHub Actions is such a great way to kickstart CICD for any project.
Have you ever used Terragrunt? I prefer it over plain Terraform when working with multiple environments. It makes it easier to handle more complex solutions.

[Camille He]

I never heard about Terragrunt before, so I just google it. It's a cool tool to solve some common tricky problems when using Terraform to manage Cloud resources in the real world. I will dive into it when I get a chance. Thx Lukas, nice sharing!

[AutomationRom]

@camillehe1992 is this repo public? if yes, can you share a link to it?