Optimizing DevOps automation in the AWS cloud with GitHub Actions

DevOps is not just about automation, but automation is core to an effective DevOps practice, driving:

• Greater consistency and repeatability.
• Faster and more efficient workflows.
• Increased traceability and visibility.

The AWS CLI is an essential automation tool for DevOps tasks in AWS cloud environments. In this article you'll see its power for automating DevOps tasks, and how it becomes even more powerful when combined with GitHub Actions.

The Power of the AWS CLI

As I’ve written previously, my team at Asurion handles disaster recovery/failover with the Secondary Takes Over Primary (STOP) pattern, following a typical STOP implementation:

1. DNS records are associated with Route 53 health checks, which are associated with specific S3 objects.
2. Health check status is controlled by uploading (or deleting) its associated S3 object.
3. A failing health check cuts off requests to its DNS target.

The AWS CLI is the obvious tool for executing the failover procedure. Its commands to trigger DNS changes in accordance with the STOP failover pattern are put-object and delete-object.

To upload the object and cut off traffic:

aws s3api put-object --bucket <bucket-name> --key <object-name> --body <file-to-upload>

To reverse the operation and restore DNS traffic:

aws s3api delete-object --bucket <your-bucket-name> --key <your-object-name>

If the response from the S3 API is OK, it’s probably safe to assume that the health check change has been triggered. But Route 53 also has a rich API, which lets us extend the script to verify the health check status has flipped.

The commands for Route 53 aren’t as simple as S3, but the steps are straightforward:

1. Query for the relevant health check ID with the aws route53 list-health-checks command.
2. Get all Route 53 health checkers for that health check ID with aws route53 get-health-check-status.
3. Repeat step 2 until every health checker has the expected status, HEALTHY or UNHEALTHY.

This approach works anytime we want to stop traffic to an endpoint, whether for disaster recovery, testing major infrastructure changes without disrupting users, or regular patching cycles.

Automating these steps in a shell script checked into version control alleviates drudgery and provides consistency and repeatability for our failover procedure. The procedures become even more efficient and transparent when executed with GitHub Actions.

GitHub Actions in action

What makes GitHub Actions unique, compared to other platforms for building, testing, and deploying infrastructure, is its seamless integration with version control. This simplifies triggering jobs to execute tasks, like our scripted failover procedure.

One very powerful feature of GitHub Actions is its matrix strategy, which trivializes executing the same procedure with different configurations. In our scenario, multiple health check changes could be triggered and verified, in parallel, with a single job.

First, following the example from GitHub's own documentation, the workflow is defined with inputs and workflow_dispatch, which let the job be triggered with the GitHub API, CLI, or browser interface.

name: AWS CLI Execution
on:
  workflow_dispatch:
    inputs:
      config1:
        description: 'Config 1'
        required: false
        type: 'boolean'
        default: false
      config2:
        description: 'Config 2'
        required: false
        type: 'boolean'
        default: false
      config3:
        description: 'Config 3'
        required: false
        type: 'boolean'
        default: false

The workflow has two jobs. The first job creates a matrix of configurations based on inputs. Any combination of the 3 inputs can be added to the matrix.

jobs:
  create-matrix:
    runs-on: ubuntu-18.04
    outputs:
      config: ${{ steps.config.outputs.matrix }}
    steps:
      - id: config
        run: >
          echo "matrix=[$(echo '${{ inputs.config1 && '"config1"' || '' }}
          ${{ inputs.config2 && '"config2"' || '' }} 
          ${{ inputs.config3 && '"config3"' || '' }} 
          | sed 's/ *$//; s/^ *//; s/  */,/g')]" >> $GITHUB_OUTPUT

The second job executes the script with the 1, 2, or 3 configurations added to the matrix in the first job.

execute-script:
    runs-on: ubuntu-18.04
    environment: prod
    needs: create-matrix
    strategy:
      matrix:
        config: ${{ fromJson(needs.create-matrix.outputs.config) }}
    name: Executing script
    steps:
      - uses: actions/checkout@v3
      - name: Executing script with ${{ matrix.config }}
        uses: './.github/actions/script-name'
        with:
          config: ${{ matrix.config }}

Summary

The AWS CLI is a powerful tool for automating DevOps tasks in the AWS cloud. GitHub Actions provides a platform for executing workflows seamlessly and transparently from version control, all without the overhead of a separate build system.

Combined, they enable transparent, frequent, incremental, reversible changes in production at scale.