DEV Community: Pipo Sanfilippo

How to Backup the Terraform states from Terraform Cloud workspaces.

Pipo Sanfilippo — Mon, 05 Jul 2021 16:27:40 +0000

This tool is the result of a no so sad story as it should have been. We had a backup.
Until today, Terraform Cloud does not provide a mechanism to backup and restore the terraform states of the workspaces. This tool is the first version of a lambda that every time a workspace changes in the terraform state, the Terraform state will save in an S3 Bucket. So in case of disaster, the Terraform state can be pushed to the workspace.

Let us begin.
This tool is a serverless and event-driven solution. We are going to use:
Lambda repository
S3
API Gateway
Terraform Cloud Workspace Notifications

The workflow of the process is:
The terraform state change in terraform cloud because a new plan was applied or that a plan has been applied and ended in an errored state. In any case, the terraform state has changed so that we will save this new state.
This change of state triggers a notification in Terraform Cloud to a generic endpoint. This generic endpoint is an API Gateway endpoint that has a lambda function integrated.

The lambda function receives the payload from the terraform cloud notification, and this payload looks like this:

{
   "resource": "/{proxy+}",
     "path": "/hello/world",
     "httpMethod": "POST",
     "headers": {
        "Accept": "*/*",
        "Accept-Encoding": "gzip, deflate",
        "cache-control": "no-cache",
        "CloudFront-Forwarded-Proto": "https",
        "CloudFront-Is-Desktop-Viewer": "true",
        "CloudFront-Is-Mobile-Viewer": "false",
        "CloudFront-Is-SmartTV-Viewer": "false",
        "CloudFront-Is-Tablet-Viewer": "false",
        "CloudFront-Viewer-Country": "US",
        "Content-Type": "application/json",
        "headerName": "headerValue",
        "Host": "gy415nuibc.execute-api.us-east-1.amazonaws.com",
        "Postman-Token": "9f583ef0-ed83-4a38-aef3-eb9ce3f7a57f",
        "User-Agent": "PostmanRuntime/2.4.5",
        "Via": "1.1 d98420743a69852491bbdea73f7680bd.cloudfront.net (CloudFront)",
        "X-Amz-Cf-Id": "pn-PWIJc6thYnZm5P0NMgOUglL1DYtl0gdeJky8tqsg8iS_sgsKD1A==",
        "X-Forwarded-For": "54.240.196.186, 54.182.214.83",
        "X-Forwarded-Port": "443",
        "X-Forwarded-Proto": "https"
    },
    "multiValueHeaders": {
        "Accept": ["*/*"],
        "Accept-Encoding": ["gzip, deflate"],
        "cache-control": ["no-cache"],
        "CloudFront-Forwarded-Proto": ["https"],
        "CloudFront-Is-Desktop-Viewer": ["true"],
        "CloudFront-Is-Mobile-Viewer": ["false"],
        "CloudFront-Is-SmartTV-Viewer": ["false"],
        "CloudFront-Is-Tablet-Viewer": ["false"],
        "CloudFront-Viewer-Country": ["US"],
        "Content-Type": ["application/json"],
        "headerName": ["headerValue"],
        "Host": ["gy415nuibc.execute-api.us-east-1.amazonaws.com"],
        "Postman-Token": ["9f583ef0-ed83-4a38-aef3-eb9ce3f7a57f"],
        "User-Agent": ["PostmanRuntime/2.4.5"],
        "Via": ["1.1 d98420743a69852491bbdea73f7680bd.cloudfront.net (CloudFront)"],
        "X-Amz-Cf-Id": ["pn-PWIJc6thYnZm5P0NMgOUglL1DYtl0gdeJky8tqsg8iS_sgsKD1A=="],
        "X-Forwarded-For": ["54.240.196.186, 54.182.214.83"],
        "X-Forwarded-Port": ["443"],
        "X-Forwarded-Proto": ["https"]
    },
   "queryStringParameters": {
      "name": "me"
    },
    "multiValueQueryStringParameters": {
        "name": ["me"]
    },
   "pathParameters": {
      "proxy": "hello/world"
   },
   "stageVariables": {
      "stageVariableName": "stageVariableValue"
   },
   "requestContext": {
      "accountId": "12345678912",
      "resourceId": "roq9wj",
      "stage": "testStage",
      "domainName": "gy415nuibc.execute-api.us-east-2.amazonaws.com",
      "domainPrefix": "y0ne18dixk",
      "requestId": "deef4878-7910-11e6-8f14-25afc3e9ae33",
      "protocol": "HTTP/1.1",
      "identity": {
         "cognitoIdentityPoolId": "theCognitoIdentityPoolId",
         "accountId": "theAccountId",
         "cognitoIdentityId": "theCognitoIdentityId",
         "caller": "theCaller",
            "apiKey": "theApiKey",
            "apiKeyId": "theApiKeyId",
            "accessKey": "ANEXAMPLEOFACCESSKEY",
         "sourceIp": "192.168.196.186",
         "cognitoAuthenticationType": "theCognitoAuthenticationType",
         "cognitoAuthenticationProvider": "theCognitoAuthenticationProvider",
         "userArn": "theUserArn",
         "userAgent": "PostmanRuntime/2.4.5",
         "user": "theUser"
      },
      "authorizer": {
         "principalId": "admin",
         "clientId": 1,
         "clientName": "Exata"
      },
      "resourcePath": "/{proxy+}",
      "httpMethod": "POST",
      "requestTime": "15/May/2020:06:01:09 +0000",
      "requestTimeEpoch": 1589522469693,
      "apiId": "gy415nuibc"
   },
   "body": "{\n  \"payload_version\":1,\n  \"notification_configuration_id\":\"nc-vxvQ5SpJJ23oAWLk\",\n  \"run_url\":\"https://app.terraform.io/app/mnsanfilippo/mnsanfilippo/runs/run-uyeKDPGKNpD9PxdC\",\n  \"run_id\":\"run-uyeKDPGKNpD9PxdC\",\n  \"run_message\":\"Queued manually using Terraform\",\n  \"run_created_at\":\"2021-05-30T16:18:00.000Z\",\n  \"run_created_by\":\"mnsanfilippo\",\n  \"workspace_id\":\"ws-WzbRhM4ojZXjRvoP\",\n  \"workspace_name\":\"mnsanfilippo\",\n  \"organization_name\":\"mnsanfilippo\",\n  \"notifications\":[\n    {\n      \"message\":\"Run Errored\",\n      \"trigger\":\"run:errored\",\n      \"run_status\":\"errored\",\n      \"run_updated_at\":\"2021-05-30T16:19:14.000Z\",\n      \"run_updated_by\":null\n    }\n  ]\n}"
}

The function is going to parse the body of the payload, searching for

workspace_id
workspace_name
organization_name
run_id

With that information, the function retrieves from Terraform Cloud the last tfstate of the workspace and will save it to S3.

To get all this working, this terraform module terraform-cloud-backup
will create and configure the resources needed.

module "example" {
  source = "https://github.com/mnsanfilippo/terraform-cloud-backup.git?ref=main"
  bucket_builds  = var.bucket_builds
  bucket_name    = var.bucket_name
  lambda_s3_key  = var.lambda_s3_key
  tf_token       = var.tf_token
  workspaces_ids = var.workspaces_ids
}

To get this up and running, we need a few stuff.

Terraform Cloud Token.
The build of the lambda in S3.
The name of the S3 bucket where to save the tfstates.
The list of the workspaces IDs to backup.

The variables should look like this:

Once that the module has been applied. There is a new notification in Terraform Cloud

And in S3 your tfstates are going to be saved as
bucket/organization/workspace/year/month/day/workspace-serial-stateID-runID.tfstate

This is all for today. In the following days, I will publish the Go-Tool to save all the tfstates on Terraform Cloud.

Testing IaC with Terratest and GitHub Actions

Pipo Sanfilippo — Tue, 12 Jan 2021 16:57:33 +0000

Index

Introduction
Creating a new AWS account for testing
What you are going to test
Project structure
Writing a terraform module
Using a terraform module
Testing the module
Testing within GitHub Actions
Using terragrunt

Introduction

Hello there!

A few months ago, I started to do some stuff in Go, and I fell in love with it, I wrote some scripts and the scripts are now in production, but I didn't like that they were only scripts, I tested them over and over, but manually of course, and it was very tedious, due to the lack of time, ignorance, and a lot of excuses, I postponed the tests of my scripts.

Last month, I started to do the AWS Solutions Architect Professional training with Adrian Cantrill, the best instructor in the world. I decided that I wanted to take it seriously. So, I rewrote my scripts now in a TDD way.

At first, I tried with LocalStack, which is great, but I encountered myself fighting most of the time with LocalStack that with my code. For example, you can create an SNS topic, publish a message into an SNS topic, but you can't subscribe to your email, and part of what should be tested to send an email.
So I found Terratest. With it you deploy real infrastructure, so that means that it has real costs.

Creating a new AWS account for testing

The first thing you have to do is create a new account for testing purposes. This is needed because we are going to use cloud-nuke to destroy everything that we had created for testing. You will sleep better if there are almost zero chances to destroy something that is in production.

What you are going to test

You are going to test the creation of an AWS Account. This will be your test account, where you are going to create and destroy resources without the worries of being in a production environment.

Project structure

├── terraform
│   ├── accounts
│   │   ├── main.tf
│   │   ├── outputs.tf
│   │   └── variables.tf
│   └── tests
│       ├── go.mod
│       ├── go.sum
│       └── organizations_accounts_test.go
├── terraform-live
│   ├── master
│   │   ├── account.hcl
│   │   └── us-east-1
│   │       ├── dev
│   │       │   ├── accounts
│   │       │   │   └── terragrunt.hcl
│   │       │   └── env.hcl
│   │       └── region.hcl
│   └── terragrunt.hcl
└── terraform-modules
    ├── README.md
    └── aws
        └── organization
            └── accounts
                ├── main.tf
                ├── outputs.tf
                └── variables.tf

terraform
In this repository exists the modules of terraform.
terraform-live
In this repository, you are going to set up the environment variables.
For example:
You need two accounts, one for testing and another one for production. Here is where you set up the different environments, but keeping the code DRY, there is no need to copy and paste the same source code and only change the names.
terraform-modules
In this repository is the construction of our modules, the resources that we need to create something and the definitions of how it should be created.
For example, you want that everyone that uses your module has to set up specific tags for the purpose, environment, owner, or unit business of the resources created.

Writing a terraform module

Use modules instead of resources. Using modules, the code keeps DRY, and is reusable. When you use the module, it is cleaner than all the resources needed. In this case, the module of an AWS account is pretty much the same that the resource, but in almost every other case, this is not going to happen.

#terraform-modules/aws/organization/accounts/main.tf
resource "aws_organizations_account" "account" {
  name  = var.account_name
  email = var.account_email
}

Using a terraform module

Now that you have your module for the account, you have to specify that source. You can have one repository for each module, or a repository with all the modules, to keep it simple. I decided to keep all the modules together.

#terraform/accounts/main.tf
module "account" {
  source = "git@github.com:mnsanfilippo/terraform-modules.git//aws/organization/accounts"
  account_name = var.account_name
  account_email = var.account_email
}

Testing the module

Testing Infrastructure as Code is not like testing other code. For example, in an app code, you write unit testing isolated from the real world. Testing Infrastructure as Code is testing how your resources behave in the real world.

You are going to test the creation of the account. In order to do that, you are going to check that the account exists. In the future, you can add more tests, like can I access correctly from the master account to the new account?.

The function TestOrganizationAccountCreation does what it says, verify the creation of an account. To test that, it retrieves the account of the organization with the account_id output from terraform, and then it checks if the email and the name are the same that are used to create the account.

This test has a problem, in the real world when you create an account, you can't delete it without completing the sign-up steps before. This is why the code has a "terraformPlan" boolean. If you want to create an account and test correctly this module, change the value to false. If you only want to make a plan of it, leave it in true.

Terratest is a library that is going to help you to test terraform code but is not going to do all the job, in this case, there is an auxiliary function (describeAccountById) to retrieve the account info from AWS.

package test

import (
    "context"
    "fmt"
    "github.com/aws/aws-sdk-go-v2/aws"
    "github.com/aws/aws-sdk-go-v2/config"
    "github.com/aws/aws-sdk-go-v2/service/organizations"
    "github.com/aws/aws-sdk-go-v2/service/organizations/types"
    "github.com/gruntwork-io/terratest/modules/random"
    "github.com/gruntwork-io/terratest/modules/terraform"
    "github.com/stretchr/testify/assert"
    "log"
    "testing"
)

var terraformPlan = true

func TestOrganizationAccountCreation(t *testing.T) {

    // Set up expected values to be checked later
    expectedAccountName := fmt.Sprintf("a4l-dev-%s", random.UniqueId())
    expectedEmail := fmt.Sprintf("mnsanfilippo+dev+%s@gmail.com", random.UniqueId())

    // Construct the terraform options with default retryable errors to handle the most common retryable errors in
    // terraform testing
    terraformOptions := terraform.WithDefaultRetryableErrors(t, &terraform.Options{

        // The path to where our Terraform code is located
        TerraformDir: "./../accounts/",

        //Variables to pass to our Terraform code using -var options
        Vars: map[string]interface{}{
            "account_name":  expectedAccountName,
            "account_email": expectedEmail,
        },

    })

    //At the end of the test, run "terraform destroy" to clean up any resources that were created
    // In this case, the account is not deleted, to be deleted you need to complete first the account sign-up steps
    //defer terraform.Destroy(t, terraformOptions)

    if terraformPlan {
        // Also, you can only do terraform plan
        terraform.Init(t, terraformOptions)
        terraform.Plan(t, terraformOptions)

    } else {
        // This will run `terraform init` and `terraform apply`and fail the test if there any errors
        terraform.InitAndApply(t, terraformOptions)

        // Look up the Organization Account by id
        accountId := terraform.Output(t, terraformOptions, "account_id")
        account := describeAccountById(accountId)

        assert.Equal(t, expectedEmail, *account.Email)
        assert.Equal(t, expectedAccountName, *account.Name)

        // I added this only to see if the account was created in the AWS console before it was deleted
        log.Println("Now you can go and check if the account was created in the AWS Console.\n" +
            "Remember, to delete the account, you first have to complete the sign-up process.")
    }
}

func describeAccountById(id string) *types.Account {

    cfg, err := config.LoadDefaultConfig(context.TODO())

    if err != nil {
        log.Fatal(err)
    }
    svc := organizations.NewFromConfig(cfg)

    req, err := svc.DescribeAccount(context.Background(), &organizations.DescribeAccountInput{AccountId: aws.String(id)})
    if err != nil {
        log.Println("Failed retrieving account by ID")
        log.Fatal(err)
    }

    return req.Account
}

You can test this simply by running

go test -v

Testing within GitHub Actions

To test the code before merging with production, you can add a workflow that validates the code every time that has a change in the repository.

Set up the following secrets in the repository

SSH_KEY
AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY
AWS_DEFAULT_REGION

name: Testing new terraform code
on:
  push:
    branches:
      - develop
  pull_request:
    branches:
      - develop
      -
jobs:
  test:
    runs-on: ubuntu-latest
    env:
      AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
      AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
      AWS_DEFAULT_REGION: ${{ secrets.AWS_DEFAULT_REGION }}

    steps:
    - uses: actions/checkout@v2
    - name: Set up Go
      uses: actions/setup-go@v2
      with:
        go-version: 1.15

    - name: 'Setup SSH Key'
      uses: webfactory/ssh-agent@v0.4.1
      with:
        ssh-private-key: ${{ secrets.SSH_KEY }}

    - name: Setup Dependencies
      working-directory: tests/
      run:  go get -v -t -d && go mod tidy

    - name: Test
      working-directory: tests/
      run: go test -v

Using Terragrunt

Terragrunt is a tool that helps to have separate environments while keeping the code DRY. Using Terragrunt, you avoid having multiple copies of "terraform/accounts/main.tf" that only differ in the environment names, or the instances types of your EC2 instances.

For example, you can specify the email and the account name that the account will have. In order to make another account, you copy this configuration and change the email and the account.

# terraform-live/master/us-east-1/dev/accounts/terragrunt.hcl
locals {
  # Automatically load environment-level variables
  environment_vars = read_terragrunt_config(find_in_parent_folders("env.hcl"))

  # Extract out common variables for reuse
  env = local.environment_vars.locals.environment
}

# Terragrunt will copy the Terraform configurations specified by the source parameter, along with any files in the
# working directory, into a temporary folder, and execute your Terraform commands in that folder.
terraform {
  source = "git@github.com:mnsanfilippo/terraform.git//accounts?ref=develop"
}

# Include all settings from the root terragrunt.hcl file
include {
  path = find_in_parent_folders()
}

# These are the variables we have to pass in to use the module specified in the terragrunt configuration above
inputs = {
  account_name = "dev"
  account_email = "mnsanfilippo+dev@gmail.com"
}

And that will be it for today. I hope that this guide can help you to start testing your code.