Index
- Introduction
- Creating a new AWS account for testing
- What you are going to test
- Project structure
- Writing a terraform module
- Using a terraform module
- Testing the module
- Testing within GitHub Actions
- Using terragrunt
Introduction
Hello there!
A few months ago, I started to do some stuff in Go, and I fell in love with it, I wrote some scripts and the scripts are now in production, but I didn't like that they were only scripts, I tested them over and over, but manually of course, and it was very tedious, due to the lack of time, ignorance, and a lot of excuses, I postponed the tests of my scripts.
Last month, I started to do the AWS Solutions Architect Professional training with Adrian Cantrill, the best instructor in the world. I decided that I wanted to take it seriously. So, I rewrote my scripts now in a TDD way.
At first, I tried with LocalStack, which is great, but I encountered myself fighting most of the time with LocalStack that with my code. For example, you can create an SNS topic, publish a message into an SNS topic, but you can't subscribe to your email, and part of what should be tested to send an email.
So I found Terratest. With it you deploy real infrastructure, so that means that it has real costs.
Creating a new AWS account for testing
The first thing you have to do is create a new account for testing purposes. This is needed because we are going to use cloud-nuke to destroy everything that we had created for testing. You will sleep better if there are almost zero chances to destroy something that is in production.
What you are going to test
You are going to test the creation of an AWS Account. This will be your test account, where you are going to create and destroy resources without the worries of being in a production environment.
Project structure
├── terraform
│ ├── accounts
│ │ ├── main.tf
│ │ ├── outputs.tf
│ │ └── variables.tf
│ └── tests
│ ├── go.mod
│ ├── go.sum
│ └── organizations_accounts_test.go
├── terraform-live
│ ├── master
│ │ ├── account.hcl
│ │ └── us-east-1
│ │ ├── dev
│ │ │ ├── accounts
│ │ │ │ └── terragrunt.hcl
│ │ │ └── env.hcl
│ │ └── region.hcl
│ └── terragrunt.hcl
└── terraform-modules
├── README.md
└── aws
└── organization
└── accounts
├── main.tf
├── outputs.tf
└── variables.tf
terraform
In this repository exists the modules of terraform.terraform-live
In this repository, you are going to set up the environment variables.
For example:
You need two accounts, one for testing and another one for production. Here is where you set up the different environments, but keeping the code DRY, there is no need to copy and paste the same source code and only change the names.terraform-modules
In this repository is the construction of our modules, the resources that we need to create something and the definitions of how it should be created.
For example, you want that everyone that uses your module has to set up specific tags for the purpose, environment, owner, or unit business of the resources created.
Writing a terraform module
Use modules instead of resources. Using modules, the code keeps DRY, and is reusable. When you use the module, it is cleaner than all the resources needed. In this case, the module of an AWS account is pretty much the same that the resource, but in almost every other case, this is not going to happen.
#terraform-modules/aws/organization/accounts/main.tf
resource "aws_organizations_account" "account" {
name = var.account_name
email = var.account_email
}
Using a terraform module
Now that you have your module for the account, you have to specify that source. You can have one repository for each module, or a repository with all the modules, to keep it simple. I decided to keep all the modules together.
#terraform/accounts/main.tf
module "account" {
source = "git@github.com:mnsanfilippo/terraform-modules.git//aws/organization/accounts"
account_name = var.account_name
account_email = var.account_email
}
Testing the module
Testing Infrastructure as Code is not like testing other code. For example, in an app code, you write unit testing isolated from the real world. Testing Infrastructure as Code is testing how your resources behave in the real world.
You are going to test the creation of the account. In order to do that, you are going to check that the account exists. In the future, you can add more tests, like can I access correctly from the master account to the new account?.
The function TestOrganizationAccountCreation does what it says, verify the creation of an account. To test that, it retrieves the account of the organization with the account_id output from terraform, and then it checks if the email and the name are the same that are used to create the account.
This test has a problem, in the real world when you create an account, you can't delete it without completing the sign-up steps before. This is why the code has a "terraformPlan" boolean. If you want to create an account and test correctly this module, change the value to false. If you only want to make a plan of it, leave it in true.
Terratest is a library that is going to help you to test terraform code but is not going to do all the job, in this case, there is an auxiliary function (describeAccountById) to retrieve the account info from AWS.
package test
import (
"context"
"fmt"
"github.com/aws/aws-sdk-go-v2/aws"
"github.com/aws/aws-sdk-go-v2/config"
"github.com/aws/aws-sdk-go-v2/service/organizations"
"github.com/aws/aws-sdk-go-v2/service/organizations/types"
"github.com/gruntwork-io/terratest/modules/random"
"github.com/gruntwork-io/terratest/modules/terraform"
"github.com/stretchr/testify/assert"
"log"
"testing"
)
var terraformPlan = true
func TestOrganizationAccountCreation(t *testing.T) {
// Set up expected values to be checked later
expectedAccountName := fmt.Sprintf("a4l-dev-%s", random.UniqueId())
expectedEmail := fmt.Sprintf("mnsanfilippo+dev+%s@gmail.com", random.UniqueId())
// Construct the terraform options with default retryable errors to handle the most common retryable errors in
// terraform testing
terraformOptions := terraform.WithDefaultRetryableErrors(t, &terraform.Options{
// The path to where our Terraform code is located
TerraformDir: "./../accounts/",
//Variables to pass to our Terraform code using -var options
Vars: map[string]interface{}{
"account_name": expectedAccountName,
"account_email": expectedEmail,
},
})
//At the end of the test, run "terraform destroy" to clean up any resources that were created
// In this case, the account is not deleted, to be deleted you need to complete first the account sign-up steps
//defer terraform.Destroy(t, terraformOptions)
if terraformPlan {
// Also, you can only do terraform plan
terraform.Init(t, terraformOptions)
terraform.Plan(t, terraformOptions)
} else {
// This will run `terraform init` and `terraform apply`and fail the test if there any errors
terraform.InitAndApply(t, terraformOptions)
// Look up the Organization Account by id
accountId := terraform.Output(t, terraformOptions, "account_id")
account := describeAccountById(accountId)
assert.Equal(t, expectedEmail, *account.Email)
assert.Equal(t, expectedAccountName, *account.Name)
// I added this only to see if the account was created in the AWS console before it was deleted
log.Println("Now you can go and check if the account was created in the AWS Console.\n" +
"Remember, to delete the account, you first have to complete the sign-up process.")
}
}
func describeAccountById(id string) *types.Account {
cfg, err := config.LoadDefaultConfig(context.TODO())
if err != nil {
log.Fatal(err)
}
svc := organizations.NewFromConfig(cfg)
req, err := svc.DescribeAccount(context.Background(), &organizations.DescribeAccountInput{AccountId: aws.String(id)})
if err != nil {
log.Println("Failed retrieving account by ID")
log.Fatal(err)
}
return req.Account
}
You can test this simply by running
go test -v
Testing within GitHub Actions
To test the code before merging with production, you can add a workflow that validates the code every time that has a change in the repository.
Set up the following secrets in the repository
- SSH_KEY
- AWS_ACCESS_KEY_ID
- AWS_SECRET_ACCESS_KEY
- AWS_DEFAULT_REGION
name: Testing new terraform code
on:
push:
branches:
- develop
pull_request:
branches:
- develop
-
jobs:
test:
runs-on: ubuntu-latest
env:
AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
AWS_DEFAULT_REGION: ${{ secrets.AWS_DEFAULT_REGION }}
steps:
- uses: actions/checkout@v2
- name: Set up Go
uses: actions/setup-go@v2
with:
go-version: 1.15
- name: 'Setup SSH Key'
uses: webfactory/ssh-agent@v0.4.1
with:
ssh-private-key: ${{ secrets.SSH_KEY }}
- name: Setup Dependencies
working-directory: tests/
run: go get -v -t -d && go mod tidy
- name: Test
working-directory: tests/
run: go test -v
Using Terragrunt
Terragrunt is a tool that helps to have separate environments while keeping the code DRY. Using Terragrunt, you avoid having multiple copies of "terraform/accounts/main.tf" that only differ in the environment names, or the instances types of your EC2 instances.
For example, you can specify the email and the account name that the account will have. In order to make another account, you copy this configuration and change the email and the account.
# terraform-live/master/us-east-1/dev/accounts/terragrunt.hcl
locals {
# Automatically load environment-level variables
environment_vars = read_terragrunt_config(find_in_parent_folders("env.hcl"))
# Extract out common variables for reuse
env = local.environment_vars.locals.environment
}
# Terragrunt will copy the Terraform configurations specified by the source parameter, along with any files in the
# working directory, into a temporary folder, and execute your Terraform commands in that folder.
terraform {
source = "git@github.com:mnsanfilippo/terraform.git//accounts?ref=develop"
}
# Include all settings from the root terragrunt.hcl file
include {
path = find_in_parent_folders()
}
# These are the variables we have to pass in to use the module specified in the terragrunt configuration above
inputs = {
account_name = "dev"
account_email = "mnsanfilippo+dev@gmail.com"
}
And that will be it for today. I hope that this guide can help you to start testing your code.
Top comments (2)
Very nice article!
I didn't understand, when do you use this action 'webfactory/ssh-agent'? in general, the ssh key.
Thanks for sharing :)
Thank you!
I used 'webfactory/ssh-agent, because GitHub Actions fails trying to download the repository