Terragrunt with Terraform in AWS

Intro

Terragrunt is a utility that can be used along with Terraform, to alleviate some of the challenges that come with using Terraform at scale. Today we will see Terragrunt in action when using it to create infra on AWS.

The main benefit of Terragrunt is to reduce the repetition in code and keep the code DRY (Don't Repeat Yourself principle).
When you have just one instance or one account of AWS, this problem may not occur. But as soon as you start managing multiple accounts or environments of AWS, you start copying code from one place to another and there are chances of manual mistakes and redundant code. Terragrunt helps with this.

Prerequisites
To use Terragrunt with Terraform on AWS, you will need to install the following:

1. AWS CLI
2. Terraform
3. Terragrunt

To use other functionalities we will see, we need:

1. TFlint (For before-hooks that run linting)
   tflint

2. SSH - Add the SSH key to your GitHub account and make sure you can pull and push from GitHub using SSH. (for pulling Terraform at run-time from a git repo) Also add github to the known hosts:
   

   (host=github.com; ssh-keyscan -H $host; for ip in $(dig @8.8.8.8 github.com +short); do ssh-keyscan -H $host,$ip; ssh-keyscan -H $ip; done) 2> /dev/null >> .ssh/known_hosts

You can use this git repo to go through the demo examples:
Terragrunt Demo

This code will create 2 EC2 instances, one using local code and one pulling the code from a remote git repo.

Workflow of using Terragrunt

1. Install the prerequisites above.
2. Create a file called terragrunt.hcl which holds the terragrunt configuration.
3. Instead of terraform commands, run terragrunt commands: terraform init --> terragrunt init

terraform plan --> terragrunt plan

terraform apply --> terragrunt apply

terraform destroy --> terragrunt destroy

Keep your code DRY with Terragrunt

Backend configuration

Backend configuration in Terraform does not allow for variables. This means people copy the configuration from one environment and use it in another one and manually change something. Which can lead to errors. With terragrunt, we keep it parameterized.

#Keep your backend DRY
remote_state {
  backend = "s3"
  generate = {
    path      = "backend.tf"
    if_exists = "overwrite_terragrunt"
  }
  config = {
    bucket = "tf-state-manum-0202041706"
    key = "${path_relative_to_include()}/terraform.tfstate"
    region         = "us-east-1"
    encrypt        = true
    dynamodb_table = "tf-lock-table"
  }
}

This above code goes in terragrunt.hcl, and means the state will be kept in different paths for each module. Please note that the key is a variable, changing for each module.

Below image shows the bucket for the backend, where different module state is kept in different prefixes.

Keep provider configuration DRY

Note the below code in terragrunt.hcl, It uses assume_role to assume a specific role in the AWS for terraform.

#Keep your provider DRY
generate "provider" {
  path = "provider.tf"
  if_exists = "overwrite_terragrunt"
  contents = <<EOF
provider "aws" {
  region = "us-east-1"
  profile = "admin"
  assume_role {
    role_arn = "arn:aws:iam::644107485976:role/github_actions_role"
  }
}
EOF
}

Keep your CLI arguments dry

In many cases, you have variables to be passed to Terraform which changes for each account and each environment. These may be kept in different files and passed to Terraform with the CLI argument -var-file

This is cumbersome. Terragrunt avoids that and injects the variables using the below code in terragrunt.hcl
Variables from account.tfvars and region.tfvars will be injected into the modules where you are running terragrunt and it will be made available to them with the syntax var.xyz

#Keep your CLI DRY
terraform {
  extra_arguments "common_vars" {
    commands = ["plan", "apply"]

    required_var_files = [
     "${get_parent_terragrunt_dir()}/account.tfvars",
     "${get_parent_terragrunt_dir()}/region.tfvars"
    ]
  }
}

Before, After, Error Hooks

Terragrunt allows you to hook into the lifecycle of terraform runs, and you can run custom code either before, after, or in case of an error in terraform.

Below code will print "start Terraform" at the start of each module terraform using echo command, using before_hook.

It will print "Finished running Terraform" using after_hook.

error_hook will execute in case of an error.

    before_hook "before_hook" {
    commands     = ["apply", "plan"]
    execute      = ["echo","Start Terraform"]
  }

  after_hook "after_hook" {
    commands     = ["apply", "plan"]
    execute      = ["echo", "Finished running Terraform"]
    run_on_error = true
  }
    error_hook "import_resource" {
    commands  = ["apply"]
    execute   = ["echo", "Error Hook executed"]
    on_errors = [
      ".*",
    ]
  }

This brings up an interesting possibility. We could run some linters or other code validators using before_hook, like below:

before_hook "before_hook" {
commands     = ["apply", "plan"]
execute      = ["tflint"]

}

This will look for the .tflint.hcl within the path where you are running the code.

Here's a minimal .tflint.hcl

config {
  module = true
}

// Plugin configuration
plugin "aws" {
  enabled = true
  version = "0.29.0"
  source  = "github.com/terraform-linters/tflint-ruleset-aws"
}

plugin "terraform" {
  enabled = true
  preset  = "recommended"
  version = "0.4.0"
  source  = "github.com/terraform-linters/tflint-ruleset-terraform"
}

RUN-ALL command

Terragrunt brings run-all command to terraform. Say you have 100 modules in a folder and want to create all that infra. Either you have to run the terraform plan and apply repeatedly 100 times manually or through a script. With terragrunt you just do:

terragrunt run-all plan 
terragrunt run-all apply

Note: you can use skip argument to skip some of the modules.

Different Versions of code in Different Environments

One powerful feature of terragrunt is to pull versioned code out of code repositories and run that terraform code. Say you have production running on stable code, and on dev, you want to try out something. a 1.1 version. You can have that version of code tagged 1.1 on GitHub (or another repository) and then pull that code at the run time. This is done by code like below:

terraform {
  source = "git::[e-mail]:manumaan/Terragrunt_Demo.git//compute?ref=v1.1.0"
}

The above code will pull the code with the 1.1.0 tag from the GitHub repo https://www.github.com/manumaan/Terragrunt_Demo and run that in that module.

Some Extra Features
Terragrunt will automatically retry any transient errors. (What is transient is defined by terragrunt). It will automatically run init if init has not been done in that path.

Debugging Terragrunt
You can specify log level during commands to get debug logs. Different levels are:

panic
fatal
error
warn
info --> This is the default
debug
trace

Terraform with Multiple Accounts/Environments

Using Terraform with multiple accounts or environments brings its challenges. Some approaches seen are:

1. Use different git branches/repos
2. Use Terraform Cloud

Git branches/repos don't add to the cost or learning complexity but do not help with repetitive code that has to be managed. Variable management is also not available.

Terraform Cloud provides all the features you want, but there is a cost. Integration with policy-as-code is a benefit. Instead of terragrunt.hcl files, here you will create workspaces and projects. Here's my article on Terraform Cloud that gives all the details: Terraform Cloud with AWS

Terragrunt takes a middle path, where it provides some of the functionalities you need for this use case, for no cost, while keeping the code repetition-free.

Below is a comparison of different approaches:

Hope this was helpful!