Seamless Cloud Infrastructure: Integrating Terragrunt and Terraform with AWS

Introduction

Welcome to our latest blog post, where we delve into the world of Infrastructure as Code (IaC) using Terragrunt and Terraform, specifically focusing on their integration with Amazon Web Services (AWS). Whether you're a seasoned DevOps professional or new to cloud infrastructure, this post will guide you through the essentials of using these powerful tools in harmony with AWS.

Understanding Terragrunt and Terraform

"The cloud is someone else's computer" - Unknown

Imagine deploying a server that runs an NGINX application with only a few lines of code? This seemed impossible a couple of years ago when we were spending hours in Datacenters swapping hardware and fixing networking issues.

Terraform, an open-source infrastructure as code software tool, allows users to define and provision a cloud infrastructure using a high-level configuration language. It's a game-changer in the world of cloud computing, bringing simplicity and predictability to cloud resource management.

Terraform state

Terraform logs information about the resources it has created in a state file. This enables Terraform to know which resources are under its control and when to update and destroy them. The terraform state file, by default, is named terraform.tfstate and is held in the same directory where Terraform is run. It is created after running terraform apply.

The actual content of this file is a JSON formatted mapping of the resources defined in the configuration and those that exist in your infrastructure. When Terraform is run, it can then use this mapping to compare infrastructure to the code and make any adjustments as necessary.

Read more about the elements of Terraform architecture.

Setting Up Your Environment

If you use either Terraform or Terragrunt, you'll need to connect your IaC (Infrastructure as Code) repository to your Cloud Provider, in my case, AWS.

There are many ways of doing that, but after +8 years working with these solutions, you get to know a few tips and tricks.

My AWS account structure is as follow:

A good practice that I learnt over the years is that it can become very difficult and time consuming to manage dozens and/or hundreds of different state files in different AWS accounts. That's why I have been deploying a SharedServices account where I host all the Terraform backends and resources that it needs in order to manage infrastructure in the cloud.

These are some of the Terraform resources I store in the SharedServices account:

• DynamoDB table for state locking
• S3 bucket to store the state files
• IAM roles and permissions
• Alerts (constantly monitoring if any of the above resources are modified, deleted or even accessed)

That being said, this is how a Terraform/Terragrunt IaC repository hosted on Github will connect to AWS:

• Terrateam: After getting into some really big issues when running Terragrunt with Github Actions, I decided to look for a better CI solution. Terrateam is my CI/CD tool of choice here.
  • Unfortunately as of December 2023, they increased their price from USD $175 to USD $496 monthly. Me being an existing customer I still pay the old amount (thank God!)
  • Alternatively, you can look at solutions like Atlantis or spacelift.
• terraform-state-lock: DynamoDB table for TF state locking
• devops-bucket: S3 Bucket to store TF state files
• terraform-execution-role: IAM Role deployed across ALL accounts. This is the Role that TF will assume when provisioning resources in AWS

If you're provisioning the above resources for the first time, you'll have to either configure Terraform to use specific AWS keys as you won't have OIDC connection yet. In my case, I chose to have those pre-requesites resources in a CloudFormation template and deploy them with StackSets.

Deploying the pre-requesites

Github OIDC connection and role:

---
#---------------------------------------------------------------
# Deployed to the SharedServices Account
#---------------------------------------------------------------
# This template deploys the necessary resources for the Terraform Backend
AWSTemplateFormatVersion: "2010-09-09"
Description: Deploys the necessary resources for the Terraform Backend
Parameters:
  GithubRoleName:
    Type: String
    Default: "github-oidc-role"
    Description: "Name of the AWS IAM Role for Terraform assume"

Resources:
  # Creates the Github OIDC Identity Provider
  GithubOidcProvider:
    Type: AWS::IAM::OIDCProvider
    Properties:
      Url: https://token.actions.githubusercontent.com
      ClientIdList:
        - sts.amazonaws.com
        - https://github.com/github_org_name/*
      ThumbprintList:
        - 6938fd4d98bab03faadb97b34396831e3780aea1
        - 1c58a3a8518e8759bf075b76b750d4f2df264fcd

  # Creates the IAM Role that Github Actions will assume to deploy infrastrcuture
  GithubActionsRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: "2012-10-17"
        Statement:
          - Effect: Allow
            Principal:
              Federated:
                - !Sub 'arn:aws:iam::${AWS::AccountId}:oidc-provider/token.actions.githubusercontent.com'
            Action:
              - 'sts:AssumeRoleWithWebIdentity'
            Condition:
              StringEquals:
                token.actions.githubusercontent.com:aud: "sts.amazonaws.com"
              StringLike:
                # https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_create_for-idp_oidc.html#idp_oidc_Create_GitHub
                token.actions.githubusercontent.com:sub: "repo:github_org_name/*"
          # https://aws.amazon.com/blogs/security/announcing-an-update-to-iam-role-trust-policy-behavior/
          - Effect: Allow
            Principal:
              AWS:
                - !Sub 'arn:aws:iam::${AWS::AccountId}:role/github-oidc-role'
            Action:
              - 'sts:AssumeRole'
      Description: Role to provide Github access to AWS
      ManagedPolicyArns:
        - arn:aws:iam::aws:policy/AdministratorAccess
      RoleName:
        Ref: GithubRoleName
      # 43200 = 12 hours
      MaxSessionDuration: 43200

NOTE: github_org_name: Change this to your Github Organization name or check this article for more details.

NOTE: Since we only want to deploy this CloudFormation template to the SharedServices account, there is no need for you to use StackSets.

Terraform backend:

---
#---------------------------------------------------------------
# Deployed to the Shared-Services account
#---------------------------------------------------------------
# This template deploys the necessary resources for the Terraform Backend
AWSTemplateFormatVersion: "2010-09-09"
Description: Deploys the necessary resources for the Terraform Backend
Parameters:
  DynamoDBTableName:
    Type: String
    Default: "terraform-state-lock"
    Description: "Name of the Terraform DynamoDB table"
  TerraformS3BucketName:
    Type: String
    Default: "devops-bucket"
    Description: "Name of the S3 Bucket for the Terraform state files"
  TerraformBackendRoleName:
    Type: String
    Default: "terraform-backend-role"
    Description: "Name of the role that users will assume for local TF development"

Resources:
  # Creates the S3 bucket that will store the Terraform State files
  TerraformS3BucketLogging:
    # checkov:skip=CKV_AWS_18: This is a logging bucket
    Type: AWS::S3::Bucket
    Properties:
      BucketName: !Join ["", [!Ref TerraformS3BucketName, "-logging"]]
      PublicAccessBlockConfiguration:
        BlockPublicAcls: true
        BlockPublicPolicy: true
        IgnorePublicAcls: true
        RestrictPublicBuckets: true
      BucketEncryption:
        ServerSideEncryptionConfiguration:
          - ServerSideEncryptionByDefault:
              SSEAlgorithm: AES256
      VersioningConfiguration:
        Status: Enabled
      Tags:
        - Key: "Backup"
          Value: "false"
        - Key: "Environment"
          Value: "SharedServices"
        - Key: "Owner"
          Value: "Terraform"

  TerraformS3BucketLoggingBucketPolicy:
    Type: AWS::S3::BucketPolicy
    Properties:
      Bucket: !Ref TerraformS3BucketLogging
      PolicyDocument:
        Version: 2012-10-17
        Statement:
          - Action:
              - "s3:List*"
              - "s3:Get*"
              - "s3:PutObject"
              - "s3:DeleteObject"
              - "s3:PutEncryptionConfiguration"
              - "s3:PutBucketPolicy"
            Effect: Allow
            Resource:
              - !Sub arn:aws:s3:::${TerraformS3BucketLogging}
              - !Sub arn:aws:s3:::${TerraformS3BucketLogging}/*
            Principal:
                AWS: !Sub '${AWS::AccountId}'

  TerraformS3Bucket:
    Type: AWS::S3::Bucket
    Properties:
      BucketName: !Ref TerraformS3BucketName
      LoggingConfiguration:
        DestinationBucketName: !Ref TerraformS3BucketLogging
        LogFilePrefix: logs
      PublicAccessBlockConfiguration:
        BlockPublicAcls: true
        BlockPublicPolicy: true
        IgnorePublicAcls: true
        RestrictPublicBuckets: true
      BucketEncryption:
        ServerSideEncryptionConfiguration:
          - ServerSideEncryptionByDefault:
              SSEAlgorithm: AES256
      VersioningConfiguration:
        Status: Enabled
      Tags:
        - Key: "Backup"
          Value: "true"
        - Key: "Environment"
          Value: "SharedServices"
        - Key: "Owner"
          Value: "Terraform"

  TerraformS3BucketPolicy:
    Type: "AWS::S3::BucketPolicy"
    Properties:
      Bucket: !Ref TerraformS3Bucket
      PolicyDocument:
        Version: "2012-10-17"
        Statement:
          - Action:
              - "s3:List*"
              - "s3:Get*"
              - "s3:PutObject"
              - "s3:DeleteObject"
              - "s3:PutEncryptionConfiguration"
              - "s3:PutBucketPolicy"
            Effect: "Allow"
            Principal:
              AWS:
                # We allow only the backend role to access the bucket
                - !Sub "arn:aws:iam::${AWS::AccountId}:role/terraform-backend-role"
            Resource:
              - !Sub arn:aws:s3:::${TerraformS3Bucket}
              - !Sub arn:aws:s3:::${TerraformS3Bucket}/*
          - Action:
              - "s3:*"
            Sid: "RootAccess"
            Effect: Allow
            Resource:
              - !Sub arn:aws:s3:::${TerraformS3Bucket}
              - !Sub arn:aws:s3:::${TerraformS3Bucket}/*
            Principal:
              AWS:
                # So that we don't lock ourselves out of the bucket
                # A specific IAM role/user should be used for this
                - !Sub "arn:aws:iam::${AWS::AccountId}:root"
          - Action:
              - "s3:*"
            Sid: "EnforcedTLS"
            Effect: Deny
            Resource:
              - !Sub arn:aws:s3:::${TerraformS3Bucket}
              - !Sub arn:aws:s3:::${TerraformS3Bucket}/*
            Principal:
              AWS: "*"
            Condition:
              Bool:
                aws:SecureTransport: "false"

  # Creates the DynamoDB Table for the TF State lock
  TerraformDynamoDBTable:
    Type: AWS::DynamoDB::Table
    Properties:
      TableName: !Ref DynamoDBTableName
      AttributeDefinitions:
        - AttributeName: "LockID"
          AttributeType: "S"
      DeletionProtectionEnabled: true
      KeySchema:
        - AttributeName: "LockID"
          KeyType: HASH
      ProvisionedThroughput:
        ReadCapacityUnits: 5
        WriteCapacityUnits: 5
      PointInTimeRecoverySpecification:
        PointInTimeRecoveryEnabled: true
      SSESpecification:
        SSEEnabled: true
        KMSMasterKeyId: alias/aws/dynamodb
        SSEType: KMS
      Tags:
        - Key: "Backup"
          Value: "true"
        - Key: "Environment"
          Value: "SharedServices"
        - Key: "Owner"
          Value: "Terraform"

  # AutoScaling rules for the DynamoDB Table
  MyTableWriteCapacityScalableTarget:
    Type: "AWS::ApplicationAutoScaling::ScalableTarget"
    DependsOn: TerraformDynamoDBTable
    Properties:
      MaxCapacity: 20
      MinCapacity: 5
      ResourceId: !Sub table/${DynamoDBTableName}
      RoleARN: !Sub arn:aws:iam::${AWS::AccountId}:role/aws-service-role/dynamodb.application-autoscaling.amazonaws.com/AWSServiceRoleForApplicationAutoScaling_DynamoDBTable
      ScalableDimension: "dynamodb:table:WriteCapacityUnits"
      ServiceNamespace: dynamodb

  MyTableWriteScalingPolicy:
    Type: "AWS::ApplicationAutoScaling::ScalingPolicy"
    Properties:
      PolicyName: WriteAutoScalingPolicy
      PolicyType: TargetTrackingScaling
      ScalingTargetId: !Ref MyTableWriteCapacityScalableTarget
      TargetTrackingScalingPolicyConfiguration:
        TargetValue: 70
        ScaleInCooldown: 60
        ScaleOutCooldown: 60
        PredefinedMetricSpecification:
          PredefinedMetricType: DynamoDBWriteCapacityUtilization

  TerraformBackendRole:
    #checkov:skip=CKV_AWS_60::Ensure IAM role allows only specific services or principals to assume it
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: "2012-10-17"
        Statement:
          # We give these roles below the ability to assume this role
          # The reason why I give the terraform-execution-role permissions to assume this role is because of local development.
          # When I run terraform locally, I want to be able to assume this role so that I can access the S3 bucket and DynamoDB table
          - Sid: "AllowAllAccounts"
            Effect: Allow
            Principal:
              AWS:
                # dev
                - arn:aws:iam::{dev_account_id}:role/terraform-execution-role
                # staging
                - arn:aws:iam::{staging_account_id}:role/terraform-execution-role
                # production
                - arn:aws:iam::{production_account_id}:role/terraform-execution-role
                # Log Archive
                - arn:aws:iam::{log-archive_account_id}:role/terraform-execution-role
                # Audit
                - arn:aws:iam::{audit_account_id}:role/terraform-execution-role
                # Master
                - arn:aws:iam::{master_account_id}:role/terraform-execution-role
                # backup
                - arn:aws:iam::{backup_account_id}:role/terraform-execution-role
                # shared-services
                - arn:aws:iam::{shared-services_account_id}:role/terraform-execution-role
                - arn:aws:iam::{shared-services_account_id}:role/github-oidc-role
            Action:
              - 'sts:AssumeRole'
          - Sid: "AllowGithubActions"
            Effect: Allow
            Principal:
              Federated:
                - !Sub 'arn:aws:iam::${AWS::AccountId}:oidc-provider/token.actions.githubusercontent.com'
            Action:
              - 'sts:AssumeRoleWithWebIdentity'
            Condition:
              StringEquals:
                token.actions.githubusercontent.com:aud: "sts.amazonaws.com"
              StringLike:
                # https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_create_for-idp_oidc.html#idp_oidc_Create_GitHub
                token.actions.githubusercontent.com:sub: "repo:github_org_name/*"
          # This is needed as we use AWS SSO and the AWSReservedSSO_DevOps* roles are used by the Engineers
          - Sid: "AllowSSOAWSAdministratorAccess"
            Effect: Allow
            Principal:
              AWS: "*"
            Action:
              - "sts:AssumeRole"
            Condition:
              ArnEquals:
                aws:PrincipalArn: "arn:aws:iam::*:role/aws-reserved/sso.amazonaws.com/*/AWSReservedSSO_DevOps*"

      Description: Role to provide Terraform access to DynamoDB and S3
      Policies:
        - PolicyName: allows-access-to-s3-policy
          PolicyDocument:
            Version: "2012-10-17"
            Statement:
              - Sid: "AllowS3Access"
                Effect: Allow
                Action:
                  - "s3:ListBucket"
                  - "s3:GetObject"
                  - "s3:PutObject"
                  - "s3:DeleteObject"
                Resource:
                  - !GetAtt TerraformS3Bucket.Arn
                  - !Join
                    - ""
                    - - "arn:aws:s3:::"
                      - !Ref TerraformS3Bucket
                      - /*
        - PolicyName: allows-access-to-dynamodb-policy
          PolicyDocument:
            Version: "2012-10-17"
            Statement:
              - Sid: "AllowDynamoDBAccess"
                Effect: Allow
                Action:
                  - "dynamodb:GetItem"
                  - "dynamodb:DeleteItem"
                  - "dynamodb:PutItem"
                  - "dynamodb:DescribeTable"
                Resource:
                  - !GetAtt TerraformDynamoDBTable.Arn
                  - !Join
                    - ""
                    - - !GetAtt TerraformDynamoDBTable.Arn
                      - /*
      RoleName: !Ref TerraformBackendRoleName

NOTE: github_org_name: Change this to your Github Organization name or check this article for more details.

NOTE: Since we only want to deploy this CloudFormation template to the SharedServices account, there is no need for you to use StackSets.

NOTE: You might notice this across some of the resources

Tags:
       - Key: "Backup"
         Value: "true"

I have AWS Backups setup which will backup anything that has a tag of Backup = true.

Connecting to AWS

Now, let's connect your Terragrunt/Terraform setup with AWS. This process involves managing AWS credentials securely and ensuring your Terraform configurations can access these credentials. This section will include detailed steps and code snippets to guide you through this process, emphasizing security and best practices.

Terrateam configuration

NOTE: More information about Terrateam's configuration here.

.terrateam/config.yml:

hooks:
  all:
    pre:
      - type: oidc
        provider: aws
        role_arn: "${TERRAFORM_GITHUB_OIDC_ROLE_ARN}"
        session_name: "terrateam"
        duration: 14400
        audience: "sts.amazonaws.com"

TERRAFORM_GITHUB_OIDC_ROLE_ARN is defined in my Github Secrets and it has the following value: arn:aws:iam::{shared-services_account_id}:role/github-oidc-role

A simple Terragrunt repository structure:

.
├── environments
│   └── test
│       ├── ap-southeast-2
│       │   ├── vpc
│       │   │   └── terragrunt.hcl
└── terragrunt.hcl

NOTE: More information about the terragrunt.hcl file can be found in this example repository.

My terragrunt.hcl file will have the backend and provider's configuration defined:

locals {
  # Automatically load region-level variables
  region_vars = read_terragrunt_config(find_in_parent_folders("region.hcl"))

  # Automatically load environment-level variables`
  environment_vars = read_terragrunt_config(find_in_parent_folders("env.hcl"))

  # Extract the variables we need for easy access
  account_name = local.environment_vars.locals.account_name
  account_id   = local.environment_vars.locals.aws_account_id
  aws_region   = local.region_vars.locals.aws_region

  # This is the S3 bucket where the Terraform State Files will be stored
  remote_state_bucket = "devops-bucket"
  # This is the DynamoDB table where Terraform will add the locking status
  dynamodb_table = "terraform-state-lock"

  # IAM Role for Terraform backend to assume
  terraform_backend_role = "arn:aws:iam::{shared-services_account_id}:role/terraform-backend-role"

  environment_path = replace(path_relative_to_include(), "environments/", "")

  # https://github.com/hashicorp/terraform/releases
  terraform_version = "latest"
  # https://github.com/gruntwork-io/terragrunt/releases
  terragrunt_version = "latest"
}

# Generate an AWS provider block
generate "provider" {
  path      = "provider.tf"
  if_exists = "overwrite_terragrunt"
  contents  = <<EOF
provider "aws" {
  region = "${local.aws_region}"

  # Only these AWS Account IDs may be operated on by this template
  allowed_account_ids = ["${local.account_id}"]

  # Assume role information
    assume_role {
        role_arn     = "arn:aws:iam::${local.account_id}:role/terraform-execution-role"
        session_name = "terragrunt-${local.account_id}-${local.aws_region}"
    }
}
EOF
}

# Configure Terragrunt to automatically store tfstate files in an S3 bucket
remote_state {
  backend = "s3"
  config = {
    encrypt        = true
    bucket         = local.remote_state_bucket
    key            = "tfstate/${path_relative_to_include()}/terraform.tfstate"
    region         = "ap-southeast-2"
    dynamodb_table = local.dynamodb_table
    role_arn       = local.terraform_backend_role
  }
  generate = {
    path      = "backend.tf"
    if_exists = "overwrite_terragrunt"
  }
}

An example of how the state files will be stored:

With that, you should be able to create the VPC resources in the Test account. :)

Conclusion

To wrap up, we'll summarize the key points covered and discuss the importance of integrating Terragrunt and Terraform with AWS for efficient cloud infrastructure management. Our goal is to equip you with the knowledge to enhance your infrastructure management practices.

Call to Action

We encourage you to share your thoughts, experiences, or questions in the comments below. And if you found this post helpful, don’t forget to follow our blog for more insightful content on cloud infrastructure and DevOps.

Comments

[Mike Tyson of the Cloud]

Love the article! I would use Brainboard though...