Introduction
Welcome to our latest blog post, where we delve into the world of Infrastructure as Code (IaC) using Terragrunt and Terraform, specifically focusing on their integration with Amazon Web Services (AWS). Whether you're a seasoned DevOps professional or new to cloud infrastructure, this post will guide you through the essentials of using these powerful tools in harmony with AWS.
Understanding Terragrunt and Terraform
"The cloud is someone else's computer" - Unknown
Imagine deploying a server that runs an NGINX application with only a few lines of code? This seemed impossible a couple of years ago when we were spending hours in Datacenters swapping hardware and fixing networking issues.
Terraform, an open-source infrastructure as code software tool, allows users to define and provision a cloud infrastructure using a high-level configuration language. It's a game-changer in the world of cloud computing, bringing simplicity and predictability to cloud resource management.
Terraform state
Terraform logs information about the resources it has created in a state file. This enables Terraform to know which resources are under its control and when to update and destroy them. The terraform state file, by default, is named terraform.tfstate and is held in the same directory where Terraform is run. It is created after running terraform apply.
The actual content of this file is a JSON formatted mapping of the resources defined in the configuration and those that exist in your infrastructure. When Terraform is run, it can then use this mapping to compare infrastructure to the code and make any adjustments as necessary.
Read more about the elements of Terraform architecture.
Setting Up Your Environment
If you use either Terraform or Terragrunt, you'll need to connect your IaC (Infrastructure as Code) repository to your Cloud Provider, in my case, AWS.
There are many ways of doing that, but after +8 years working with these solutions, you get to know a few tips and tricks.
My AWS account structure is as follow:
A good practice that I learnt over the years is that it can become very difficult and time consuming to manage dozens and/or hundreds of different state files in different AWS accounts. That's why I have been deploying a SharedServices
account where I host all the Terraform backends and resources that it needs in order to manage infrastructure in the cloud.
These are some of the Terraform resources I store in the SharedServices
account:
- DynamoDB table for state locking
- S3 bucket to store the state files
- IAM roles and permissions
- Alerts (constantly monitoring if any of the above resources are modified, deleted or even accessed)
That being said, this is how a Terraform/Terragrunt IaC repository hosted on Github will connect to AWS:
-
Terrateam: After getting into some really big issues when running Terragrunt with Github Actions, I decided to look for a better CI solution. Terrateam is my CI/CD tool of choice here.
- Unfortunately as of December 2023, they increased their price from USD $175 to USD $496 monthly. Me being an existing customer I still pay the old amount (thank God!)
- Alternatively, you can look at solutions like Atlantis or spacelift.
-
terraform-state-lock
: DynamoDB table for TF state locking -
devops-bucket
: S3 Bucket to store TF state files -
terraform-execution-role
: IAM Role deployed across ALL accounts. This is the Role that TF will assume when provisioning resources in AWS
If you're provisioning the above resources for the first time, you'll have to either configure Terraform to use specific AWS keys as you won't have OIDC
connection yet. In my case, I chose to have those pre-requesites resources in a CloudFormation template and deploy them with StackSets.
Deploying the pre-requesites
Github OIDC connection and role:
---
#---------------------------------------------------------------
# Deployed to the SharedServices Account
#---------------------------------------------------------------
# This template deploys the necessary resources for the Terraform Backend
AWSTemplateFormatVersion: "2010-09-09"
Description: Deploys the necessary resources for the Terraform Backend
Parameters:
GithubRoleName:
Type: String
Default: "github-oidc-role"
Description: "Name of the AWS IAM Role for Terraform assume"
Resources:
# Creates the Github OIDC Identity Provider
GithubOidcProvider:
Type: AWS::IAM::OIDCProvider
Properties:
Url: https://token.actions.githubusercontent.com
ClientIdList:
- sts.amazonaws.com
- https://github.com/github_org_name/*
ThumbprintList:
- 6938fd4d98bab03faadb97b34396831e3780aea1
- 1c58a3a8518e8759bf075b76b750d4f2df264fcd
# Creates the IAM Role that Github Actions will assume to deploy infrastrcuture
GithubActionsRole:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Version: "2012-10-17"
Statement:
- Effect: Allow
Principal:
Federated:
- !Sub 'arn:aws:iam::${AWS::AccountId}:oidc-provider/token.actions.githubusercontent.com'
Action:
- 'sts:AssumeRoleWithWebIdentity'
Condition:
StringEquals:
token.actions.githubusercontent.com:aud: "sts.amazonaws.com"
StringLike:
# https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_create_for-idp_oidc.html#idp_oidc_Create_GitHub
token.actions.githubusercontent.com:sub: "repo:github_org_name/*"
# https://aws.amazon.com/blogs/security/announcing-an-update-to-iam-role-trust-policy-behavior/
- Effect: Allow
Principal:
AWS:
- !Sub 'arn:aws:iam::${AWS::AccountId}:role/github-oidc-role'
Action:
- 'sts:AssumeRole'
Description: Role to provide Github access to AWS
ManagedPolicyArns:
- arn:aws:iam::aws:policy/AdministratorAccess
RoleName:
Ref: GithubRoleName
# 43200 = 12 hours
MaxSessionDuration: 43200
NOTE:
github_org_name
: Change this to your Github Organization name or check this article for more details.NOTE: Since we only want to deploy this CloudFormation template to the
SharedServices
account, there is no need for you to use StackSets.
---
#---------------------------------------------------------------
# Deployed to the Shared-Services account
#---------------------------------------------------------------
# This template deploys the necessary resources for the Terraform Backend
AWSTemplateFormatVersion: "2010-09-09"
Description: Deploys the necessary resources for the Terraform Backend
Parameters:
DynamoDBTableName:
Type: String
Default: "terraform-state-lock"
Description: "Name of the Terraform DynamoDB table"
TerraformS3BucketName:
Type: String
Default: "devops-bucket"
Description: "Name of the S3 Bucket for the Terraform state files"
TerraformBackendRoleName:
Type: String
Default: "terraform-backend-role"
Description: "Name of the role that users will assume for local TF development"
Resources:
# Creates the S3 bucket that will store the Terraform State files
TerraformS3BucketLogging:
# checkov:skip=CKV_AWS_18: This is a logging bucket
Type: AWS::S3::Bucket
Properties:
BucketName: !Join ["", [!Ref TerraformS3BucketName, "-logging"]]
PublicAccessBlockConfiguration:
BlockPublicAcls: true
BlockPublicPolicy: true
IgnorePublicAcls: true
RestrictPublicBuckets: true
BucketEncryption:
ServerSideEncryptionConfiguration:
- ServerSideEncryptionByDefault:
SSEAlgorithm: AES256
VersioningConfiguration:
Status: Enabled
Tags:
- Key: "Backup"
Value: "false"
- Key: "Environment"
Value: "SharedServices"
- Key: "Owner"
Value: "Terraform"
TerraformS3BucketLoggingBucketPolicy:
Type: AWS::S3::BucketPolicy
Properties:
Bucket: !Ref TerraformS3BucketLogging
PolicyDocument:
Version: 2012-10-17
Statement:
- Action:
- "s3:List*"
- "s3:Get*"
- "s3:PutObject"
- "s3:DeleteObject"
- "s3:PutEncryptionConfiguration"
- "s3:PutBucketPolicy"
Effect: Allow
Resource:
- !Sub arn:aws:s3:::${TerraformS3BucketLogging}
- !Sub arn:aws:s3:::${TerraformS3BucketLogging}/*
Principal:
AWS: !Sub '${AWS::AccountId}'
TerraformS3Bucket:
Type: AWS::S3::Bucket
Properties:
BucketName: !Ref TerraformS3BucketName
LoggingConfiguration:
DestinationBucketName: !Ref TerraformS3BucketLogging
LogFilePrefix: logs
PublicAccessBlockConfiguration:
BlockPublicAcls: true
BlockPublicPolicy: true
IgnorePublicAcls: true
RestrictPublicBuckets: true
BucketEncryption:
ServerSideEncryptionConfiguration:
- ServerSideEncryptionByDefault:
SSEAlgorithm: AES256
VersioningConfiguration:
Status: Enabled
Tags:
- Key: "Backup"
Value: "true"
- Key: "Environment"
Value: "SharedServices"
- Key: "Owner"
Value: "Terraform"
TerraformS3BucketPolicy:
Type: "AWS::S3::BucketPolicy"
Properties:
Bucket: !Ref TerraformS3Bucket
PolicyDocument:
Version: "2012-10-17"
Statement:
- Action:
- "s3:List*"
- "s3:Get*"
- "s3:PutObject"
- "s3:DeleteObject"
- "s3:PutEncryptionConfiguration"
- "s3:PutBucketPolicy"
Effect: "Allow"
Principal:
AWS:
# We allow only the backend role to access the bucket
- !Sub "arn:aws:iam::${AWS::AccountId}:role/terraform-backend-role"
Resource:
- !Sub arn:aws:s3:::${TerraformS3Bucket}
- !Sub arn:aws:s3:::${TerraformS3Bucket}/*
- Action:
- "s3:*"
Sid: "RootAccess"
Effect: Allow
Resource:
- !Sub arn:aws:s3:::${TerraformS3Bucket}
- !Sub arn:aws:s3:::${TerraformS3Bucket}/*
Principal:
AWS:
# So that we don't lock ourselves out of the bucket
# A specific IAM role/user should be used for this
- !Sub "arn:aws:iam::${AWS::AccountId}:root"
- Action:
- "s3:*"
Sid: "EnforcedTLS"
Effect: Deny
Resource:
- !Sub arn:aws:s3:::${TerraformS3Bucket}
- !Sub arn:aws:s3:::${TerraformS3Bucket}/*
Principal:
AWS: "*"
Condition:
Bool:
aws:SecureTransport: "false"
# Creates the DynamoDB Table for the TF State lock
TerraformDynamoDBTable:
Type: AWS::DynamoDB::Table
Properties:
TableName: !Ref DynamoDBTableName
AttributeDefinitions:
- AttributeName: "LockID"
AttributeType: "S"
DeletionProtectionEnabled: true
KeySchema:
- AttributeName: "LockID"
KeyType: HASH
ProvisionedThroughput:
ReadCapacityUnits: 5
WriteCapacityUnits: 5
PointInTimeRecoverySpecification:
PointInTimeRecoveryEnabled: true
SSESpecification:
SSEEnabled: true
KMSMasterKeyId: alias/aws/dynamodb
SSEType: KMS
Tags:
- Key: "Backup"
Value: "true"
- Key: "Environment"
Value: "SharedServices"
- Key: "Owner"
Value: "Terraform"
# AutoScaling rules for the DynamoDB Table
MyTableWriteCapacityScalableTarget:
Type: "AWS::ApplicationAutoScaling::ScalableTarget"
DependsOn: TerraformDynamoDBTable
Properties:
MaxCapacity: 20
MinCapacity: 5
ResourceId: !Sub table/${DynamoDBTableName}
RoleARN: !Sub arn:aws:iam::${AWS::AccountId}:role/aws-service-role/dynamodb.application-autoscaling.amazonaws.com/AWSServiceRoleForApplicationAutoScaling_DynamoDBTable
ScalableDimension: "dynamodb:table:WriteCapacityUnits"
ServiceNamespace: dynamodb
MyTableWriteScalingPolicy:
Type: "AWS::ApplicationAutoScaling::ScalingPolicy"
Properties:
PolicyName: WriteAutoScalingPolicy
PolicyType: TargetTrackingScaling
ScalingTargetId: !Ref MyTableWriteCapacityScalableTarget
TargetTrackingScalingPolicyConfiguration:
TargetValue: 70
ScaleInCooldown: 60
ScaleOutCooldown: 60
PredefinedMetricSpecification:
PredefinedMetricType: DynamoDBWriteCapacityUtilization
TerraformBackendRole:
#checkov:skip=CKV_AWS_60::Ensure IAM role allows only specific services or principals to assume it
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Version: "2012-10-17"
Statement:
# We give these roles below the ability to assume this role
# The reason why I give the terraform-execution-role permissions to assume this role is because of local development.
# When I run terraform locally, I want to be able to assume this role so that I can access the S3 bucket and DynamoDB table
- Sid: "AllowAllAccounts"
Effect: Allow
Principal:
AWS:
# dev
- arn:aws:iam::{dev_account_id}:role/terraform-execution-role
# staging
- arn:aws:iam::{staging_account_id}:role/terraform-execution-role
# production
- arn:aws:iam::{production_account_id}:role/terraform-execution-role
# Log Archive
- arn:aws:iam::{log-archive_account_id}:role/terraform-execution-role
# Audit
- arn:aws:iam::{audit_account_id}:role/terraform-execution-role
# Master
- arn:aws:iam::{master_account_id}:role/terraform-execution-role
# backup
- arn:aws:iam::{backup_account_id}:role/terraform-execution-role
# shared-services
- arn:aws:iam::{shared-services_account_id}:role/terraform-execution-role
- arn:aws:iam::{shared-services_account_id}:role/github-oidc-role
Action:
- 'sts:AssumeRole'
- Sid: "AllowGithubActions"
Effect: Allow
Principal:
Federated:
- !Sub 'arn:aws:iam::${AWS::AccountId}:oidc-provider/token.actions.githubusercontent.com'
Action:
- 'sts:AssumeRoleWithWebIdentity'
Condition:
StringEquals:
token.actions.githubusercontent.com:aud: "sts.amazonaws.com"
StringLike:
# https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_create_for-idp_oidc.html#idp_oidc_Create_GitHub
token.actions.githubusercontent.com:sub: "repo:github_org_name/*"
# This is needed as we use AWS SSO and the AWSReservedSSO_DevOps* roles are used by the Engineers
- Sid: "AllowSSOAWSAdministratorAccess"
Effect: Allow
Principal:
AWS: "*"
Action:
- "sts:AssumeRole"
Condition:
ArnEquals:
aws:PrincipalArn: "arn:aws:iam::*:role/aws-reserved/sso.amazonaws.com/*/AWSReservedSSO_DevOps*"
Description: Role to provide Terraform access to DynamoDB and S3
Policies:
- PolicyName: allows-access-to-s3-policy
PolicyDocument:
Version: "2012-10-17"
Statement:
- Sid: "AllowS3Access"
Effect: Allow
Action:
- "s3:ListBucket"
- "s3:GetObject"
- "s3:PutObject"
- "s3:DeleteObject"
Resource:
- !GetAtt TerraformS3Bucket.Arn
- !Join
- ""
- - "arn:aws:s3:::"
- !Ref TerraformS3Bucket
- /*
- PolicyName: allows-access-to-dynamodb-policy
PolicyDocument:
Version: "2012-10-17"
Statement:
- Sid: "AllowDynamoDBAccess"
Effect: Allow
Action:
- "dynamodb:GetItem"
- "dynamodb:DeleteItem"
- "dynamodb:PutItem"
- "dynamodb:DescribeTable"
Resource:
- !GetAtt TerraformDynamoDBTable.Arn
- !Join
- ""
- - !GetAtt TerraformDynamoDBTable.Arn
- /*
RoleName: !Ref TerraformBackendRoleName
NOTE:
github_org_name
: Change this to your Github Organization name or check this article for more details.NOTE: Since we only want to deploy this CloudFormation template to the
SharedServices
account, there is no need for you to use StackSets.NOTE: You might notice this across some of the resources
Tags: - Key: "Backup" Value: "true"
I have AWS Backups setup which will backup anything that has a tag of
Backup = true
.
Connecting to AWS
Now, let's connect your Terragrunt/Terraform setup with AWS. This process involves managing AWS credentials securely and ensuring your Terraform configurations can access these credentials. This section will include detailed steps and code snippets to guide you through this process, emphasizing security and best practices.
Terrateam configuration
NOTE: More information about Terrateam's configuration here.
.terrateam/config.yml
:
hooks:
all:
pre:
- type: oidc
provider: aws
role_arn: "${TERRAFORM_GITHUB_OIDC_ROLE_ARN}"
session_name: "terrateam"
duration: 14400
audience: "sts.amazonaws.com"
TERRAFORM_GITHUB_OIDC_ROLE_ARN
is defined in my Github Secrets and it has the following value: arn:aws:iam::{shared-services_account_id}:role/github-oidc-role
A simple Terragrunt repository structure:
.
├── environments
│ └── test
│ ├── ap-southeast-2
│ │ ├── vpc
│ │ │ └── terragrunt.hcl
└── terragrunt.hcl
NOTE: More information about the
terragrunt.hcl
file can be found in this example repository.
My terragrunt.hcl
file will have the backend and provider's configuration defined:
locals {
# Automatically load region-level variables
region_vars = read_terragrunt_config(find_in_parent_folders("region.hcl"))
# Automatically load environment-level variables`
environment_vars = read_terragrunt_config(find_in_parent_folders("env.hcl"))
# Extract the variables we need for easy access
account_name = local.environment_vars.locals.account_name
account_id = local.environment_vars.locals.aws_account_id
aws_region = local.region_vars.locals.aws_region
# This is the S3 bucket where the Terraform State Files will be stored
remote_state_bucket = "devops-bucket"
# This is the DynamoDB table where Terraform will add the locking status
dynamodb_table = "terraform-state-lock"
# IAM Role for Terraform backend to assume
terraform_backend_role = "arn:aws:iam::{shared-services_account_id}:role/terraform-backend-role"
environment_path = replace(path_relative_to_include(), "environments/", "")
# https://github.com/hashicorp/terraform/releases
terraform_version = "latest"
# https://github.com/gruntwork-io/terragrunt/releases
terragrunt_version = "latest"
}
# Generate an AWS provider block
generate "provider" {
path = "provider.tf"
if_exists = "overwrite_terragrunt"
contents = <<EOF
provider "aws" {
region = "${local.aws_region}"
# Only these AWS Account IDs may be operated on by this template
allowed_account_ids = ["${local.account_id}"]
# Assume role information
assume_role {
role_arn = "arn:aws:iam::${local.account_id}:role/terraform-execution-role"
session_name = "terragrunt-${local.account_id}-${local.aws_region}"
}
}
EOF
}
# Configure Terragrunt to automatically store tfstate files in an S3 bucket
remote_state {
backend = "s3"
config = {
encrypt = true
bucket = local.remote_state_bucket
key = "tfstate/${path_relative_to_include()}/terraform.tfstate"
region = "ap-southeast-2"
dynamodb_table = local.dynamodb_table
role_arn = local.terraform_backend_role
}
generate = {
path = "backend.tf"
if_exists = "overwrite_terragrunt"
}
}
An example of how the state files will be stored:
With that, you should be able to create the VPC resources in the Test account. :)
Conclusion
To wrap up, we'll summarize the key points covered and discuss the importance of integrating Terragrunt and Terraform with AWS for efficient cloud infrastructure management. Our goal is to equip you with the knowledge to enhance your infrastructure management practices.
Call to Action
We encourage you to share your thoughts, experiences, or questions in the comments below. And if you found this post helpful, don’t forget to follow our blog for more insightful content on cloud infrastructure and DevOps.
Top comments (7)
@lpossamai Thanks for sharing! Do you use run-all or somehow detect changed terragrunt modules and only run apply for them?
Thanks! I do not run
run-all
as it is not recommended. The CI tool that I use detects the changes and runsplan/apply
for those directories only. An example can be found here. Cheers!Terrateam does not support terragrunt dependencies. For example, if you have two terragrunt modules:
and you make changes to both, terrateam won't first apply vpc and only once completed mysql.
Perhaps you solve this by making two PRs - vpc and only after mysql?
You're right, sorry I misunderstood your question. They don't support dependencies. They say it's in their roadmap.
Yes, I use two PRs.
I had a call with them last week, it is in the backlog but not high priority, if an enterprise client shows up and this is a must-have requirement for them, then it would help speed it up.
That's right. It is in their backlog... I'll try to push for it on my side, as a customer.
toil.kitemaker.co/sharing/items/1d...
Love the article! I would use Brainboard though...