Terraform at Scale: How We Manage 500+ Resources Across 8 AWS Accounts Without Losing Our Minds

November 2026 | ~18 min read

The terraform destroy That Took Down Staging

It was 3:47 PM on a Thursday. An engineer ran terraform destroy to clean up a sandbox VPC. Except they were in the wrong Terraform workspace. Staging was gone — 47 resources deleted in 90 seconds. The RDS instance. The EKS cluster. The VPN connections. All of it.

Recovery took 6 hours. Not because the infrastructure was hard to rebuild — we had the Terraform code. But because nobody could figure out what the staging state file actually contained. The state was a single 50MB monolith covering every resource across every environment. Running terraform plan took 15 minutes. Finding which resources belonged to staging vs. production vs. shared-services required reading through 12,000 lines of state JSON.

The real problem wasn't the accidental destroy. It was everything that made it possible:

• One monolithic state file covering 500+ resources across 8 AWS accounts
• No CI/CD for Terraform — every engineer applied from their laptop
• Constant state lock conflicts — 6 engineers fighting for the same DynamoDB lock 12 times a week
• No blast radius controls — a single terraform apply could touch any resource in any account
• No drift detection — console changes piled up silently until the next plan showed 40 unexpected diffs
• 15-minute plan times — so engineers stopped running plans before applying

We rebuilt the entire Terraform workflow in 4 weeks. Here's everything.

The Numbers

┌──────────────────────────────────────────────────────────────────────┐
│                    TERRAFORM AT SCALE RESULTS                        │
├───────────────────────────────────────┬────────────┬─────────────────┤
│ Metric                                │ Before     │ After           │
├───────────────────────────────────────┼────────────┼─────────────────┤
│ Resources managed                     │ 500+       │ 500+            │
│ AWS accounts                          │ 8          │ 8               │
│ State files                           │ 1 (50MB)   │ 24 (<3MB each)  │
│ Plan time                             │ 15 min     │ 45 sec          │
│ State lock conflicts/week             │ 12         │ 0               │
│ Drift incidents/quarter               │ 8          │ 0               │
│ Time to apply a change                │ 45 min     │ 8 min           │
│ Blast radius (max resources/apply)    │ 500+       │ ~30             │
│ Accidental destroys (6 month window)  │ 2          │ 0               │
│ Engineer onboarding time              │ 2 weeks    │ 2 hours         │
│ Engineers applying from laptops       │ 6          │ 0               │
└───────────────────────────────────────┴────────────┴─────────────────┘

Zero additional AWS cost. This was purely an operational improvement — better tooling, better structure, better workflows.

Table of Contents

• The Monolith Problem
• State Architecture
• Multi-Account Strategy
• Module Composition Patterns
• CI/CD for Terraform
• Blast Radius Controls
• Drift Detection and Reconciliation
• Developer Experience
• Lessons Learned
• Action Items

The Monolith Problem

Here's what our Terraform looked like before:

infrastructure/
├── main.tf           # 2,400 lines
├── variables.tf      # 800 lines
├── outputs.tf        # 400 lines
├── terraform.tfstate # 50MB, 12,000 lines
└── terraform.tfvars  # 200 lines, workspace-switched

One directory. One state file. Every resource — VPCs, subnets, security groups, RDS instances, EKS clusters, S3 buckets, IAM roles, CloudFront distributions, Lambda functions — all in the same Terraform root. Workspaces for environments.

The symptoms were predictable:

15-minute plans. Every terraform plan refreshed 500+ resources across 8 AWS accounts. Engineers started skipping plans and running terraform apply directly. "It was fine last time" is not a deployment strategy.

Constant state locks. Six engineers, one DynamoDB lock. Someone runs a plan, goes to lunch, forgets to cancel. Everyone else is blocked. We had a Slack channel called #terraform-lock where people asked "who has the lock?" twelve times a week.

50MB state file. Every terraform plan downloaded 50MB from S3, parsed 12,000 lines of JSON, made 500+ API calls to check resource status, and uploaded the result back. The state file contained secrets (RDS passwords, API keys) that every engineer could read.

Maximum blast radius. A single terraform apply could modify any resource in any account. There was no isolation between the VPC that routes production traffic and the S3 bucket that stores dev logs. One bad variable change could cascade.

The root cause was simple: one state file = one blast radius = maximum risk.

┌──────────────────────────────────────────────────────────────────┐
│                     THE MONOLITH STATE                            │
│                                                                   │
│  ┌─────────┐ ┌─────────┐ ┌─────────┐ ┌─────────┐               │
│  │   VPC   │ │   EKS   │ │   RDS   │ │   IAM   │               │
│  │ (prod)  │ │ (prod)  │ │ (prod)  │ │ (global)│               │
│  └─────────┘ └─────────┘ └─────────┘ └─────────┘               │
│  ┌─────────┐ ┌─────────┐ ┌─────────┐ ┌─────────┐               │
│  │   VPC   │ │   EKS   │ │   RDS   │ │   S3    │               │
│  │(staging)│ │(staging)│ │(staging)│ │  (all)  │               │
│  └─────────┘ └─────────┘ └─────────┘ └─────────┘               │
│  ┌─────────┐ ┌─────────┐ ┌─────────┐ ┌─────────┐               │
│  │ Lambda  │ │CloudFrnt│ │  Route53│ │   ...   │               │
│  │  (all)  │ │  (all)  │ │  (all)  │ │  (all)  │   500+ total  │
│  └─────────┘ └─────────┘ └─────────┘ └─────────┘               │
│                                                                   │
│  ONE state lock. ONE blast radius. ONE bad day.                   │
└──────────────────────────────────────────────────────────────────┘

The fix wasn't workspaces (we'd tried that — same state structure, different instances). It was splitting the state by domain and environment.

State Architecture

We split the monolith into 24 state files organized by domain and environment:

infrastructure/
├── networking/
│   ├── prod/
│   │   ├── main.tf
│   │   ├── variables.tf
│   │   └── backend.tf      # → s3://tf-state/networking/prod/terraform.tfstate
│   └── staging/
│       ├── main.tf
│       ├── variables.tf
│       └── backend.tf      # → s3://tf-state/networking/staging/terraform.tfstate
├── eks/
│   ├── prod/
│   └── staging/
├── data/
│   ├── prod/               # RDS, ElastiCache, OpenSearch
│   └── staging/
├── apps/
│   ├── payments-prod/
│   ├── payments-staging/
│   ├── platform-prod/
│   └── platform-staging/
├── security/
│   └── global/             # IAM roles, KMS keys, GuardDuty
├── dns/
│   └── global/             # Route53 zones, records
└── modules/                # Shared modules (versioned via git tags)
    ├── vpc/
    ├── eks-cluster/
    ├── rds-instance/
    └── service-deployment/

Each leaf directory is an independent Terraform root with its own state file, its own lock, and its own blast radius.

The backend configuration uses a consistent pattern:

terraform {
  backend "s3" {
    bucket         = "company-terraform-state"
    key            = "networking/prod/terraform.tfstate"
    region         = "us-east-1"
    dynamodb_table = "terraform-locks"
    encrypt        = true
  }
}

The state bucket itself is managed by a bootstrap root that's applied once and rarely touched:

resource "aws_s3_bucket" "terraform_state" {
  bucket = "company-terraform-state"
}

resource "aws_s3_bucket_versioning" "terraform_state" {
  bucket = aws_s3_bucket.terraform_state.id
  versioning_configuration {
    status = "Enabled"
  }
}

resource "aws_s3_bucket_server_side_encryption_configuration" "terraform_state" {
  bucket = aws_s3_bucket.terraform_state.id
  rule {
    apply_server_side_encryption_by_default {
      sse_algorithm = "aws:kms"
    }
  }
}

resource "aws_s3_bucket_public_access_block" "terraform_state" {
  bucket                  = aws_s3_bucket.terraform_state.id
  block_public_acls       = true
  block_public_policy     = true
  ignore_public_acls      = true
  restrict_public_buckets = true
}

resource "aws_dynamodb_table" "terraform_locks" {
  name         = "terraform-locks"
  billing_mode = "PAY_PER_REQUEST"
  hash_key     = "LockID"

  attribute {
    name = "LockID"
    type = "S"
  }
}

Cross-state references use terraform_remote_state data sources or SSM Parameter Store. We prefer SSM because it decouples state files — a networking change doesn't require re-planning the EKS root:

# In networking/prod — publish VPC ID
resource "aws_ssm_parameter" "vpc_id" {
  name  = "/infrastructure/networking/prod/vpc_id"
  type  = "String"
  value = module.vpc.vpc_id
}

# In eks/prod — consume VPC ID
data "aws_ssm_parameter" "vpc_id" {
  name = "/infrastructure/networking/prod/vpc_id"
}

module "eks" {
  source = "../../modules/eks-cluster"
  vpc_id = data.aws_ssm_parameter.vpc_id.value
  # ...
}

State migration from the monolith was the hardest part. We used terraform state mv to move resources one domain at a time, starting with the lowest-risk domain (DNS):

# Move Route53 resources to the new dns/global state
terraform state mv \
  -state=terraform.tfstate \
  -state-out=dns/global/terraform.tfstate \
  'aws_route53_zone.primary' \
  'aws_route53_zone.primary'

terraform state mv \
  -state=terraform.tfstate \
  -state-out=dns/global/terraform.tfstate \
  'aws_route53_record.apex' \
  'aws_route53_record.apex'

We migrated one domain per day over two weeks, running terraform plan in both the source and destination after each move to confirm zero diff.

Multi-Account Strategy

Eight AWS accounts, each with a dedicated purpose:

┌──────────────────────────────────────────────────────────────┐
│                    AWS ORGANIZATIONS                          │
│                                                               │
│  ┌─────────────┐                                             │
│  │  Management  │  ← Organization root, billing, SCPs        │
│  └──────┬──────┘                                             │
│         │                                                     │
│    ┌────┴─────────────────────────────────┐                  │
│    │                                       │                  │
│  ┌─┴──────────┐  ┌────────────┐  ┌───────┴──────┐          │
│  │  Security   │  │  Shared    │  │  Workloads   │          │
│  │  Account    │  │  Services  │  │  OU          │          │
│  │             │  │            │  │              │          │
│  │ GuardDuty   │  │ ECR        │  │ ┌──────────┐│          │
│  │ Config      │  │ Terraform  │  │ │   Prod   ││          │
│  │ CloudTrail  │  │ State      │  │ ├──────────┤│          │
│  │ IAM IdC     │  │ CI/CD      │  │ │ Staging  ││          │
│  └────────────┘  │ Artifacts  │  │ ├──────────┤│          │
│                   └────────────┘  │ │   Dev    ││          │
│                                    │ ├──────────┤│          │
│                                    │ │ Sandbox  ││          │
│                                    │ ├──────────┤│          │
│                                    │ │   Data   ││          │
│                                    │ └──────────┘│          │
│                                    └─────────────┘          │
└──────────────────────────────────────────────────────────────┘

Terraform runs in the shared-services account and assumes roles into target accounts:

locals {
  account_ids = {
    management      = "111111111111"
    security        = "222222222222"
    shared_services = "333333333333"
    prod            = "444444444444"
    staging         = "555555555555"
    dev             = "666666666666"
    sandbox         = "777777777777"
    data            = "888888888888"
  }
}

provider "aws" {
  region = "us-east-1"

  assume_role {
    role_arn = "arn:aws:iam::${local.account_ids[var.environment]}:role/TerraformExecutionRole"
  }

  default_tags {
    tags = {
      ManagedBy   = "terraform"
      Environment = var.environment
      Domain      = var.domain
    }
  }
}

The TerraformExecutionRole in each account trusts the shared-services account and has scoped permissions — not AdministratorAccess:

resource "aws_iam_role" "terraform_execution" {
  name = "TerraformExecutionRole"

  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Effect    = "Allow"
      Principal = { AWS = "arn:aws:iam::${local.account_ids.shared_services}:root" }
      Action    = "sts:AssumeRole"
      Condition = {
        StringEquals = {
          "aws:PrincipalTag/team" = "platform"
        }
      }
    }]
  })
}

resource "aws_iam_role_policy_attachment" "terraform_execution" {
  role       = aws_iam_role.terraform_execution.name
  policy_arn = "arn:aws:iam::aws:policy/PowerUserAccess"
}

We use PowerUserAccess instead of AdministratorAccess because Terraform shouldn't manage IAM policies in workload accounts — that's the security account's job.

Module Composition Patterns

We use a three-tier module architecture:

┌─────────────────────────────────────────────────────────────┐
│  Tier 3: Composition Roots (environment-specific)           │
│  infrastructure/eks/prod/main.tf                            │
│  infrastructure/apps/payments-prod/main.tf                  │
│  These call tier-2 modules with environment-specific values │
├─────────────────────────────────────────────────────────────┤
│  Tier 2: Service Modules (business logic)                   │
│  modules/eks-cluster/         modules/service-deployment/   │
│  modules/rds-instance/        modules/static-site/          │
│  These compose tier-1 resources into business units          │
├─────────────────────────────────────────────────────────────┤
│  Tier 1: Infrastructure Resources (AWS primitives)          │
│  aws_eks_cluster              aws_rds_cluster               │
│  aws_vpc                      aws_s3_bucket                 │
│  Raw Terraform resources — no abstraction                    │
└─────────────────────────────────────────────────────────────┘

Module interface design: Inputs should be business concepts, not AWS implementation details. Bad: instance_class, engine_version, allocated_storage. Good: size (small/medium/large), high_availability (bool), environment (string).

# modules/rds-instance/variables.tf

variable "name" {
  type        = string
  description = "Database name (used for identifier prefix and tags)"
}

variable "size" {
  type        = string
  description = "T-shirt size: small, medium, large, xlarge"
  validation {
    condition     = contains(["small", "medium", "large", "xlarge"], var.size)
    error_message = "Size must be small, medium, large, or xlarge."
  }
}

variable "high_availability" {
  type        = bool
  description = "Enable Multi-AZ deployment"
  default     = false
}

variable "backup_retention_days" {
  type        = number
  description = "Number of days to retain automated backups"
  default     = 7
}

# modules/rds-instance/main.tf

locals {
  size_map = {
    small  = { instance_class = "db.t4g.medium",  storage = 50,  iops = 3000 }
    medium = { instance_class = "db.r6g.large",   storage = 100, iops = 6000 }
    large  = { instance_class = "db.r6g.xlarge",  storage = 500, iops = 12000 }
    xlarge = { instance_class = "db.r6g.2xlarge", storage = 1000, iops = 20000 }
  }

  config = local.size_map[var.size]
}

resource "aws_db_instance" "this" {
  identifier_prefix          = "${var.name}-"
  instance_class             = local.config.instance_class
  allocated_storage          = local.config.storage
  iops                       = local.config.iops
  storage_type               = "gp3"
  engine                     = "postgres"
  engine_version             = "16.4"
  multi_az                   = var.high_availability
  backup_retention_period    = var.backup_retention_days
  deletion_protection        = var.high_availability
  skip_final_snapshot        = !var.high_availability
  final_snapshot_identifier  = var.high_availability ? "${var.name}-final" : null
  copy_tags_to_snapshot      = true

  tags = {
    Name = var.name
  }
}

Versioning: Modules are versioned via git tags. Composition roots pin to specific versions:

# infrastructure/data/prod/main.tf

module "payments_db" {
  source = "git::https://github.com/company/terraform-modules.git//rds-instance?ref=v2.3.1"

  name              = "payments"
  size              = "large"
  high_availability = true
}

Never use ref=main in production. A module change in main shouldn't automatically propagate to every environment. Upgrade versions deliberately: staging first, wait a week, then production.

Anti-patterns we eliminated:

• Modules calling modules calling modules (max depth: 2)
• Passing 40+ variables through a module that just forwards them
• count and for_each on modules (prefer separate module calls for clarity)
• Local path module sources in production (impossible to version or roll back)

CI/CD for Terraform

The single biggest improvement. We went from "apply from laptops" to "nobody touches terraform apply directly."

┌────────────────────────────────────────────────────────────┐
│                    TERRAFORM CI/CD FLOW                      │
│                                                              │
│  Engineer                                                    │
│     │                                                        │
│     ├── Push branch                                          │
│     │     │                                                  │
│     │     ▼                                                  │
│     │   GitHub Actions: terraform plan                       │
│     │     │                                                  │
│     │     ▼                                                  │
│     │   Plan output posted as PR comment                     │
│     │     │                                                  │
│     ├── Open PR, request review                              │
│     │     │                                                  │
│     │     ▼                                                  │
│     │   Reviewer approves (reviews plan output)              │
│     │     │                                                  │
│     ├── Merge to main                                        │
│     │     │                                                  │
│     │     ▼                                                  │
│     │   GitHub Actions: terraform apply                      │
│     │     │                                                  │
│     │     ▼                                                  │
│     │   Slack notification with apply result                 │
│     │                                                        │
│  Nobody runs terraform apply locally. Ever.                  │
└────────────────────────────────────────────────────────────┘

The GitHub Actions workflow detects which state files changed and only plans/applies those:

# .github/workflows/terraform.yml
name: Terraform

on:
  pull_request:
    paths:
      - 'infrastructure/**'
  push:
    branches: [main]
    paths:
      - 'infrastructure/**'

permissions:
  id-token: write
  contents: read
  pull-requests: write

jobs:
  detect-changes:
    runs-on: ubuntu-latest
    outputs:
      directories: ${{ steps.changed.outputs.directories }}
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0
      - id: changed
        run: |
          dirs=$(git diff --name-only ${{ github.event.before }} ${{ github.sha }} \
            | grep '^infrastructure/' \
            | grep -v '^infrastructure/modules/' \
            | xargs -I{} dirname {} \
            | sort -u \
            | jq -R -s -c 'split("\n") | map(select(length > 0))')
          echo "directories=$dirs" >> "$GITHUB_OUTPUT"

  plan:
    needs: detect-changes
    if: github.event_name == 'pull_request'
    runs-on: ubuntu-latest
    strategy:
      matrix:
        directory: ${{ fromJson(needs.detect-changes.outputs.directories) }}
    steps:
      - uses: actions/checkout@v4

      - uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: arn:aws:iam::333333333333:role/GitHubActionsTerraform
          aws-region: us-east-1

      - uses: hashicorp/setup-terraform@v3
        with:
          terraform_version: 1.9.0

      - name: Terraform Init
        working-directory: ${{ matrix.directory }}
        run: terraform init -input=false

      - name: Terraform Plan
        id: plan
        working-directory: ${{ matrix.directory }}
        run: terraform plan -input=false -no-color -out=tfplan
        continue-on-error: true

      - name: Comment Plan on PR
        uses: actions/github-script@v7
        with:
          script: |
            const output = `#### Terraform Plan: \`${{ matrix.directory }}\`
            \`\`\`
            ${{ steps.plan.outputs.stdout }}
            \`\`\`
            *Plan: ${{ steps.plan.outcome }}*`;
            github.rest.issues.createComment({
              issue_number: context.issue.number,
              owner: context.repo.owner,
              repo: context.repo.repo,
              body: output.substring(0, 65536)
            });

      - name: Fail on Plan Error
        if: steps.plan.outcome == 'failure'
        run: exit 1

  apply:
    needs: detect-changes
    if: github.ref == 'refs/heads/main' && github.event_name == 'push'
    runs-on: ubuntu-latest
    strategy:
      max-parallel: 1
      matrix:
        directory: ${{ fromJson(needs.detect-changes.outputs.directories) }}
    steps:
      - uses: actions/checkout@v4

      - uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: arn:aws:iam::333333333333:role/GitHubActionsTerraform
          aws-region: us-east-1

      - uses: hashicorp/setup-terraform@v3
        with:
          terraform_version: 1.9.0

      - name: Terraform Init
        working-directory: ${{ matrix.directory }}
        run: terraform init -input=false

      - name: Terraform Apply
        working-directory: ${{ matrix.directory }}
        run: terraform apply -input=false -auto-approve

OIDC federation — no static AWS keys in GitHub Secrets:

resource "aws_iam_openid_connect_provider" "github_actions" {
  url = "https://token.actions.githubusercontent.com"

  client_id_list = ["sts.amazonaws.com"]

  thumbprint_list = ["6938fd4d98bab03faadb97b34396831e3780aea1"]
}

resource "aws_iam_role" "github_actions_terraform" {
  name = "GitHubActionsTerraform"

  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Effect = "Allow"
      Principal = {
        Federated = aws_iam_openid_connect_provider.github_actions.arn
      }
      Action = "sts:AssumeRoleWithWebIdentity"
      Condition = {
        StringEquals = {
          "token.actions.githubusercontent.com:aud" = "sts.amazonaws.com"
        }
        StringLike = {
          "token.actions.githubusercontent.com:sub" = "repo:company/infrastructure:*"
        }
      }
    }]
  })
}

Blast Radius Controls

State splitting handles the biggest risk, but we added three more layers:

1. Lifecycle prevent_destroy on critical resources:

resource "aws_db_instance" "production" {
  # ... configuration ...

  lifecycle {
    prevent_destroy = true
  }
}

resource "aws_s3_bucket" "data_lake" {
  bucket = "company-data-lake-prod"

  lifecycle {
    prevent_destroy = true
  }
}

If someone accidentally adds a change that would destroy these resources, terraform plan fails instead of showing a destroy action.

2. OPA policies via Conftest:

We run Open Policy Agent checks on the plan output before apply:

# policy/terraform.rego

package terraform

deny[msg] {
  resource := input.resource_changes[_]
  resource.change.actions[_] == "delete"
  resource.address == "aws_db_instance.production"
  msg := sprintf("BLOCKED: Cannot destroy production database %s", [resource.address])
}

deny[msg] {
  resource := input.resource_changes[_]
  resource.type == "aws_security_group_rule"
  resource.change.after.cidr_blocks[_] == "0.0.0.0/0"
  resource.change.after.type == "ingress"
  msg := sprintf("BLOCKED: Security group %s allows ingress from 0.0.0.0/0", [resource.address])
}

deny[msg] {
  resource := input.resource_changes[_]
  resource.type == "aws_s3_bucket"
  not resource.change.after.server_side_encryption_configuration
  msg := sprintf("BLOCKED: S3 bucket %s missing encryption configuration", [resource.address])
}

The CI pipeline runs conftest test on the plan JSON output:

      - name: Policy Check
        working-directory: ${{ matrix.directory }}
        run: |
          terraform show -json tfplan > plan.json
          conftest test plan.json --policy ../../policy/

3. Infracost for plan-time cost estimation:

Every PR gets a cost estimate comment so reviewers can catch runaway spend before merge:

      - name: Infracost
        uses: infracost/actions/setup@v3
        with:
          api-key: ${{ secrets.INFRACOST_API_KEY }}

      - name: Generate Cost Estimate
        run: |
          infracost breakdown \
            --path=${{ matrix.directory }} \
            --format=json \
            --out-file=/tmp/infracost.json

      - name: Post Cost Comment
        run: |
          infracost comment github \
            --path=/tmp/infracost.json \
            --repo=${{ github.repository }} \
            --pull-request=${{ github.event.pull_request.number }} \
            --github-token=${{ secrets.GITHUB_TOKEN }}

Drift Detection and Reconciliation

Console changes are the silent killer. Someone tweaks a security group in the AWS console during an incident. Someone enables a feature flag in RDS settings. Someone changes a tag for a one-off report. None of it goes through Terraform. The state slowly diverges from reality.

We run nightly drift detection across all 24 state files:

# .github/workflows/drift-detection.yml
name: Drift Detection

on:
  schedule:
    - cron: '0 6 * * *'  # 6 AM UTC daily
  workflow_dispatch:

jobs:
  detect-drift:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        directory:
          - infrastructure/networking/prod
          - infrastructure/networking/staging
          - infrastructure/eks/prod
          - infrastructure/eks/staging
          - infrastructure/data/prod
          - infrastructure/data/staging
          - infrastructure/security/global
          - infrastructure/dns/global
          # ... all 24 directories
    steps:
      - uses: actions/checkout@v4

      - uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: arn:aws:iam::333333333333:role/GitHubActionsTerraform
          aws-region: us-east-1

      - uses: hashicorp/setup-terraform@v3

      - name: Terraform Init
        working-directory: ${{ matrix.directory }}
        run: terraform init -input=false

      - name: Check for Drift
        id: drift
        working-directory: ${{ matrix.directory }}
        run: |
          set +e
          terraform plan -input=false -detailed-exitcode -no-color 2>&1 | tee /tmp/plan.txt
          EXIT_CODE=$?
          echo "exit_code=$EXIT_CODE" >> "$GITHUB_OUTPUT"
          if [ $EXIT_CODE -eq 2 ]; then
            echo "drift=true" >> "$GITHUB_OUTPUT"
          else
            echo "drift=false" >> "$GITHUB_OUTPUT"
          fi

      - name: Alert on Drift
        if: steps.drift.outputs.drift == 'true'
        run: |
          PLAN=$(cat /tmp/plan.txt | tail -50)
          curl -X POST "${{ secrets.SLACK_WEBHOOK }}" \
            -H 'Content-Type: application/json' \
            -d "{
              \"text\": \":warning: Terraform drift detected in \`${{ matrix.directory }}\`\",
              \"blocks\": [{
                \"type\": \"section\",
                \"text\": {
                  \"type\": \"mrkdwn\",
                  \"text\": \":warning: *Drift detected* in \`${{ matrix.directory }}\`\n\`\`\`${PLAN:0:2900}\`\`\`\"
                }
              }]
            }"

terraform plan -detailed-exitcode returns:

• 0 = no changes (no drift)
• 1 = error
• 2 = changes detected (drift!)

When drift is detected, the Slack alert includes the plan diff so the on-call engineer can decide: import the change into Terraform, or revert it in the console.

Common drift sources we've seen:

• Console security group changes during incidents (most common)
• RDS parameter changes via the console after performance investigations
• Tag changes by cost allocation tools
• AWS service-linked resources created automatically

Developer Experience

Good tooling eliminates entire categories of mistakes:

Pre-commit hooks catch issues before they reach CI:

# .pre-commit-config.yaml
repos:
  - repo: https://github.com/antonbabenko/pre-commit-terraform
    rev: v1.96.0
    hooks:
      - id: terraform_fmt
      - id: terraform_validate
      - id: terraform_tflint
        args:
          - '--args=--only=terraform_deprecated_interpolation'
          - '--args=--only=terraform_unused_declarations'
          - '--args=--only=terraform_naming_convention'
      - id: terraform_docs
        args:
          - '--args=--lockfile=false'
      - id: terraform_checkov
        args:
          - '--args=--quiet'
          - '--args=--compact'
  - repo: https://github.com/pre-commit/pre-commit-hooks
    rev: v5.0.0
    hooks:
      - id: check-merge-conflict
      - id: end-of-file-fixer
      - id: trailing-whitespace
      - id: detect-private-key

Makefile for consistent operations:

# Makefile

DOMAIN ?= networking
ENV    ?= prod
DIR    := infrastructure/$(DOMAIN)/$(ENV)

.PHONY: init plan apply fmt validate lint

init:
    cd $(DIR) && terraform init -input=false

plan:
    cd $(DIR) && terraform plan -input=false

apply:
    @echo "ERROR: Use CI/CD pipeline to apply. Direct applies are disabled."
    @echo "Push your changes and create a PR."
    @exit 1

fmt:
    terraform fmt -recursive infrastructure/

validate:
    @for dir in $$(find infrastructure -name "*.tf" -not -path "*/modules/*" \
      | xargs dirname | sort -u); do \
      echo "Validating $$dir..."; \
      cd $$dir && terraform validate && cd - > /dev/null; \
    done

lint:
    pre-commit run --all-files

Note the apply target deliberately fails — it forces engineers to use the CI/CD pipeline. This was unpopular for the first week and universally loved by week three.

Version pinning with .terraform-version:

1.9.0

Combined with tfenv, every engineer uses the same Terraform version. No more "it plans clean on my machine."

Lessons Learned

1. Split state files by blast radius, not by resource type. We initially tried splitting by resource type (all VPCs in one state, all RDS in another). Wrong move — a VPC change still required re-planning the networking for every environment. Split by domain AND environment: networking/prod, networking/staging. Each state is an independent blast radius.

2. CI/CD for Terraform is non-negotiable at scale. Every month we delayed CI/CD, we had another incident caused by a local apply. The "but it's slower" argument evaporates when you realize an 8-minute CI apply is faster than a 6-hour recovery from a bad local apply.

3. Drift detection catches 90% of issues before they become incidents. The nightly cron caught 47 drift instances in its first month. Most were harmless tag changes, but three were security group modifications that bypassed our network policies. Those three alone justified the entire effort.

4. Module versioning saves you from "it works on my machine." Before versioning, updating a shared module immediately affected every environment. After versioning, we update staging to v2.4.0, let it bake for a week, then promote to production. Zero surprise breakages since.

5. The best Terraform code looks boring. No clever tricks. No deep nesting. No dynamic blocks when a simple resource will do. No count when for_each is clearer. The engineers who come after you should be able to understand every file in under 60 seconds. Boring is maintainable.

Action Items

Phase 1: Split the Monolith (Weeks 1-2)

• [ ] Create S3 bucket and DynamoDB table for state
• [ ] Design directory structure by domain/environment
• [ ] Migrate state files with terraform state mv, one domain per day
• [ ] Verify zero-diff plans in both source and destination after each move
• [ ] Delete old monolith state after full migration

Phase 2: CI/CD Pipeline (Weeks 3-4)

• [ ] Set up OIDC federation between GitHub Actions and AWS
• [ ] Create TerraformExecutionRole in each account
• [ ] Write GitHub Actions workflow with plan-on-PR and apply-on-merge
• [ ] Add Infracost integration for cost estimates on PRs
• [ ] Enable branch protection rules requiring plan review
• [ ] Disable local terraform apply via Makefile

Phase 3: Guardrails and DX (Weeks 5-6)

• [ ] Set up nightly drift detection cron with Slack alerts
• [ ] Write OPA policies for critical resource protection
• [ ] Add prevent_destroy lifecycle to production databases and S3 buckets
• [ ] Configure pre-commit hooks (fmt, validate, tflint, terraform-docs, checkov)
• [ ] Pin Terraform version with tfenv and .terraform-version
• [ ] Write onboarding doc for new engineers

---

This is post #14 in my infrastructure engineering series. Previous posts cover Kubernetes cost optimization, multi-tenant EKS platforms, and AWS data transfer costs.

Comments

[IICTP]

Really appreciated how concrete this is — splitting state by blast radius rather than resource type, and preferring SSM over terraform_remote_state so a networking change doesn't drag in another state file, are the kind of choices that only come from living with the monolith. The Makefile apply target that deliberately fails to force CI/CD made me smile too. How did you handle the two-week migration in practice — did you freeze changes to a domain while its state was being moved, or keep applying through the old monolith until the zero-diff check passed?