The Security Group Change Terraform Never Told Me About

Terraform had a blind spot in my three-tier project.

The code could create the infrastructure. The state lived in S3. But if someone changed a security group in the AWS console, Terraform would not tell me when it happened.

I would discover it the next time I ran terraform plan.

That could be tomorrow or six months later.

So I built a drift detector that runs the same plan inside CodeBuild and sends an SNS email when Terraform finds a difference.

I made CodeBuild the entry point. One run compares the deployed infrastructure with Terraform and reports any difference by email.

The Stack

CodeBuild run                 → starts the detector
        ↓
CodeBuild                     → installs Terraform and clones the repo
        ↓
terraform plan                → compares code, state, and AWS
        ↓
SNS                           → sends an email when drift is detected

Why I Used CodeBuild Instead of Lambda

My first idea was Lambda.

A scheduled function could run terraform plan, then SNS could send the result.

The problem was packaging Terraform. I could build a Lambda container image and store it in ECR, but that adds a Dockerfile, image builds, image updates, and another repository to maintain.

CodeBuild already does what this job needs: start a clean Linux environment, run a script, stream logs to CloudWatch, and exit.

For a Terraform plan job, that was the simpler choice.

Step 1: Build the CodeBuild Job

The CodeBuild project does not build application source. Its source type is NO_SOURCE, and Terraform injects the buildspec directly:

resource "aws_codebuild_project" "drift" {
  name          = "terraform-drift"
  description   = "Runs terraform plan against Three-Tier-Infra to detect drift"
  build_timeout = 10
  service_role  = aws_iam_role.codebuild.arn

  artifacts {
    type = "NO_ARTIFACTS"
  }

  environment {
    compute_type = "BUILD_GENERAL1_SMALL"
    image        = "aws/codebuild/amazonlinux2-x86_64-standard:5.0"
    type         = "LINUX_CONTAINER"

    environment_variable {
      name  = "SNS_TOPIC_ARN"
      value = aws_sns_topic.drift.arn
    }
  }

  source {
    type      = "NO_SOURCE"
    buildspec = file("${path.module}/buildspec.yml")
  }

  logs_config {
    cloudwatch_logs {
      group_name = "/codebuild/terraform-drift"
    }
  }
}

Passing SNS_TOPIC_ARN as an environment variable keeps the generated ARN out of the buildspec. If Terraform recreates the topic, the next apply updates CodeBuild with the new ARN.

Step 2: Run Terraform Plan and Capture Drift

This was the buildspec used by the detector:

version: 0.2

phases:
  install:
    commands:
      - curl -o terraform.zip https://releases.hashicorp.com/terraform/1.10.0/terraform_1.10.0_linux_amd64.zip
      - unzip terraform.zip -d /usr/local/bin

  pre_build:
    commands:
      - git clone https://github.com/lalitbagga/Three-Tier-Infra.git /tmp/Three-Tier-Infra
      - cd /tmp/Three-Tier-Infra
      - terraform init

  build:
    commands:
      - cd /tmp/Three-Tier-Infra
      - EXIT_CODE=0
      - terraform plan -detailed-exitcode -out=plan.tfplan -lock=false || EXIT_CODE=$?
      - |
        if [ "$EXIT_CODE" -eq 2 ]; then
          echo "Drift detected. Publishing details to SNS."
          aws sns publish \
            --topic-arn "$SNS_TOPIC_ARN" \
            --subject "Terraform Drift Detected" \
            --message "Drift found in Three-Tier-Infra"
        elif [ "$EXIT_CODE" -eq 1 ]; then
          echo "Error during terraform plan."
          exit 1
        else
          echo "No drift detected."
        fi

The important part is -detailed-exitcode.

terraform plan -detailed-exitcode returns:

0 → plan succeeded, no changes
1 → Terraform failed
2 → plan succeeded, changes detected

The buildspec captures that result before CodeBuild decides whether the job succeeded:

EXIT_CODE=0
terraform plan -detailed-exitcode -out=plan.tfplan -lock=false || EXIT_CODE=$?

That gives the pipeline three clear paths: do nothing when the infrastructure matches, stop when Terraform fails, and notify when Terraform finds drift.

The SSH Key Only Existed on My Laptop

The Three-Tier project originally created its EC2 key pair from a local file:

public_key = file("~/.ssh/bastion-key.pub")

That worked on my machine. CodeBuild starts in an ephemeral container where that file does not exist.

I moved the public key to SSM Parameter Store and changed the Three-Tier repository:

data "aws_ssm_parameter" "bastion_key" {
  name = "/threeTier/bastionKeyPublic"
}

resource "aws_key_pair" "bastion_key" {
  key_name   = "bastion-key"
  public_key = data.aws_ssm_parameter.bastion_key.value
}

The CodeBuild role received ssm:GetParameter for the /threeTier/* path.

This removed the developer-laptop dependency. The same Terraform plan can now run locally or in CodeBuild.

Step 3: Send the Email Through SNS

The SNS topic and email subscription are managed by Terraform:

resource "aws_sns_topic" "drift" {
  name = "terraform-drift-alerts"
}

resource "aws_sns_topic_subscription" "drift" {
  topic_arn = aws_sns_topic.drift.arn
  protocol  = "email"
  endpoint  = var.alert_email
}

CodeBuild also needs sns:Publish permission scoped to that topic. CloudWatch permissions allow the build to create its log group and write the execution logs.

The role has four jobs:

S3 read              → read Terraform state
SSM read             → retrieve shared parameters
SNS publish          → send the drift alert
CloudWatch Logs      → record the build

It does not have S3 write permissions and it cannot apply Terraform changes.

Verification

I started the CodeBuild project directly:

aws codebuild start-build \
  --project-name terraform-drift \
  --region us-east-2

The command returns a build ID. Use it to inspect the result:

aws codebuild batch-get-builds \
  --ids <build-id> \
  --region us-east-2

The successful drift path produced three pieces of evidence:

CodeBuild completed the Terraform plan
        ↓
The build published to terraform-drift-alerts
        ↓
The subscribed email received "Terraform Drift Detected"

The detected change included the SSM refactor made in the Three-Tier repository.

That confirmed the complete scope of this version: start the detector, compare the infrastructure, identify drift, and deliver the result by email.

What Is Next

Running the detector on demand proved the idea, but an email that says "drift happened" is not enough.

The next phase converts the Terraform plan to JSON, sends the event to SQS, and uses Lambda to classify every change as HIGH, MEDIUM, or LOW.

EventBridge will be added in an upcoming blog.