DEV Community: Lalit Bagga

My Terraform Drift Pipeline Fixed the Change, Then Forgot It

Lalit Bagga — Fri, 07 Aug 2026 00:51:46 +0000

My Terraform drift pipeline could detect a manual EC2 tag change, classify it as LOW, and run Terraform to remove it.

Then the pipeline moved on.

The evidence existed, but it was spread across CodeBuild output, Lambda logs, and an SNS message. If I wanted to know what changed, how it was classified, and whether remediation started, I had to reconstruct the event from multiple AWS services.

The pipeline could act on drift. It could not remember drift.

Phase 4 added that memory: a durable DynamoDB record, a read only API, and a small dashboard that turns the event history into something I can inspect without opening three AWS consoles.

The Stack

Terraform drift event
        ↓
SNS
        ↓
Severity Lambda
        ├── classifies HIGH / MEDIUM / LOW
        ├── starts remediation for eligible LOW drift
        └── writes the audit event to DynamoDB
                            ↓
                    API Gateway HTTP API
                            ↓
                    Read only Lambda
                            ↓
                    DynamoDB Query
                            ↓
             CloudFront → static dashboard
                            ↑
                      private S3 bucket

The browser receives static HTML, CSS, and JavaScript from CloudFront. JavaScript calls API Gateway, the API Lambda queries DynamoDB, and the returned JSON becomes the live dashboard.

There is no EC2 web server and no application process running continuously.

Step 1: Store Every Classified Event

I created a DynamoDB table with a composite key:

resource "aws_dynamodb_table" "drift_events" {
  name         = "terraform-drift-events"
  billing_mode = "PAY_PER_REQUEST"
  hash_key     = "project"
  range_key    = "timestamp"

  attribute {
    name = "project"
    type = "S"
  }

  attribute {
    name = "timestamp"
    type = "S"
  }
}

project groups the history for one Terraform project. The ISO 8601 timestamp orders its events.

DynamoDB only requires attribute definitions for keys and indexes. Fields such as high_count, changes, and status still belong in each item, but they do not belong in the table schema block.

I passed the table name into the existing severity Lambda instead of putting it directly in the code:

environment {
  variables = {
    DRIFT_EVENTS_TABLE       = aws_dynamodb_table.drift_events.name
    REMEDIATION_PROJECT_NAME = aws_codebuild_project.remediation.name
  }
}

The same Lambda role received permission to write to that specific table:

{
  Effect = "Allow"
  Action = [
    "dynamodb:PutItem",
    "dynamodb:UpdateItem",
    "dynamodb:GetItem",
    "dynamodb:Query",
    "dynamodb:Scan",
  ]
  Resource = aws_dynamodb_table.drift_events.arn
}

After classification, Lambda builds one audit item:

item = {
    "project": project,
    "timestamp": timestamp,
    "drift_count": len(changes),
    "high_count": len(classified["HIGH"]),
    "medium_count": len(classified["MEDIUM"]),
    "low_count": len(classified["LOW"]),
    "changes": classified,
    "action_taken": action_taken,
    "status": status,
}

if remediation_build_id:
    item["remediation_build_id"] = remediation_build_id

drift_table.put_item(Item=item)

The record stores the decision alongside the evidence used to make it. I do not need to correlate an SNS payload with a separate classification log just to understand one event.

For a change that needs human review, the item records manual_review. When Lambda starts the remediation build, it records remediation_triggered and includes the CodeBuild build ID.

That turns the audit item into the connection between detection, classification, and the action taken by the pipeline.

Step 2: Add a Read Only History API

The dashboard should not receive DynamoDB credentials. It calls an HTTP API instead.

I created a second Lambda whose only data permission is:

Action   = ["dynamodb:Query"]
Resource = aws_dynamodb_table.drift_events.arn

The route requires a project:

GET /drift-history?project=Three-Tier-Infra

The Lambda queries the partition key and returns newest events first:

response = table.query(
    KeyConditionExpression="#project = :project",
    ExpressionAttributeNames={"#project": "project"},
    ExpressionAttributeValues={":project": project},
    ScanIndexForward=False,
)

This is why the table uses project as its partition key. The API does not need to scan the complete table to retrieve one project's history.

DynamoDB numbers arrive in Python as Decimal, which json.dumps cannot serialize directly. I added a small serializer:

def json_serializer(value):
    if isinstance(value, Decimal):
        return int(value) if value % 1 == 0 else float(value)
    raise TypeError(f"Cannot serialize {type(value)}")

API Gateway uses an HTTP API with a Lambda proxy integration:

resource "aws_apigatewayv2_route" "get_drift_events" {
  api_id    = aws_apigatewayv2_api.drift_api.id
  route_key = "GET /drift-history"
  target    = "integrations/${aws_apigatewayv2_integration.drift_api.id}"
}

Calling the deployed route returned HTTP 200 and the stored event:

curl \
  "https://8zg1x51wne.execute-api.us-east-2.amazonaws.com/drift-history?project=Three-Tier-Infra"

The response contained:

{
  "events": [
    {
      "project": "Three-Tier-Infra",
      "timestamp": "2026-07-26T15:29:32Z",
      "drift_count": 1,
      "high_count": 0,
      "medium_count": 0,
      "low_count": 1,
      "action_taken": "remediation_triggered",
      "status": "REMEDIATION_STARTED"
    }
  ]
}

The changed resource was:

module.compute.aws_instance.bastion_host

Terraform reported it as an aws_instance update.

Step 3: Build a Dashboard Without a Web Server

The dashboard is static in how it is hosted, not in the data it displays.

S3 stores three files:

dashboard/
├── index.html
├── app.js
└── styles.css

CloudFront serves those files over HTTPS. app.js calls the API whenever the page loads or the user refreshes the data.

The dashboard shows total events, severity counts, detection timestamps, Terraform resource addresses, actions, and remediation status. A severity filter narrows the history, and the page refreshes automatically every 60 seconds.

The S3 bucket remains private. CloudFront uses Origin Access Control to read objects:

resource "aws_cloudfront_origin_access_control" "drift_dashboard" {
  name                              = "terraform-drift-dashboard"
  origin_access_control_origin_type = "s3"
  signing_behavior                  = "always"
  signing_protocol                  = "sigv4"
}

The bucket policy grants s3:GetObject to the CloudFront service only when the request comes from this distribution.

Step 4: Keep the Public Surface Read Only

I deliberately did not add a Fix drift button.

The first dashboard is for visibility. It cannot start CodeBuild, run Terraform, or change DynamoDB records.

The API exposes one GET route. I added API Gateway throttling:

default_route_settings {
  throttling_burst_limit = 20
  throttling_rate_limit  = 10
}

I also restricted browser CORS responses to the deployed CloudFront origin:

cors_configuration {
  allow_headers = ["content-type"]
  allow_methods = ["GET"]
  allow_origins = [
    "https://${aws_cloudfront_distribution.drift_dashboard.domain_name}"
  ]
  max_age = 3600
}

CORS is not authentication. Someone who knows the API URL can still call it outside a browser.

That is an accepted limitation for this public dashboard. The API returns resource addresses, classifications, and actions. It does not return raw Terraform state, credentials, plan values, or CloudWatch logs.

An operator remediation API would require authentication, authorization, review, concurrency controls, and an audit record of who approved the action. It does not belong in this first dashboard.

Verification

I created real drift by changing the tag on:

module.compute.aws_instance.bastion_host

The detection pipeline reported one LOW update. Lambda started remediation and wrote this event to DynamoDB:

project:              Three-Tier-Infra
drift_count:          1
high_count:           0
medium_count:         0
low_count:            1
action_taken:         remediation_triggered
status:               REMEDIATION_STARTED

The record also contained the remediation CodeBuild ID and the classified resource details.

After remediation, I returned to the EC2 instance and confirmed the tag was restored to the value declared in Terraform.

I then tested the deployed layers independently:

CloudFront index.html: HTTP 200
CloudFront app.js:     HTTP 200
API history query:     HTTP 200
CORS origin:           CloudFront dashboard URL

The API response to the dashboard origin included:

access-control-allow-origin: https://dqjunwd8v0pry.cloudfront.net

The complete verified path was:

EC2 tag changed
→ Terraform detected one update
→ Lambda classified it LOW
→ Lambda wrote the event to DynamoDB
→ remediation restored the tag
→ API returned the audit event
→ dashboard displayed the history

The DynamoDB status remains REMEDIATION_STARTED. That proves CodeBuild accepted the build request, not that the build ultimately succeeded. I verified success separately by observing the restored EC2 tag.

Updating the audit item from REMEDIATION_STARTED to REMEDIATION_SUCCEEDED is separate completion tracking work.

What Comes Next

The system can now detect drift, classify it, remediate the tested LOW tag scenario, remember the event, expose it through an API, and display it in a dashboard.

The next step is making the audit lifecycle more complete. The current item proves that CodeBuild accepted the remediation request, but it does not record the final build result.

Tracking that result would let the same event move from REMEDIATION_STARTED to REMEDIATION_SUCCEEDED or REMEDIATION_FAILED.

I Added a Tag in AWS. Terraform Removed It Automatically.

Lalit Bagga — Tue, 04 Aug 2026 22:30:26 +0000

I opened the AWS console, selected an EC2 instance, added a tag, and saved it.

The tag did not exist in Terraform. The deployed infrastructure now said one thing while the code said another.

That was the test.

My drift detector already knew how to find the difference and classify the EC2 update as LOW. The missing part was closing the loop: once the system understood that the change was low severity, it needed to restore the infrastructure without waiting for me to run Terraform manually.

So I added a separate remediation pipeline.

The Stack

Manual EC2 tag added in AWS
        ↓
Drift CodeBuild runs terraform plan
        ↓
SNS publishes the structured change
        ↓
Lambda classifies the update as LOW
        ↓
Remediation CodeBuild runs terraform apply
        ↓
Terraform removes the unmanaged tag

Detection and remediation use different CodeBuild projects. The detector reads the infrastructure and reports differences. The remediation project has a separate IAM role because it can change AWS resources.

Step 1: Make EC2 Updates Eligible for Remediation

The classifier already grouped resource types into HIGH, MEDIUM, and LOW categories.

An EC2 instance belongs to MEDIUM_RISK_TYPES. Initially, that meant even a tag update was classified as MEDIUM and stopped before remediation.

The resource type alone was not enough. The action mattered too.

I updated the classifier so an in-place update on a MEDIUM resource becomes LOW, while creates remain MEDIUM and delete or replacement actions become HIGH:

def classify_change(resource_type, actions):
    if resource_type in HIGH_RISK_TYPES:
        return "HIGH"
    elif resource_type in MEDIUM_RISK_TYPES:
        if "delete" in actions or "replace" in actions:
            return "HIGH"
        elif "update" in actions:
            return "LOW"
        return "MEDIUM"
    else:
        return "LOW"

For this test, Terraform reported the manually added EC2 tag as an update. That moved it into the LOW path.

Step 2: Keep Deletions Out of the Automatic Path

LOW did not automatically mean “start Terraform.” I also filtered out deletions:

low_changes_to_remediate = [
    change for change in classified["LOW"]
    if "delete" not in change.get("actions", [])
]

This distinction matters. Updating an existing resource to match the code is different from recreating something that a person deliberately removed. Deletions stay in the manual-review path.

If an eligible LOW change remains, Lambda starts the remediation CodeBuild project:

if low_changes_to_remediate:
    codebuild.start_build(
        projectName=os.environ.get(
            "REMEDIATION_PROJECT_NAME",
            "terraform-drift-remediation",
        )
    )

The Lambda role only needs permission to start that specific project:

{
  Effect   = "Allow"
  Action   = ["codebuild:StartBuild"]
  Resource = aws_codebuild_project.remediation.arn
}

Lambda makes the routing decision. CodeBuild performs the repair.

Step 3: Run Remediation Separately

I did not add terraform apply to the existing drift detector. Detection should remain read-only even when remediation fails or is disabled.

The second build starts in a clean environment, installs Terraform, clones the Three-Tier repository, and applies the declared configuration:

version: 0.2

phases:
  install:
    commands:
      - curl -o terraform.zip https://releases.hashicorp.com/terraform/1.10.0/terraform_1.10.0_linux_amd64.zip
      - unzip terraform.zip -d /usr/local/bin
  pre_build:
    commands:
      - git clone https://github.com/lalitbagga/Three-Tier-Infra.git /tmp/Three-Tier-Infra
      - cd /tmp/Three-Tier-Infra
      - terraform init
  build:
    commands:
      - cd /tmp/Three-Tier-Infra
      - terraform apply -auto-approve -lock=false

The remediation project uses its own CodeBuild role with write access to the AWS services managed by the Three-Tier project. During implementation, Terraform exposed several missing permissions as it evaluated and changed resources, so I expanded the experimental role to cover the project.

That role is intentionally separate from the detector. A job that only runs terraform plan should not inherit the permissions required for terraform apply.

The Repository Was Part of the Control Loop

One remediation run tried to include unrelated resource actions because CodeBuild was cloning a version of the Three-Tier repository that did not match the configuration I intended to operate.

The problem was not the classifier. Lambda correctly identified the triggering drift. The remediation build was applying the complete repository it cloned from GitHub.

I pushed the current Three-Tier configuration before running the test again.

This is easy to miss in CI. Terraform does not know what exists only on a developer's laptop. If CodeBuild clones GitHub, the committed branch is the configuration Terraform will enforce.

The Tag Disappeared

With the current repository in GitHub, I repeated the test:

Opened the EC2 instance in the AWS console.
Added a tag manually and saved it.
Started the Terraform drift detector.
The plan reported an update to the EC2 instance.
Lambda classified the update as LOW.
Lambda started the remediation CodeBuild project.
CodeBuild ran Terraform apply.
I returned to the EC2 Tags view and confirmed that the manually added tag was gone.

Terraform restored the infrastructure to what was declared in Git.

Git configuration: tag absent
AWS resource:       tag added manually
Terraform result:   tag removed

The system was no longer only telling me about drift. For the LOW update I tested, it detected the change, classified it, and repaired it.

MEDIUM, HIGH, and deletion paths still stop for human review. That boundary is intentional. Automatic remediation is useful only when the system can distinguish a small correction from a change that deserves a person.

What Comes Next

The infrastructure is back in the expected state, but the evidence is spread across CodeBuild and CloudWatch logs.

The next part of the system will keep a remediation history in a database, expose it through an API, and visualize it in Grafana. I want to see what changed, how it was classified, whether remediation ran, and whether it succeeded without reconstructing the story from separate AWS services.

Detection found the tag. Classification decided it was LOW. Remediation removed it.

Now the system needs to remember that it happened.

My Terraform Drift Alert Could Not Explain the Drift

Lalit Bagga — Wed, 22 Jul 2026 02:40:34 +0000

My Terraform drift detector had a problem.

It could run terraform plan, detect drift, and send me an email through SNS. But the message only told me that drift existed.

It did not tell me which resources changed. It did not tell me whether the change affected IAM, networking, or something less critical.

So I changed the alert into a structured event and sent it to two places: SQS for audit processing and Lambda for severity classification.

The Stack

CodeBuild                  → runs terraform plan
        ↓
terraform show -json       → extracts the changed resources
        ↓
SNS                        → publishes one structured event
        ↓
        ├── SQS            → keeps the event for audit processing
        └── Lambda         → classifies changes as HIGH, MEDIUM, or LOW
                 ↓
            CloudWatch     → stores the classification logs

Step 1: Publish Structured Drift Details

The build already saved the Terraform plan:

terraform plan -detailed-exitcode -out=plan.tfplan -lock=false || EXIT_CODE=$?

Exit code 2 means Terraform found changes. When that happens, I convert the saved plan to JSON:

terraform show -json plan.tfplan > /tmp/plan.json

Terraform plan JSON is large. SNS and Lambda do not need all of it, so I used jq to keep only the useful fields:

DRIFT_CHANGES=$(jq -c '[
  .resource_changes[] |
  select(.change.actions != ["no-op"]) |
  {
    address: .address,
    type: .type,
    actions: .change.actions
  }
]' /tmp/plan.json)

Each change now contains:

{
  "address": "module.compute.aws_iam_role.ec2_role",
  "type": "aws_iam_role",
  "actions": ["create"]
}

The important part here is what I left out. I did not publish the complete before-and-after resource values. Resource address, type, and action are enough for classification without sending the entire Terraform plan through SNS.

Step 2: Send the Event to SQS

I created an SQS queue called terraform-drift-audit and subscribed it to the SNS topic.

resource "aws_sqs_queue" "drift_audit" {
  name = "terraform-drift-audit"
}

resource "aws_sns_topic_subscription" "sqs" {
  topic_arn = aws_sns_topic.drift.arn
  protocol  = "sqs"
  endpoint  = aws_sqs_queue.drift_audit.arn
}

I also added the queue URL as a Terraform output:

output "sqs_queue_url" {
  description = "URL of the drift audit SQS queue"
  value       = aws_sqs_queue.drift_audit.url
}

After terraform apply, I can retrieve the URL without opening the AWS console:

terraform output -raw sqs_queue_url

That URL is what I used with the AWS CLI to check whether SNS delivered the message.

SQS also needs permission to receive messages from SNS. I restricted the queue policy to the drift topic instead of allowing every SNS topic:

Condition = {
  ArnEquals = {
    "aws:SourceArn" = aws_sns_topic.drift.arn
  }
}

SQS is not the final audit database. It keeps the event available for a future consumer. Permanent drift history belongs to a later phase.

The Queue Existed in the Wrong Region

I copied the queue URL into this command:

aws sqs receive-message \
  --queue-url https://sqs.us-east-2.amazonaws.com/<account-id>/terraform-drift-audit \
  --max-number-of-messages 10

AWS returned NonExistentQueue.

The queue existed. My AWS CLI was using us-east-1, while the Terraform project created the queue in us-east-2.

The fix was adding the region explicitly to the same command:

aws sqs receive-message \
  --queue-url https://sqs.us-east-2.amazonaws.com/<account-id>/terraform-drift-audit \
  --max-number-of-messages 10 \
  --region us-east-2

The resource name was correct. The CLI was asking the wrong regional endpoint.

After fixing the region, I could see the structured drift message in SQS.

Step 3: Classify Every Change with Lambda

The Lambda function receives the same SNS event that went to SQS.

SNS does not send my drift JSON as the top-level Lambda event. It wraps the message inside Records[].Sns.Message. The function must first extract that string and parse it back into JSON.

This is the complete classifier:

import json
import os
from datetime import datetime, timezone

# Severity definitions
HIGH_RISK_TYPES = {
    "aws_security_group",
    "aws_security_group_rule",
    "aws_iam_role",
    "aws_iam_policy",
    "aws_iam_role_policy",
    "aws_iam_user",
    "aws_iam_group",
    "aws_iam_access_key",
}

MEDIUM_RISK_TYPES = {
    "aws_instance",
    "aws_db_instance",
    "aws_ecs_cluster",
    "aws_ecs_service",
    "aws_ecs_task_definition",
    "aws_lb",
    "aws_lb_listener",
    "aws_lb_target_group",
    "aws_nat_gateway",
    "aws_route_table",
    "aws_network_acl",
}


def classify_change(resource_type, actions):
    """Classify a single resource change by severity."""
    if resource_type in HIGH_RISK_TYPES:
        return "HIGH"
    elif resource_type in MEDIUM_RISK_TYPES:
        # Delete or replace is more concerning than create
        if "delete" in actions or "replace" in actions:
            return "HIGH"
        return "MEDIUM"
    else:
        return "LOW"


def lambda_handler(event, context):
    """Process SNS message with drift details and classify severity."""
    print(f"Received event: {json.dumps(event)}")

    for record in event.get("Records", []):
        # SNS message is in the SNS record
        sns_message = record.get("Sns", {}).get("Message", "{}")

        try:
            drift_data = json.loads(sns_message)
        except json.JSONDecodeError:
            print(f"Failed to parse SNS message: {sns_message}")
            continue

        timestamp = drift_data.get(
            "timestamp",
            datetime.now(timezone.utc).isoformat()
        )
        project = drift_data.get("project", "unknown")
        changes = drift_data.get("changes", [])

        # Classify each change
        classified = {"HIGH": [], "MEDIUM": [], "LOW": []}

        for change in changes:
            address = change.get("address", "unknown")
            resource_type = change.get("type", "unknown")
            actions = change.get("actions", [])

            severity = classify_change(resource_type, actions)
            classified[severity].append({
                "address": address,
                "type": resource_type,
                "actions": actions,
            })

        # Build summary
        summary = {
            "timestamp": timestamp,
            "project": project,
            "total_changes": len(changes),
            "high_count": len(classified["HIGH"]),
            "medium_count": len(classified["MEDIUM"]),
            "low_count": len(classified["LOW"]),
            "classified": classified,
        }

        # Log the classified drift
        print(
            f"Drift classification: {json.dumps(summary, indent=2)}"
        )

        # TODO Phase 3: Auto-remediate HIGH severity changes
        # TODO Phase 3: Alert on MEDIUM severity changes

    return {"statusCode": 200, "body": "Drift classified"}

The code does four things:

Extracts the SNS message from the Lambda event.
Parses the structured drift JSON.
Classifies each resource by type and action.
Writes the complete summary to CloudWatch Logs.

IAM and security group resources are HIGH. Resources such as EC2, RDS, ECS, load balancers, NAT gateways, and route tables start as MEDIUM. Everything else starts as LOW.

A delete or replacement on a MEDIUM resource becomes HIGH. This classifier is deliberately simple. It is a deterministic first version that I can read, test, and improve later.

The Lambda Package Needed Another Provider

Terraform packages the Python file using archive_file:

data "archive_file" "severity_lambda" {
  type        = "zip"
  source_file = "${path.module}/lambda/severity_classifier.py"
  output_path = "${path.module}/lambda/severity_classifier.zip"
}

The first run failed because the archive provider was missing from .terraform.lock.hcl.

The fix was:

terraform init -upgrade

That downloaded hashicorp/archive and updated the lock file.

The generated ZIP does not need to be committed. Terraform creates it when the configuration runs.

Verification

I triggered a real drift event and checked both destinations from the terminal.

First I read up to 10 messages from SQS:

aws sqs receive-message \
  --queue-url https://sqs.us-east-2.amazonaws.com/<account-id>/terraform-drift-audit \
  --max-number-of-messages 10 \
  --region us-east-2

The structured message appeared in the SQS queue. Lambda received the event and wrote this summary to CloudWatch:

aws logs tail /aws/lambda/terraform-drift-severity \
  --follow \
  --region us-east-2

--follow keeps the terminal attached to the log group. When the next drift event invokes Lambda, the classification appears directly in the terminal.

{
  "high_count": 8,
  "medium_count": 13,
  "low_count": 38
}

The HIGH results included an IAM role and an ALB security group:

module.compute.aws_iam_role.ec2_role
module.security.aws_security_group.alb_sg

Lambda classified all 59 resource changes reported by that Terraform plan. This confirmed that the structured SNS event reached Lambda and that each change was assigned a severity.

What Is Next

The system can now tell me what changed and how serious each change might be.

It still does not remediate anything.

That is intentional. The next step is deciding what should happen after classification. I will start with LOW-severity updates that are safe to test automatically. MEDIUM, HIGH, and deletion paths will remain outside automatic remediation and will later require human review.

That is Phase 3.

The Security Group Change Terraform Never Told Me About

Lalit Bagga — Wed, 15 Jul 2026 14:06:02 +0000

Terraform had a blind spot in my three-tier project.

The code could create the infrastructure. The state lived in S3. But if someone changed a security group in the AWS console, Terraform would not tell me when it happened.

I would discover it the next time I ran terraform plan.

That could be tomorrow or six months later.

So I built a drift detector that runs the same plan inside CodeBuild and sends an SNS email when Terraform finds a difference.

I made CodeBuild the entry point. One run compares the deployed infrastructure with Terraform and reports any difference by email.

The Stack

CodeBuild run                 → starts the detector
        ↓
CodeBuild                     → installs Terraform and clones the repo
        ↓
terraform plan                → compares code, state, and AWS
        ↓
SNS                           → sends an email when drift is detected

Why I Used CodeBuild Instead of Lambda

My first idea was Lambda.

A scheduled function could run terraform plan, then SNS could send the result.

The problem was packaging Terraform. I could build a Lambda container image and store it in ECR, but that adds a Dockerfile, image builds, image updates, and another repository to maintain.

CodeBuild already does what this job needs: start a clean Linux environment, run a script, stream logs to CloudWatch, and exit.

For a Terraform plan job, that was the simpler choice.

Step 1: Build the CodeBuild Job

The CodeBuild project does not build application source. Its source type is NO_SOURCE, and Terraform injects the buildspec directly:

resource "aws_codebuild_project" "drift" {
  name          = "terraform-drift"
  description   = "Runs terraform plan against Three-Tier-Infra to detect drift"
  build_timeout = 10
  service_role  = aws_iam_role.codebuild.arn

  artifacts {
    type = "NO_ARTIFACTS"
  }

  environment {
    compute_type = "BUILD_GENERAL1_SMALL"
    image        = "aws/codebuild/amazonlinux2-x86_64-standard:5.0"
    type         = "LINUX_CONTAINER"

    environment_variable {
      name  = "SNS_TOPIC_ARN"
      value = aws_sns_topic.drift.arn
    }
  }

  source {
    type      = "NO_SOURCE"
    buildspec = file("${path.module}/buildspec.yml")
  }

  logs_config {
    cloudwatch_logs {
      group_name = "/codebuild/terraform-drift"
    }
  }
}

Passing SNS_TOPIC_ARN as an environment variable keeps the generated ARN out of the buildspec. If Terraform recreates the topic, the next apply updates CodeBuild with the new ARN.

Step 2: Run Terraform Plan and Capture Drift

This was the buildspec used by the detector:

version: 0.2

phases:
  install:
    commands:
      - curl -o terraform.zip https://releases.hashicorp.com/terraform/1.10.0/terraform_1.10.0_linux_amd64.zip
      - unzip terraform.zip -d /usr/local/bin

  pre_build:
    commands:
      - git clone https://github.com/lalitbagga/Three-Tier-Infra.git /tmp/Three-Tier-Infra
      - cd /tmp/Three-Tier-Infra
      - terraform init

  build:
    commands:
      - cd /tmp/Three-Tier-Infra
      - EXIT_CODE=0
      - terraform plan -detailed-exitcode -out=plan.tfplan -lock=false || EXIT_CODE=$?
      - |
        if [ "$EXIT_CODE" -eq 2 ]; then
          echo "Drift detected. Publishing details to SNS."
          aws sns publish \
            --topic-arn "$SNS_TOPIC_ARN" \
            --subject "Terraform Drift Detected" \
            --message "Drift found in Three-Tier-Infra"
        elif [ "$EXIT_CODE" -eq 1 ]; then
          echo "Error during terraform plan."
          exit 1
        else
          echo "No drift detected."
        fi

The important part is -detailed-exitcode.

terraform plan -detailed-exitcode returns:

0 → plan succeeded, no changes
1 → Terraform failed
2 → plan succeeded, changes detected

The buildspec captures that result before CodeBuild decides whether the job succeeded:

EXIT_CODE=0
terraform plan -detailed-exitcode -out=plan.tfplan -lock=false || EXIT_CODE=$?

That gives the pipeline three clear paths: do nothing when the infrastructure matches, stop when Terraform fails, and notify when Terraform finds drift.

The SSH Key Only Existed on My Laptop

The Three-Tier project originally created its EC2 key pair from a local file:

public_key = file("~/.ssh/bastion-key.pub")

That worked on my machine. CodeBuild starts in an ephemeral container where that file does not exist.

I moved the public key to SSM Parameter Store and changed the Three-Tier repository:

data "aws_ssm_parameter" "bastion_key" {
  name = "/threeTier/bastionKeyPublic"
}

resource "aws_key_pair" "bastion_key" {
  key_name   = "bastion-key"
  public_key = data.aws_ssm_parameter.bastion_key.value
}

The CodeBuild role received ssm:GetParameter for the /threeTier/* path.

This removed the developer-laptop dependency. The same Terraform plan can now run locally or in CodeBuild.

Step 3: Send the Email Through SNS

The SNS topic and email subscription are managed by Terraform:

resource "aws_sns_topic" "drift" {
  name = "terraform-drift-alerts"
}

resource "aws_sns_topic_subscription" "drift" {
  topic_arn = aws_sns_topic.drift.arn
  protocol  = "email"
  endpoint  = var.alert_email
}

CodeBuild also needs sns:Publish permission scoped to that topic. CloudWatch permissions allow the build to create its log group and write the execution logs.

The role has four jobs:

S3 read              → read Terraform state
SSM read             → retrieve shared parameters
SNS publish          → send the drift alert
CloudWatch Logs      → record the build

It does not have S3 write permissions and it cannot apply Terraform changes.

Verification

I started the CodeBuild project directly:

aws codebuild start-build \
  --project-name terraform-drift \
  --region us-east-2

The command returns a build ID. Use it to inspect the result:

aws codebuild batch-get-builds \
  --ids <build-id> \
  --region us-east-2

The successful drift path produced three pieces of evidence:

CodeBuild completed the Terraform plan
        ↓
The build published to terraform-drift-alerts
        ↓
The subscribed email received "Terraform Drift Detected"

The detected change included the SSM refactor made in the Three-Tier repository.

That confirmed the complete scope of this version: start the detector, compare the infrastructure, identify drift, and deliver the result by email.

What Is Next

Running the detector on demand proved the idea, but an email that says "drift happened" is not enough.

The next phase converts the Terraform plan to JSON, sends the event to SQS, and uses Lambda to classify every change as HIGH, MEDIUM, or LOW.

EventBridge will be added in an upcoming blog.

Observability for ECS: Prometheus, Grafana, and Crash Testing

Lalit Bagga — Thu, 09 Jul 2026 14:10:47 +0000

CloudWatch logs tell you what happened after something goes wrong. They are reactive. What I wanted was something that shows me what is happening right now. CPU, memory, request rates, the shape of the system while it is running. That is observability, and CloudWatch alone is not enough for it.

So I added Prometheus and Grafana, ran a load test, and then deliberately crashed the app to watch ECS recover on its own.

The Stack

Node.js app on ECS Fargate  → exposes /metrics endpoint
        ↓
Prometheus on EC2            → scrapes /metrics every 15 seconds
        ↓
Grafana on EC2               → visualizes the data from Prometheus
        ↓
Uptime Robot                 → monitors public URL availability

Prometheus and Grafana run together on a t3.micro EC2 instance using Docker Compose. Free tier eligible. No managed services needed.

Step 1: Expose Metrics from the Node App

Prometheus works by scraping an HTTP endpoint that exposes metrics in a specific format. The app needs to provide that endpoint.

Install the Prometheus client library:

npm install prom-client

Add the metrics endpoint to index.js:

const express = require('express')
const client = require('prom-client')

const app = express()
const port = 3000

const collectDefaultMetrics = client.collectDefaultMetrics
collectDefaultMetrics()

app.get('/', (req, res) => {
  res.json({ message: 'Welcome to Three Tier App' })
})

app.get('/health', (req, res) => {
  res.json({ status: 'ok' })
})

app.get('/metrics', async (req, res) => {
  res.set('Content-Type', client.register.contentType)
  res.end(await client.register.metrics())
})

app.listen(port, () => {
  console.log(`App listening on port ${port}`)
})

collectDefaultMetrics() automatically collects Node.js runtime metrics, heap memory, CPU time, event loop lag, garbage collection. No manual instrumentation needed for the basics.

Hitting /metrics now returns something like:

# HELP nodejs_heap_size_used_bytes Process heap size used from Node.js in bytes.
# TYPE nodejs_heap_size_used_bytes gauge
nodejs_heap_size_used_bytes 4567890

Push this change through the GitHub Actions pipeline and the running ECS task will expose metrics at /metrics through the ALB.

Step 2: Run Prometheus and Grafana on EC2

I added a monitoring EC2 instance to my existing compute Terraform module. The user_data script installs Docker, installs Docker Compose, creates the configuration files, and starts both containers on boot.

Prometheus config:

global:
  scrape_interval: 15s

scrape_configs:
  - job_name: 'three-tier-app'
    static_configs:
      - targets: ['your-alb-dns-name']
    metrics_path: '/metrics'

Prometheus scrapes the ALB URL at /metrics every 15 seconds. The ALB forwards requests to the healthy ECS task.

Docker Compose:

services:
  prometheus:
    image: prom/prometheus
    ports:
      - "9090:9090"
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
      - prometheus-data:/prometheus

  grafana:
    image: grafana/grafana
    ports:
      - "3000:3000"
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=admin123
    volumes:
      - grafana-data:/var/lib/grafana

volumes:
  prometheus-data:
  grafana-data:

The named volumes mean Prometheus data and Grafana dashboards survive container restarts. If the containers restart the historical data is still there.

Security group for the monitoring instance:

Port 3000 → open to 0.0.0.0/0  (Grafana, public)
Port 9090 → not open            (Prometheus, internal only)
Port 22   → open to my IP only  (SSH management)

Prometheus is not exposed publicly. It contains raw infrastructure metrics you do not want the public reading. Grafana talks to Prometheus internally using the Docker Compose service name, not localhost.

The Docker Networking Lesson

When I first tried to connect Grafana to Prometheus I used http://localhost:9090 as the URL. It failed immediately.

The reason is that inside a Docker container localhost refers to that container itself, not the host machine and not other containers. Prometheus is a separate container. To reach it from Grafana you use the service name defined in Docker Compose:

http://prometheus:9090

Docker Compose creates an internal network where containers can reference each other by service name. This is fundamental Docker networking and easy to miss the first time.

Step 3: Connect Grafana to Prometheus

Once both containers are running:

Open Grafana at http://your-ec2-ip:3000
Go to Connections → Data Sources → Add data source → Prometheus
URL: http://prometheus:9090
Save and Test

The response should say the Prometheus API was queried successfully.

Then create a dashboard. The most useful metrics to start with:

nodejs_heap_size_used_bytes   → memory usage over time
process_cpu_seconds_total     → CPU consumption

Step 4: Load Testing

With the dashboard open I ran a load test using hey:

hey -n 10000 -c 50 http://your-alb-url/

This sends 10,000 requests with 50 concurrent connections. Watching the Grafana dashboard during the test showed memory climbing and CPU spiking in real time. The metrics endpoint was doing its job.

Step 5: Crash Testing

The most interesting part of observability is proving your system recovers from failure. I added a crash endpoint to the app:

app.get('/crash', (req, res) => {
  res.json({ message: 'Crashing...' })
  process.exit(1)
})

Deployed it through the pipeline, then triggered it:

curl http://your-alb-url/crash

What happened next:

Container process exited
        ↓
ECS detected unhealthy task
        ↓
Task deregistered from ALB target group
        ↓
ECS started a new task automatically
        ↓
New task passed health checks
        ↓
Task registered with ALB
        ↓
App responding again

Recovery took about 1 to 2 minutes. During that window the ALB returned 503s. Uptime Robot sent a down alert within 5 minutes, then a recovery alert once the new task was healthy.

The Grafana dashboard showed a gap in metrics during the crash window, then metrics resuming when the new task came up.

What This Proves

Nobody manually intervened. The container crashed, ECS detected it, a new task started, and traffic resumed. That is self-healing infrastructure working as designed.

The combination of Prometheus metrics, Grafana dashboards, and Uptime Robot gives three different views of the system:

Uptime Robot  → is it up or down right now
Grafana       → what is the system doing over time
CloudWatch    → what did the application log

Each answers a different question. Together they give you enough visibility to understand your system under normal load and under failure.

#aws#ecs#prometheus#grafana#devops

Making My ECS App Public with an Application Load Balancer

Lalit Bagga — Fri, 03 Jul 2026 14:46:57 +0000

My Node.js app was running on ECS Fargate and the CloudWatch logs confirmed it was healthy. But there was no way to actually reach it. No public URL. No way to test it from a browser. The container was alive but invisible.

The fix is an Application Load Balancer. Here is how I wired it up.

Why the App Was Unreachable

ECS Fargate tasks run in private subnets by design. No public IP, no direct internet access. That is the correct security posture for an application server. You never want your app directly exposed to the internet.

The ALB sits in the public subnet and acts as the entry point. Traffic flows like this:

Internet
        ↓
ALB (public subnet, internet facing)
        ↓
ECS Fargate task (private subnet, no public IP)

The app never needs a public IP. Only the ALB does.

What You Need to Wire Together

An ALB setup has three moving parts that work together:

Security Group controls who can connect to the ALB. Port 80 open to the world.

Target Group is where the ALB sends traffic after it is allowed in. It holds a list of healthy ECS tasks and runs health checks against them. If a task fails health checks it gets removed from rotation.

Listener watches a specific port on the ALB and decides what to do with incoming traffic. In this case listen on port 80 and forward to the target group.

A common mistake is thinking the security group is enough. It is not. The security group decides whether traffic is allowed. The listener decides what to do with it. The target group decides where it goes. All three are required.

One Thing ALB Requires

An ALB must span at least two subnets in different Availability Zones. This is an AWS requirement, not optional.

The reason is high availability. If one AZ goes down the ALB continues serving traffic through the other. You get one ALB, one DNS name, one entry point. AWS handles the redundancy behind the scenes.

This means your VPC needs public subnets in at least two different AZs. If you are defining subnets in Terraform always specify the availability zone explicitly:

resource "aws_subnet" "main_subnet_public_1" {
  vpc_id            = aws_vpc.main.id
  cidr_block        = "10.0.1.0/24"
  availability_zone = "us-east-2a"
}

resource "aws_subnet" "main_subnet_public_2" {
  vpc_id            = aws_vpc.main.id
  cidr_block        = "10.0.2.0/24"
  availability_zone = "us-east-2b"
}

If you leave out availability_zone AWS picks one automatically. You may end up with both subnets in the same AZ and the ALB will refuse to deploy.

Terraform for the ALB

I added an ALB module to my existing three tier Terraform project.

Security group for the ALB:

resource "aws_security_group" "alb_sg" {
  name   = "alb_sg"
  vpc_id = var.vpc_id

  tags = {
    Name = "alb_sg"
  }
}

resource "aws_vpc_security_group_ingress_rule" "alb_sg_ingress" {
  security_group_id = aws_security_group.alb_sg.id
  from_port         = 80
  to_port           = 80
  ip_protocol       = "tcp"
  cidr_ipv4         = "0.0.0.0/0"
}

resource "aws_vpc_security_group_egress_rule" "alb_sg_egress" {
  security_group_id = aws_security_group.alb_sg.id
  ip_protocol       = "-1"
  cidr_ipv4         = "0.0.0.0/0"
}

The ALB, target group, and listener:

resource "aws_lb" "main" {
  name               = "three-tier-alb"
  internal           = false
  load_balancer_type = "application"
  security_groups    = [var.alb_sg_id]
  subnets            = [var.public_subnet_1_id, var.public_subnet_2_id]

  tags = {
    Name = "three-tier-alb"
  }
}

resource "aws_lb_target_group" "main" {
  name        = "three-tier-tg"
  port        = 3000
  protocol    = "HTTP"
  vpc_id      = var.vpc_id
  target_type = "ip"

  health_check {
    path                = "/health"
    healthy_threshold   = 2
    unhealthy_threshold = 3
    interval            = 30
  }
}

resource "aws_lb_listener" "main" {
  load_balancer_arn = aws_lb.main.arn
  port              = 80
  protocol          = "HTTP"

  default_action {
    type             = "forward"
    target_group_arn = aws_lb_target_group.main.arn
  }
}

One thing worth noting on the target group: target_type = "ip" is required for Fargate. Fargate tasks do not have EC2 instance IDs. They have IP addresses. If you use instance as the target type the target group will never register any healthy targets and you will get 503s.

Locking Down the ECS Security Group

Before adding the ALB my ECS security group allowed inbound traffic from anywhere on port 3000. Once the ALB is in place that should change. Only the ALB should be able to talk to the ECS tasks. Not the public internet.

resource "aws_vpc_security_group_ingress_rule" "ecs_sg_ingress" {
  security_group_id            = aws_security_group.ecs_sg.id
  from_port                    = 3000
  to_port                      = 3000
  ip_protocol                  = "tcp"
  referenced_security_group_id = aws_security_group.alb_sg.id
}

Instead of cidr_ipv4 = "0.0.0.0/0" the ingress rule now references the ALB security group directly. Only traffic coming from the ALB is allowed to reach the ECS task on port 3000.

Wiring ECS to the Target Group

The ECS service needs to know about the target group so it can register tasks automatically. When a new task starts ECS registers its IP with the target group. When a task stops it deregisters.

resource "aws_ecs_service" "app" {
  name            = "three-tier-app-service"
  cluster         = aws_ecs_cluster.main.id
  task_definition = aws_ecs_task_definition.app.arn
  desired_count   = 1
  launch_type     = "FARGATE"

  network_configuration {
    subnets         = [var.private_subnet_id]
    security_groups = [var.ecs_sg_id]
  }

  load_balancer {
    target_group_arn = var.target_group_arn
    container_name   = "app-container"
    container_port   = 3000
  }

  depends_on = [var.alb_listener_arn]
}

The depends_on on the listener matters. ECS should not start registering tasks with the target group until the listener exists. Without this Terraform might try to create the ECS service before the listener is ready and health checks will fail in a loop.

The Result

After terraform apply the output shows the ALB DNS name:

alb_dns_name = "three-tier-alb-119067097.us-east-2.elb.amazonaws.com"

Hitting that URL in the browser returns the app response. Hitting /health returns a healthy status. The app is now publicly accessible through the load balancer with the ECS task completely hidden in the private subnet.

What Is Next

The app is live but there is no visibility into what it is actually doing. The next step is adding Prometheus and Grafana to collect real metrics and build dashboards, along with load testing and crash recovery to prove the infrastructure holds up under pressure.

aws#ecs#terraform#devops#loadbalancer

From EC2 to ECS: Containerizing My Three Tier App with GitHub Actions

Lalit Bagga — Mon, 22 Jun 2026 13:30:35 +0000

My three tier architecture had a problem. The app server was a plain EC2 instance sitting in a private subnet doing absolutely nothing. No application running on it. No way to deploy to it without SSHing in manually. That is not how modern infrastructure works.

So I containerized the app, pushed it to ECR, deployed it to ECS Fargate, and wired up a GitHub Actions pipeline that handles every deployment automatically. Here is how it went.

The App

I kept the application simple on purpose. The goal was never to build a complex application. The goal was to prove the full deployment pipeline works end to end. A Node.js Express API with two endpoints:

const express = require('express')
const app = express()
const port = 3000

app.get('/', (req, res) => {
  res.json({ message: 'Welcome to Three Tier App' })
})

app.get('/health', (req, res) => {
  res.json({ status: 'ok' })
})

app.listen(port, () => {
  console.log(`App listening on port ${port}`)
})

The health endpoint matters more than the root endpoint. ECS uses it to determine whether the container is healthy and ready to serve traffic. If health checks fail the task gets killed and restarted.

The Dockerfile

GitHub Copilot suggested a multi stage build and it was the right call:

FROM node:22-alpine AS deps

WORKDIR /app

COPY package*.json ./
RUN npm ci --omit=dev && npm cache clean --force

FROM node:22-alpine AS runtime

ENV NODE_ENV=production
WORKDIR /app

COPY --from=deps /app/node_modules ./node_modules
COPY index.js ./
COPY package.json ./

USER node

EXPOSE 3000

CMD ["node", "index.js"]

Two stages. The first stage installs dependencies. The second stage is the actual runtime image and only copies what it needs from the first stage. Build tools and npm cache never make it into the final image. Smaller image, faster pulls, better security.

USER node runs the process as a non root user. If the container gets compromised an attacker does not get root access to the underlying system.

The Architecture

Before writing any Terraform, it helps to understand what ECS actually needs:

Cluster — a logical boundary for your tasks. Think of it as the environment.

Task Definition — a blueprint. It tells ECS which image to run, how much CPU and memory to give it, which port to expose, and how to check if the container is healthy.

Service — keeps your task running. If the container crashes the service restarts it. You define how many copies should be running at any time.

Fargate — the launch type. Instead of managing EC2 instances as worker nodes, Fargate handles the underlying infrastructure. You only think about the container.

Execution Role — an IAM role that gives ECS permission to pull your image from ECR and write logs to CloudWatch.

Terraform for ECS

I added an ECS module to my existing three tier infrastructure:

resource "aws_ecs_cluster" "main" {
  name = "three-tier-app-cluster"
}

resource "aws_ecs_task_definition" "app" {
  family                   = "three-tier-app-task"
  network_mode             = "awsvpc"
  requires_compatibilities = ["FARGATE"]
  cpu                      = "256"
  memory                   = "512"
  execution_role_arn       = aws_iam_role.ecs_execution_role.arn

  container_definitions = jsonencode([
    {
      name      = "app-container"
      image     = var.ecr_repository_url
      essential = true
      portMappings = [
        {
          containerPort = 3000
          protocol      = "tcp"
        }
      ]
      healthCheck = {
        command     = ["CMD-SHELL", "wget -q -O /dev/null http://localhost:3000/health || exit 1"]
        interval    = 30
        timeout     = 5
        retries     = 3
        startPeriod = 10
      }
      logConfiguration = {
        logDriver = "awslogs"
        options = {
          "awslogs-group"         = "/ecs/three-tier-app"
          "awslogs-region"        = "us-east-2"
          "awslogs-stream-prefix" = "ecs"
        }
      }
    }
  ])
}

resource "aws_ecs_service" "app" {
  name            = "three-tier-app-service"
  cluster         = aws_ecs_cluster.main.id
  task_definition = aws_ecs_task_definition.app.arn
  desired_count   = 1
  launch_type     = "FARGATE"

  network_configuration {
    subnets         = [var.private_subnet_id]
    security_groups = [var.ecs_sg_id]
  }
}

The Errors

Two things broke and both were worth understanding.

Exit code 255 with no logs

The container kept exiting with code 255 and there was nothing in CloudWatch to explain why. The task definition had no logging configuration so ECS was swallowing all the output.

Added CloudWatch logging to the container definition and the actual error became visible immediately.

exec format error

exec /usr/local/bin/docker-entrypoint.sh: exec format error

I built the Docker image on a Mac with Apple Silicon which uses ARM architecture. ECS Fargate runs on AMD64. The image architecture did not match the platform and the container could not start.

The fix is building with an explicit platform flag:

docker buildx build --platform linux/amd64 -t three-tier-app .

This is easy to miss when developing locally on a Mac. The container runs fine locally and only fails when deployed to ECS.

The Assumption That Was Wrong

I assumed ECS would automatically pick up a new image when I pushed to ECR. Push a new image with the latest tag, ECS detects it, redeploys. That is not how it works.

ECS pins to the exact image digest that was running when the task started. Even if you push a new latest tag to ECR, the running task keeps using the old image until you explicitly trigger a new deployment.

This is actually intentional. You do not want your production containers randomly restarting because someone pushed a new image. You want deployments to be deliberate and controlled.

The way to trigger a new deployment is:

aws ecs update-service --cluster three-tier-app-cluster \
  --service three-tier-app-service \
  --force-new-deployment

Which is exactly what the GitHub Actions pipeline automates.

The GitHub Actions Pipeline

The pipeline lives in .github/workflows/deploy.yml and triggers on every push to main:

name: Deploy to ECS

on:
  push:
    branches:
      - main

env:
  AWS_REGION: us-east-2
  ECR_REPOSITORY: three-tier-app
  ECS_CLUSTER: three-tier-app-cluster
  ECS_SERVICE: three-tier-app-service

jobs:
  deploy:
    name: Build and Deploy
    runs-on: ubuntu-latest

    steps:
      - name: Checkout code
        uses: actions/checkout@v4

      - name: Configure AWS credentials
        uses: aws-actions/configure-aws-credentials@v4
        with:
          aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
          aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
          aws-region: ${{ env.AWS_REGION }}

      - name: Login to Amazon ECR
        id: login-ecr
        uses: aws-actions/amazon-ecr-login@v2

      - name: Build, tag, and push image to ECR
        env:
          ECR_REGISTRY: ${{ steps.login-ecr.outputs.registry }}
          IMAGE_TAG: ${{ github.sha }}
        run: |
          docker buildx build --platform linux/amd64 \
            -t $ECR_REGISTRY/$ECR_REPOSITORY:$IMAGE_TAG \
            -t $ECR_REGISTRY/$ECR_REPOSITORY:latest \
            --push .

      - name: Deploy to ECS
        run: |
          aws ecs update-service \
            --cluster ${{ env.ECS_CLUSTER }} \
            --service ${{ env.ECS_SERVICE }} \
            --force-new-deployment \
            --region ${{ env.AWS_REGION }}

Each image gets tagged twice. Once with the git commit SHA for traceability and once with latest for convenience. If something goes wrong you can identify exactly which commit is running in production.

AWS credentials are stored as GitHub secrets and never appear in the workflow file.

Verifying It Works

The app is running in a private subnet with no public IP. Without an Application Load Balancer there is no public URL to hit. But two things confirm everything is working.

CloudWatch logs show the app started successfully:

App listening on port 3000

And the ECS console shows the task as running and healthy with a recent timestamp after every pipeline run.

The pipeline going green means the image built, pushed to ECR, and ECS started a new deployment. CloudWatch logs confirm the new container came up healthy.

What Is Next

The app is running but invisible. The next step is adding an Application Load Balancer to expose it publicly. The ALB sits in the public subnet, accepts traffic from the internet, and forwards it to the ECS task in the private subnet.

Once that is in place the full URL will be accessible and the three tier story is complete.

#aws #ecs #docker #devops #githubactions

Refactoring Terraform: From One File to Modules

Lalit Bagga — Mon, 08 Jun 2026 13:31:33 +0000

My three-tier AWS architecture worked. VPC, subnets, bastion host, app server, RDS, all deployed and running. But my main.tf was a flat file with everything mixed together. Security groups next to route tables next to RDS instances next to IAM roles.

It worked for a learning project. It would not work in a real team environment where multiple people need to understand, maintain, and extend the infrastructure.

So I refactored it into modules. Here is what I learned.

What Is a Module

A module is just a folder with its own Terraform files. Nothing magic about it. You move related resources into that folder, define what it needs as inputs, define what it exposes as outputs, and then call it from your root configuration.

The root main.tf becomes an orchestrator, it calls each module and wires them together by passing outputs from one into inputs of another.

The Final Structure

Before refactoring everything lived in one file. After:

three-tier/
├── main.tf               ← calls all modules, wires them together
├── variables.tf
├── outputs.tf
└── module/
    ├── networking/
    │   ├── main.tf
    │   ├── variable.tf
    │   └── outputs.tf
    ├── security/
    │   ├── main.tf
    │   ├── variable.tf
    │   └── outputs.tf
    ├── compute/
    │   ├── main.tf
    │   ├── variable.tf
    │   └── output.tf
    └── database/
        ├── main.tf
        ├── variable.tf
        └── output.tf

Each module owns one concern:

networking  → VPC, subnets, IGW, NAT gateway, route tables
security    → security groups and all ingress/egress rules
compute     → IAM roles, instance profile, SSM, key pair, EC2 instances
database    → RDS instance, DB subnet group

The Core Pattern: Outputs and Variables

This is the most important thing to understand before you start. Modules cannot reach outside themselves. If the compute module needs the VPC ID, it cannot just reference aws_vpc.main.id that resource lives in the networking module now.

The pattern is always three steps:

Step 1 Output it from the source module:

# module/networking/outputs.tf
output "vpc_id" {
  value = aws_vpc.main.id
}

Step 2 Declare it as a variable in the receiving module:

# module/security/variable.tf
variable "vpc_id" {
  description = "VPC ID from networking module"
  type        = string
}

Step 3 Pass it through the root main.tf:

# main.tf
module "security" {
  source = "./module/security"
  vpc_id = module.networking.vpc_id
}

Every cross-module reference follows this exact pattern. Once you internalize it the errors stop being confusing.

The Dependency Order

Modules depend on each other in a specific order. Networking has no dependencies so it goes first. Security needs the VPC ID from networking. Compute and database both need outputs from networking and security.

networking  → no dependencies
    ↓
security    → needs vpc_id from networking
    ↓
compute     → needs subnet IDs from networking
            → needs bastion_sg_id, private_sg_id from security
database    → needs db subnet IDs from networking
            → needs db_sg_id from security

Terraform figures out the order automatically based on these references. You do not need to use depends_on explicitly as soon as you reference module.networking.vpc_id, Terraform knows networking must complete before security starts.

How I Approached the Refactor

I did it one module at a time, starting with networking. The process for each module was:

Create the module folder and files
Move the relevant resources into module/networking/main.tf
Add a module "networking" call in root main.tf
Run terraform plan
Fix the errors — usually missing outputs or undeclared variables
Repeat for next module

The errors I kept hitting all looked like this:

Error: Reference to undeclared resource
  on main.tf line 38, in resource "aws_security_group" "bastion_sg":
  vpc_id = aws_vpc.main.id

A managed resource "aws_vpc" "main" has not been declared in the root module.

This means a resource is trying to reference something that has moved into a module. The fix is always the same , output it from the module, declare a variable in the receiving module, pass it through root.

The State Migration Problem

Here is something nobody warns you about when refactoring Terraform into modules.

When you move a resource from root into a module, its address in the state file changes. What was aws_vpc.main becomes module.networking.aws_vpc.main. Terraform sees this as a different resource, it thinks the old one was deleted and a new one needs to be created.

Running terraform plan after the refactor showed this:

Plan: 27 to add, 0 to change, 27 to destroy.

That is not what you want. It would destroy and recreate all your infrastructure.

The proper fix for a production environment is terraform state mv , a command that tells Terraform a resource just moved, it was not deleted. You run one command per resource:

terraform state mv aws_vpc.main module.networking.aws_vpc.main
terraform state mv aws_subnet.main_subnet_public_1 module.networking.aws_subnet.main_subnet_public_1
# ... one for every resource

For a learning project with no real traffic or data at risk, the simpler path is:

terraform destroy
terraform apply

Destroy everything, apply fresh from the new module structure. Same end result, no manual state migration required.

The apply completed cleanly:

Apply complete! Resources: 35 added, 0 changed, 0 destroyed.

What the Root main.tf Looks Like Now

The root main.tf went from a flat list of 43+ resources to a clean orchestration file:

module "networking" {
  source     = "./module/networking"
  aws_region = var.aws_region
}

module "security" {
  source = "./module/security"
  vpc_id = module.networking.vpc_id
}

module "compute" {
  source                = "./module/compute"
  public_subnet_id      = module.networking.public_subnet_id
  private_subnet_id     = module.networking.private_subnet_id
  bastion_sg_id         = module.security.bastion_sg_id
  private_sg_id         = module.security.private_sg_id
}

module "database" {
  source         = "./module/database"
  db_subnet_1_id = module.networking.db_subnet_1_id
  db_subnet_2_id = module.networking.db_subnet_2_id
  db_sg_id       = module.security.db_sg_id
}

You can read this and immediately understand the infrastructure. Four modules, clear dependencies, no hunting through hundreds of lines to find what you need.

What I Learned

Modules are just folders. There is no magic. The mental shift is understanding that resources can no longer reference each other directly once they live in different modules. Everything goes through outputs and variables.

Start with networking. It has no dependencies so there are no wiring errors to debug. Get networking working first, then add security, then compute and database.

The state migration problem is real. In production you would never destroy and recreate. You would use terraform state mv or moved blocks to migrate state without downtime. For a learning project, destroy and recreate is fine, but knowing why the problem exists is important.

The root main.tf should be an orchestrator, not a resource file. If you have resource blocks in your root main.tf alongside module calls, that is a signal something belongs in a module.

What Is Next

The next step is enabling RDS IAM Authentication, replacing the hardcoded database password with token-based access. Storing credentials directly in Terraform is a bad practice and there is a cleaner way to handle it.

#aws #terraform #devops #infrastructureascode #modules