DEV Community: Lalit Bagga

The Rollback Worked. My Next Deploy Could Break It Again.

Lalit Bagga — Thu, 27 Aug 2026 19:06:29 +0000

My application was working again. That should have been the end of the problem.

It was not.

In [the first article in this series(https://blog.lalitbagga.com/posts/why-i-chose-k3s-debian-homelab), I built an always on Kubernetes lab on my home server. I deployed a small application, changed it from Version 2 to Version 3, and then practised a rollback.

Kubernetes restored Version 2 successfully. But the YAML file on my computer the file containing the instructions for the application still said Version 3.

I had repaired what was running, but I had not repaired the instructions that I would use next time. If I applied that file again, I could accidentally bring Version 3 back.

I corrected the YAML manually and confirmed that the file and cluster agreed. That solved the immediate problem, but it also exposed a weakness in my process: recovery depended on me remembering to repair two places.

That became the starting point for this project.

The Real Problem Was Two Different Answers

Kubernetes always has a live answer to the question, “What should run now?” My
YAML file had a saved answer to the same question.

After the rollback, those answers were different:

Running in Kubernetes: Version 2
Saved in YAML:          Version 3

Neither answer was hidden. The weakness was that nothing continuously checked
whether they agreed.

I wanted one durable place to record the version I had chosen. If somebody changed the cluster directly, the system should notice the difference and restore the recorded choice. If the recorded choice itself was bad, recovery should mean changing that record not making another temporary cluster edit.

That is the problem GitOps is meant to address.

I Considered Who Should Be Allowed to Change the Cluster

My first idea was the most direct one: let the application pipeline deploy the new version.

A pipeline, usually called continuous integration or CI, is an automated worker. It runs tests and builds the application whenever code changes. I could also give it the password like Kubernetes credentials needed to change the cluster.

That would be convenient, but it would give the application build process a second, much more powerful job. A mistake or compromised workflow could move directly from building code to changing the cluster.

I wanted a smaller boundary:

CI may build and publish a release.
CI may propose which release should run.
CI may not change Kubernetes directly.

The proposal would become a pull request. I would review it and merge it into Git, the same version history system used for the application code. Git would therefore hold the durable answer to “Which version should this environment run?”

This was not about removing every credential from CI. It still needed limited GitHub permission to publish an image and open a pull request. The important restriction was precise: application CI would not receive a Kubernetes credential or permission to deploy to the cluster.

I Designed the Process Before Choosing the Tools

At this point, I could describe the process without naming a Kubernetes tool:

Developer changes the application
        ↓
CI tests it and builds a release
        ↓
CI opens a pull request with the exact release
        ↓
A human reviews and merges the change
        ↓
A worker inside Kubernetes reads Git
        ↓
That worker makes the cluster match Git

This operating model is called GitOps. Git stores the desired state the version we intend to run. Software inside the cluster repeatedly compares that record with reality and corrects differences.

I chose Argo CD for that inside-the-cluster worker. Argo CD watches the Git repository and reconciles the cluster. “Reconcile” simply means compare what is running with what should be running, then work to make them agree.

This separation gave each part one main responsibility:

CI proves that the application can be built.
Git records the reviewed version choice.
Argo CD makes Kubernetes follow that choice.
Kubernetes reports whether the application becomes healthy.

I Needed an Exact Version, Not a Moving Name

A release also needed an identity that could not silently change.

Container images often use names such as latest or version-3. These are tags: convenient labels that can be moved to different image contents later. That would weaken Git as the record of exactly what I approved.

I therefore chose an image digest. A digest is a fingerprint calculated from the image contents. If the contents change, the fingerprint changes. Git would store that exact fingerprint, so the selected release could not change while keeping the same name.

CI would publish the image, learn its digest, and open a pull request changing only that value. It would then stop and wait for review.

I Used Helm to Keep the Kubernetes Instructions Together

The application needed several Kubernetes instructions: how to run it, how to reach it, how to check its health, and which image digest to use.

I packaged those instructions with Helm. Helm is a way to create a reusable Kubernetes template. The structure stays the same while selected values such as the image digest or number of copies can change.

Before allowing Argo CD to use the Helm package, I planned to test it on its own: install it, verify the application, upgrade it, roll it back, remove it, and install it again in a clean location.

This separated two questions:

Does the Kubernetes package work correctly?
Does the automated delivery process use it correctly?

If both failed at the same time, troubleshooting would be much harder.

I Chose a Small Application That Could Fail Safely

A successful first deployment would not prove much. I needed an application that could help test both healthy and unhealthy releases without risking real data.

I built a small Node.js service that could answer four simple questions:

/health   Is the program alive?
/ready    Should Kubernetes send it user traffic?
/version  Which source version is running?
/metrics  How many requests has it handled?

The health and readiness checks were deliberately separate. This allowed me to create a safe bad release: the program would keep running, but its readiness check would fail, so Kubernetes would refuse to send new traffic to it.

The version response would help trace one change from source code, through the built image and Git, to the running application.

I Kept Argo CD Private

My home server already ran other services and had limited free memory. I did not need another public management screen.

I chose the smaller Argo CD Core installation. Its management components would remain reachable only inside the cluster, and I gave them explicit CPU and memory limits.

Argo CD still needed meaningful power inside Kubernetes to do its job. Keeping it private did not make it harmless. I therefore limited this application to its repository, its Kubernetes namespace, and the types of resources its Helm package created.

I Defined the Failure Tests Before Claiming Success

A green installation screen would only prove that the tools started. I wrote the tests before implementing the system:

Test	Question it had to answer
Follow one release	Can I connect the source change to the built image, the version recorded in Git, and the running application?
Change the cluster by hand	Does Argo CD notice that the cluster no longer matches Git and correct it?
Approve a bad readiness setting	What happens when Git is followed correctly but the chosen configuration is harmful?
Revert the Git change	Can I recover by correcting the durable record instead of editing only the cluster?
Time a normal release	How long does a reviewed version take to become healthy?
Measure resources	Does Argo CD fit on the shared home server without creating pressure?

The two failure cases were intentionally different.

Changing the cluster by hand would make the cluster wrong while Git remained correct. Argo CD should restore the version recorded in Git. This disagreement is called drift.

The readiness experiment would test the opposite problem: Git itself would contain a harmful setting. Argo CD should apply it faithfully, while Kubernetes should report that the new copy was not ready. Recovery would require reverting the Git change.

That distinction became the most important result of the project.

What the Design Did Not Promise

This design did not make Git automatically correct. A reviewer could still approve a bad value, and Argo CD could apply it perfectly. Argo CD also still needed permission to change the cluster.

The single server lab could not prove high availability. One successful request also could not prove that no user request was ever lost.

The goal was smaller and testable: keep direct cluster access out of application CI, record the exact selected release in Git, and observe how the system behaved when either the cluster or Git was wrong.

What Came Next

The rollback in my first article taught me that recovering the running application and correcting its saved instructions were separate jobs. This design joined them through one reviewable path.

Part 2
implements that path. It follows one release from CI to Git and into Kubernetes, then records what happened when I changed the cluster by hand, approved a bad readiness setting, and recovered by reverting Git.

The complete implementation and sanitized evidence are in the public
pull-based-kubernetes-delivery repository.

I Turned a $7 Lightsail VM Into a Staging Server

Lalit Bagga — Thu, 20 Aug 2026 19:37:38 +0000

My side projects had a staging gap.

Local Docker was useful for checking an application before a release, but it still ran on my laptop. A larger AWS setup with a VPC, load balancer, container platform, and deployment pipeline would answer a different question and create much more infrastructure than I needed.

I wanted something in the middle: one server with a stable address, controlled access, repeatable deployment, a health check, and enough recovery behavior to be useful during development.

So I tested Amazon Lightsail as a small staging environment instead of treating it as a simplified hosting tutorial.

The Smallest Useful Staging Stack

I deliberately kept the architecture narrow:

Developer laptop
  ├── Terraform → Lightsail instance, SSH key, static IP, firewall
  └── deploy.sh → SSH/SCP → Docker Compose
                              └── Node.js app
                                   ├── /health
                                   └── /version

The instance ran Ubuntu 24.04 in ca-central-1 on the Micro 1 GB Linux plan with public IPv4. AWS currently lists that bundle at $7 per month, billed hourly up to the monthly maximum.

There was no database, load balancer, Kubernetes cluster, or CI/CD system. The point was to learn whether one developer could operate a useful staging VM without quietly building a small platform team around it.

Step 1: Define More Than an Instance

The Terraform resource for the VM was only one part of the environment:

resource "aws_lightsail_instance" "lightsail_instance" {
  name              = "${var.project_name}-lightsail-instance"
  availability_zone = var.availability_zone
  blueprint_id      = var.blueprint_id
  bundle_id         = var.bundle_id
  key_pair_name     = aws_lightsail_key_pair.lightsail_instance_key.name
  user_data         = file("${path.module}/../scripts/bootstrap.sh")
}

I also needed a custom SSH public key, a static IP, the attachment connecting that address to the instance, and explicit firewall rules.

The static IP was not just extra Terraform. Lightsail's default public IPv4 address can change when an instance is stopped and started. A staging endpoint is much easier to use when its address stays stable, and Lightsail lets the same static address move to a replacement instance later.

The SSH key had a similarly clear boundary. Terraform uploaded only the public key:

resource "aws_lightsail_key_pair" "lightsail_instance_key" {
  name       = "${var.project_name}-key"
  public_key = file(pathexpand(var.ssh_public_key_path))
}

The private key never entered Terraform, the repository, or the article.

Step 2: Replace the Default Firewall Rules

A base Lightsail Linux instance can start with SSH on port 22 and HTTP on port 80 open to every address. That is convenient for a first connection, but it was wider than this experiment required.

I replaced those defaults with two rules restricted to my temporary public /32 address:

resource "aws_lightsail_instance_public_ports" "lightsail_instance_firewall" {
  instance_name = aws_lightsail_instance.lightsail_instance.name

  port_info {
    protocol  = "tcp"
    from_port = 22
    to_port   = 22
    cidrs     = [var.developer_cidr]
  }

  port_info {
    protocol  = "tcp"
    from_port = 80
    to_port   = 80
    cidrs     = [var.developer_cidr]
  }
}

The Lightsail firewall documentation recommends limiting SSH to the address that needs administrative access. I restricted HTTP as well because this was a private experiment, not a public site.

For a real shared staging environment, the source ranges and access model would need another review. A developer's changing home IP is manageable for a short test but awkward for a team.

Step 3: Make Application State Observable

The application was intentionally small. It used Node's built-in HTTP module and exposed two endpoints:

if (request.url === "/health") {
  return sendJson(response, 200, { status: "ok" });
}

if (request.url === "/version") {
  return sendJson(response, 200, { version });
}

/health answered whether the process was responding. /version answered whether the expected deployment was running.

Those endpoints were more useful than a generic home page because each test had a precise success condition.

Docker Compose added the recovery policy and its own health check:

services:
  app:
    build:
      context: .
    environment:
      APP_VERSION: ${APP_VERSION:-local}
    ports:
      - "${HOST_PORT:-8080}:3000"
    restart: unless-stopped
    healthcheck:
      test:
        - CMD
        - node
        - -e
        - >-
          require('node:http').get('http://127.0.0.1:3000/health',
          response => process.exit(response.statusCode === 200 ? 0 : 1))
          .on('error', () => process.exit(1))
      interval: 5s
      timeout: 3s
      retries: 6

The container also ran as a non-root application user. No application secret was required, and the generated .env file on the server had mode 0600.

Step 4: Keep Deployment Separate From Terraform

Terraform created the server. A separate deployment script copied the application files over SSH and ran Compose:

./scripts/deploy.sh "$host" v2 "<SSH_PRIVATE_KEY_PATH>"

On the server, the important command was:

sudo docker compose up --build --detach --remove-orphans

That separation mattered. Changing v1 to v2 did not require replacing the instance or changing Terraform state.

The verification script then checked both endpoints until they agreed with the requested release:

./scripts/verify.sh "$host" v2 120

The observed result was:

PASS health=ok version=v2 recovery_seconds=0

The complete v1 to v2 deployment took 6 seconds, and the first verification request already returned the new version.

What the Runtime Tests Showed

Configuration alone does not prove that recovery works, so I tested two common situations.

First, I stopped the application process inside the container:

./scripts/test-container-restart.sh \
  "<HOST>" \
  "http://<HOST>" \
  "<SSH_PRIVATE_KEY_PATH>"

Docker Compose restarted the application. The health and version endpoints returned after 3 seconds:

PASS health=ok version=v1 recovery_seconds=3

Next, I rebooted the complete Lightsail server:

./scripts/test-instance-reboot.sh \
  "<HOST>" \
  v1 \
  lightsail-staging-lab-lightsail-instance \
  "<AWS_PROFILE>" \
  ca-central-1

The application returned without another deployment:

PASS health=ok version=v1 recovery_seconds=25
PASS reboot_recovery_seconds=29

The complete reboot test took 29 seconds. This confirmed that Docker started with the server and Compose brought the application back automatically.

What Lightsail Simplified

For this experiment, Lightsail removed a large amount of AWS setup.

I did not have to design a VPC, public subnet, internet gateway, route table, EC2 security group, or separate EBS volume. The instance bundle made the compute, memory, SSD storage, public addressing, and transfer allowance understandable as one small plan.

The static IP and firewall were also easier to reason about than a larger network stack. Terraform still made those decisions explicit, but there were fewer components to connect.

That simplicity made Lightsail a reasonable fit for one small staging server.

What Lightsail Still Leaves to the Developer

Lightsail simplified the AWS layer. It did not operate the application for me.

I still owned:

operating-system updates;
Docker installation and service startup;
application deployment and rollback;
logs and alerting;
SSH-key handling;
firewall source-address changes;
backups and data recovery; and
verification after every change.

It also remained one VM. There was no multi-zone availability, autoscaling, managed rollout controller, or automatic load-balancer health replacement.

I would not choose this setup for a system that needs private application tiers, multiple instances, team-scale access control, managed deployments, or strong availability guarantees. At that point, ECS, App Runner, Elastic Beanstalk, or a deliberately designed EC2 platform would deserve comparison.

Cost and Cleanup

The selected plan was $7 per month, but this experiment did not run for a month. AWS bills Lightsail instance bundles hourly up to the monthly maximum.

I did not collect final billing data, so I will not present an exact invoice. The resources existed for less than one day, making the theoretical instance-cost upper bound less than $0.24, before any free-tier discount. That is an estimate, not a billing record.

The experiment ended with:

terraform -chdir=terraform destroy \
  -var-file=terraform.tfvars

Terraform removed the instance, firewall configuration, static-IP attachment, static IP, and Lightsail key pair. A final state check returned no managed resources. The project created no snapshots, extra disks, load balancer, database, DNS zone, or Lightsail container service.

What I Would Use It For

Lightsail passed the part of the experiment that mattered most to my daily development workflow.

It gave me a small remote staging server with controlled access, a stable endpoint, fast redeployment, observable health, automatic process recovery, and reboot recovery. Terraform kept the AWS resources understandable, while Docker Compose kept application deployment separate.

I would use this pattern for a personal project, a short-lived demo, or a small integration environment where one VM is an acceptable availability tradeoff.

The next validation is narrower: run the final portable launch script through a completely unattended replacement and retain that result. Until then, this is a tested staging server pattern, not a production platform and not a claim of complete disaster recovery.

I Wanted Kubernetes Without an Always On EKS Bill

Lalit Bagga — Tue, 11 Aug 2026 23:52:08 +0000

I wanted a Kubernetes lab that would still be there the next morning.

A temporary cluster on my laptop would be useful for testing manifests, but it would disappear when I closed the lid. Amazon EKS would give me the managed AWS experience I eventually want to test, but keeping it online just so I could practise Deployments and Services did not make sense for me. The EKS control plane alone is currently priced at $0.10 per cluster-hour under standard support, before worker nodes, storage, and networking.

I already had another option: an Intel N100 home server with 16 GiB of memory running Debian and OpenMediaVault.

It was already always on. It was also already busy.

At the time of my retained snapshot, the server was running 28 Docker containers. Kubernetes had to fit beside those workloads, not take over the machine. That constraint shaped the entire decision.

The Lab I Actually Needed

My goal was not to reproduce a production cluster at home. Four CPU cores cannot provide a convincing multi-node failure lab while also protecting the services already using the server.

I needed something narrower:

Existing Debian/OpenMediaVault server
        ├── Existing Docker workloads
        └── Single-node Kubernetes lab
                ├── Always available
                ├── Privately administered
                └── Small enough to share the host

The home server would be my persistent learning environment. EKS could come later as a short-lived environment for AWS-specific behavior such as IAM, cloud load balancers, and managed control-plane operations.

That made the selection criteria practical:

It had to run directly on Debian.
It had to support a useful single-node cluster.
I needed control over the resources left for Kubernetes.
Bundled components could not claim ports already used by the server.
The setup had to remain close enough to Kubernetes that the manifests could move to EKS later.

Why k3s Fit the Server

I considered MicroK8s, k0s, and EKS, but k3s matched this host and the way I wanted to operate the lab.

MicroK8s has a convenient add-on workflow and is a natural choice when Ubuntu and snap are already part of the environment. k0s is also lightweight and packages Kubernetes as a single binary with a configurable architecture.

Both were reasonable options. My server was already Debian-based, and I wanted a small native service with straightforward configuration and a large body of homelab examples. k3s gave me that operating model without asking me to rebuild the host around a different workflow.

The official k3s requirements list two CPU cores and 2 GB of memory for a server node. A single-server cluster can use embedded SQLite by default, while other datastore options remain available if the topology grows later.

On this server, the installed k3s binary measured about 78.2 MiB. More important than the number was what I could configure around it: resource reservations, eviction thresholds, and the bundled components I wanted to disable.

That was the real reason I selected k3s. It was not simply “lightweight Kubernetes.” It gave me a Kubernetes control plane I could shape around a server that already had a job.

Protecting a Server That Was Already Busy

Running the k3s installer was the easy part. The real risk was forgetting that this server already had a job.

Before installing anything, I wrote a host-audit script. It records CPU, memory, swap, disk space, listening ports, systemd services, Docker containers, and recent kernel warnings. This gave me a clear picture of the server before Kubernetes touched it.

The raw audit stays outside Git because it may contain hostnames, IP addresses, mount paths, and internal service names. Only reviewed and sanitized results belong in the repository.

I also made the installation script stop when it finds something unexpected. It refuses to continue when:

it is not running with the required privileges;
k3s is already running;
an existing k3s installation or configuration is present; or
the Kubernetes API or kubelet ports are already occupied.

The k3s configuration was deliberately conservative:

write-kubeconfig-mode: "0640"
write-kubeconfig-group: "users"
disable:
  - traefik
  - servicelb
node-label:
  - "environment=homelab"
kubelet-arg:
  - "system-reserved=cpu=1,memory=6Gi"
  - "kube-reserved=cpu=250m,memory=512Mi"
  - "eviction-hard=memory.available<1Gi,nodefs.available<10%"

Ports 80 and 443 were already being used by services on the server. Because the bundled Traefik ingress controller and ServiceLB also expect to use those ports, I disabled both components.

The resource settings were equally important. Kubernetes was told to leave one CPU core and 6 GiB of memory available for the host. I also allowed another 250 millicores and 512 MiB for Kubernetes itself.

These settings are not a hard wall around the existing Docker containers. They simply reduce the resources Kubernetes can assign to Pods and give the kubelet thresholds for reacting to memory or disk pressure.

That leaves less capacity for the lab, but that is intentional. On a shared four-core server, protecting the existing workload matters more than giving Kubernetes every available resource.

Installing and Checking the Cluster

Once the checks passed and the configuration was written, the script used the official k3s installer:

curl --proto '=https' --tlsv1.2 -sfL https://get.k3s.io | sh -

It then waited for the node to report that it was ready:

k3s kubectl wait \
  --for=condition=Ready \
  node --all \
  --timeout=180s

I kept installation and verification separate. A successful installer only proves that the installation command completed. It does not prove that the cluster is healthy or that the rest of the server survived the change.

The verification script checks:

the k3s systemd service;
the installed k3s version;
the Kubernetes node and Pods;
the resources available to workloads; and
whether Traefik or ServiceLB appeared unexpectedly.

The installed server was running k3s v1.36.3+k3s1. At the time of the retained snapshot:

Kubernetes could allocate 2750 millicores of CPU to Pods;
about 7.9 GiB of memory was allocatable to Pods;
all six observed Kubernetes Pods were running;
all 28 existing Docker containers were still running;
none of the Docker containers reported being unhealthy; and
no systemd unit had failed.

The k3s service was using approximately 726 MiB of memory, with a recorded peak of 1.19 GiB. The host still had about 9.7 GiB of available memory and 296 GiB of available disk space.

One warning remained: 834 MiB of the server's 976 MiB swap was in use. That does not automatically mean the server was under active memory pressure, but it was something I still needed to investigate.

These measurements are a snapshot, not a performance benchmark. They do not prove that the server will remain healthy under every workload.

They answer a smaller and more useful question: after adding k3s, was the cluster running without an obvious failure in the services already sharing the machine?

At that moment, yes.

Then I Made Kubernetes Do Real Work

A ready node proves that Kubernetes started. It does not prove that I can operate an application through it.

I deployed a small Nginx workload with three replicas, readiness and liveness probes, and CPU and memory constraints. A Service gave the replaceable Pods one stable internal address, and NodePort exposed the same application over my private network.

Private client
      ↓
Home server:30081
      ↓
Kubernetes Service
      ↓
EndpointSlice
      ↓
Three Nginx Pods

I verified the Service from inside the cluster:

klab -n lab exec deployment/hello-nginx -- \
  wget -qO- http://hello-nginx:80

The request returned the expected page and identified the Pod that served it. The same application was reachable through the private NodePort path.

I then changed the page and watched the Deployment replace the Pods through a rolling update. Kubernetes created a new ReplicaSet, scaled it up, and scaled the previous one down while the Service remained stable.

Finally, I rolled the Deployment back and reconciled the YAML with the running version. A clean kubectl diff confirmed that the source and live cluster agreed again.

That sequence mattered more to me than a successful installer message. The server was now a persistent place to practise the controller relationships I wanted to understand: Deployment to ReplicaSet, Service to EndpointSlice, rollout to rollback, and desired state to live state.

Why EKS Still Belongs in the Plan

Choosing k3s did not remove EKS from the journey. It gave EKS a more focused role.

According to the current EKS pricing page, a standard-support cluster costs $0.10 per hour before the compute and supporting AWS resources are added. That is roughly $73 for a 730-hour month for the control plane alone.

I do not need to pay that continuously while learning what a ReplicaSet does or why a Service follows labels. The home k3s cluster can hold that state and remain available every day.

When I need to verify AWS-specific behavior, I can create EKS deliberately, capture the evidence, and remove it. The local cluster handles repetition; the cloud cluster handles cloud-specific proof.

What Comes Next

The first application lab exposed the next platform problem.

I rolled Kubernetes back to Version 2, but my YAML still declared Version 3. The cluster and source disagreed, and Kubernetes alone had no reason to repair that relationship.

The next milestone is GitOps with Argo CD: make Git the declared source of truth, reproduce the drift deliberately, and observe reconciliation instead of correcting it by hand.

That is where this stops being only a Kubernetes lab and starts becoming a delivery platform.

My Terraform Drift Pipeline Fixed the Change, Then Forgot It

Lalit Bagga — Fri, 07 Aug 2026 00:51:46 +0000

My Terraform drift pipeline could detect a manual EC2 tag change, classify it as LOW, and run Terraform to remove it.

Then the pipeline moved on.

The evidence existed, but it was spread across CodeBuild output, Lambda logs, and an SNS message. If I wanted to know what changed, how it was classified, and whether remediation started, I had to reconstruct the event from multiple AWS services.

The pipeline could act on drift. It could not remember drift.

Phase 4 added that memory: a durable DynamoDB record, a read only API, and a small dashboard that turns the event history into something I can inspect without opening three AWS consoles.

The Stack

Terraform drift event
        ↓
SNS
        ↓
Severity Lambda
        ├── classifies HIGH / MEDIUM / LOW
        ├── starts remediation for eligible LOW drift
        └── writes the audit event to DynamoDB
                            ↓
                    API Gateway HTTP API
                            ↓
                    Read only Lambda
                            ↓
                    DynamoDB Query
                            ↓
             CloudFront → static dashboard
                            ↑
                      private S3 bucket

The browser receives static HTML, CSS, and JavaScript from CloudFront. JavaScript calls API Gateway, the API Lambda queries DynamoDB, and the returned JSON becomes the live dashboard.

There is no EC2 web server and no application process running continuously.

Step 1: Store Every Classified Event

I created a DynamoDB table with a composite key:

resource "aws_dynamodb_table" "drift_events" {
  name         = "terraform-drift-events"
  billing_mode = "PAY_PER_REQUEST"
  hash_key     = "project"
  range_key    = "timestamp"

  attribute {
    name = "project"
    type = "S"
  }

  attribute {
    name = "timestamp"
    type = "S"
  }
}

project groups the history for one Terraform project. The ISO 8601 timestamp orders its events.

DynamoDB only requires attribute definitions for keys and indexes. Fields such as high_count, changes, and status still belong in each item, but they do not belong in the table schema block.

I passed the table name into the existing severity Lambda instead of putting it directly in the code:

environment {
  variables = {
    DRIFT_EVENTS_TABLE       = aws_dynamodb_table.drift_events.name
    REMEDIATION_PROJECT_NAME = aws_codebuild_project.remediation.name
  }
}

The same Lambda role received permission to write to that specific table:

{
  Effect = "Allow"
  Action = [
    "dynamodb:PutItem",
    "dynamodb:UpdateItem",
    "dynamodb:GetItem",
    "dynamodb:Query",
    "dynamodb:Scan",
  ]
  Resource = aws_dynamodb_table.drift_events.arn
}

After classification, Lambda builds one audit item:

item = {
    "project": project,
    "timestamp": timestamp,
    "drift_count": len(changes),
    "high_count": len(classified["HIGH"]),
    "medium_count": len(classified["MEDIUM"]),
    "low_count": len(classified["LOW"]),
    "changes": classified,
    "action_taken": action_taken,
    "status": status,
}

if remediation_build_id:
    item["remediation_build_id"] = remediation_build_id

drift_table.put_item(Item=item)

The record stores the decision alongside the evidence used to make it. I do not need to correlate an SNS payload with a separate classification log just to understand one event.

For a change that needs human review, the item records manual_review. When Lambda starts the remediation build, it records remediation_triggered and includes the CodeBuild build ID.

That turns the audit item into the connection between detection, classification, and the action taken by the pipeline.

Step 2: Add a Read Only History API

The dashboard should not receive DynamoDB credentials. It calls an HTTP API instead.

I created a second Lambda whose only data permission is:

Action   = ["dynamodb:Query"]
Resource = aws_dynamodb_table.drift_events.arn

The route requires a project:

GET /drift-history?project=Three-Tier-Infra

The Lambda queries the partition key and returns newest events first:

response = table.query(
    KeyConditionExpression="#project = :project",
    ExpressionAttributeNames={"#project": "project"},
    ExpressionAttributeValues={":project": project},
    ScanIndexForward=False,
)

This is why the table uses project as its partition key. The API does not need to scan the complete table to retrieve one project's history.

DynamoDB numbers arrive in Python as Decimal, which json.dumps cannot serialize directly. I added a small serializer:

def json_serializer(value):
    if isinstance(value, Decimal):
        return int(value) if value % 1 == 0 else float(value)
    raise TypeError(f"Cannot serialize {type(value)}")

API Gateway uses an HTTP API with a Lambda proxy integration:

resource "aws_apigatewayv2_route" "get_drift_events" {
  api_id    = aws_apigatewayv2_api.drift_api.id
  route_key = "GET /drift-history"
  target    = "integrations/${aws_apigatewayv2_integration.drift_api.id}"
}

Calling the deployed route returned HTTP 200 and the stored event:

curl \
  "https://8zg1x51wne.execute-api.us-east-2.amazonaws.com/drift-history?project=Three-Tier-Infra"

The response contained:

{
  "events": [
    {
      "project": "Three-Tier-Infra",
      "timestamp": "2026-07-26T15:29:32Z",
      "drift_count": 1,
      "high_count": 0,
      "medium_count": 0,
      "low_count": 1,
      "action_taken": "remediation_triggered",
      "status": "REMEDIATION_STARTED"
    }
  ]
}

The changed resource was:

module.compute.aws_instance.bastion_host

Terraform reported it as an aws_instance update.

Step 3: Build a Dashboard Without a Web Server

The dashboard is static in how it is hosted, not in the data it displays.

S3 stores three files:

dashboard/
├── index.html
├── app.js
└── styles.css

CloudFront serves those files over HTTPS. app.js calls the API whenever the page loads or the user refreshes the data.

The dashboard shows total events, severity counts, detection timestamps, Terraform resource addresses, actions, and remediation status. A severity filter narrows the history, and the page refreshes automatically every 60 seconds.

The S3 bucket remains private. CloudFront uses Origin Access Control to read objects:

resource "aws_cloudfront_origin_access_control" "drift_dashboard" {
  name                              = "terraform-drift-dashboard"
  origin_access_control_origin_type = "s3"
  signing_behavior                  = "always"
  signing_protocol                  = "sigv4"
}

The bucket policy grants s3:GetObject to the CloudFront service only when the request comes from this distribution.

Step 4: Keep the Public Surface Read Only

I deliberately did not add a Fix drift button.

The first dashboard is for visibility. It cannot start CodeBuild, run Terraform, or change DynamoDB records.

The API exposes one GET route. I added API Gateway throttling:

default_route_settings {
  throttling_burst_limit = 20
  throttling_rate_limit  = 10
}

I also restricted browser CORS responses to the deployed CloudFront origin:

cors_configuration {
  allow_headers = ["content-type"]
  allow_methods = ["GET"]
  allow_origins = [
    "https://${aws_cloudfront_distribution.drift_dashboard.domain_name}"
  ]
  max_age = 3600
}

CORS is not authentication. Someone who knows the API URL can still call it outside a browser.

That is an accepted limitation for this public dashboard. The API returns resource addresses, classifications, and actions. It does not return raw Terraform state, credentials, plan values, or CloudWatch logs.

An operator remediation API would require authentication, authorization, review, concurrency controls, and an audit record of who approved the action. It does not belong in this first dashboard.

Verification

I created real drift by changing the tag on:

module.compute.aws_instance.bastion_host

The detection pipeline reported one LOW update. Lambda started remediation and wrote this event to DynamoDB:

project:              Three-Tier-Infra
drift_count:          1
high_count:           0
medium_count:         0
low_count:            1
action_taken:         remediation_triggered
status:               REMEDIATION_STARTED

The record also contained the remediation CodeBuild ID and the classified resource details.

After remediation, I returned to the EC2 instance and confirmed the tag was restored to the value declared in Terraform.

I then tested the deployed layers independently:

CloudFront index.html: HTTP 200
CloudFront app.js:     HTTP 200
API history query:     HTTP 200
CORS origin:           CloudFront dashboard URL

The API response to the dashboard origin included:

access-control-allow-origin: https://dqjunwd8v0pry.cloudfront.net

The complete verified path was:

EC2 tag changed
→ Terraform detected one update
→ Lambda classified it LOW
→ Lambda wrote the event to DynamoDB
→ remediation restored the tag
→ API returned the audit event
→ dashboard displayed the history

The DynamoDB status remains REMEDIATION_STARTED. That proves CodeBuild accepted the build request, not that the build ultimately succeeded. I verified success separately by observing the restored EC2 tag.

Updating the audit item from REMEDIATION_STARTED to REMEDIATION_SUCCEEDED is separate completion tracking work.

What Comes Next

The system can now detect drift, classify it, remediate the tested LOW tag scenario, remember the event, expose it through an API, and display it in a dashboard.

The next step is making the audit lifecycle more complete. The current item proves that CodeBuild accepted the remediation request, but it does not record the final build result.

Tracking that result would let the same event move from REMEDIATION_STARTED to REMEDIATION_SUCCEEDED or REMEDIATION_FAILED.

I Added a Tag in AWS. Terraform Removed It Automatically.

Lalit Bagga — Tue, 04 Aug 2026 22:30:26 +0000

I opened the AWS console, selected an EC2 instance, added a tag, and saved it.

The tag did not exist in Terraform. The deployed infrastructure now said one thing while the code said another.

That was the test.

My drift detector already knew how to find the difference and classify the EC2 update as LOW. The missing part was closing the loop: once the system understood that the change was low severity, it needed to restore the infrastructure without waiting for me to run Terraform manually.

So I added a separate remediation pipeline.

The Stack

Manual EC2 tag added in AWS
        ↓
Drift CodeBuild runs terraform plan
        ↓
SNS publishes the structured change
        ↓
Lambda classifies the update as LOW
        ↓
Remediation CodeBuild runs terraform apply
        ↓
Terraform removes the unmanaged tag

Detection and remediation use different CodeBuild projects. The detector reads the infrastructure and reports differences. The remediation project has a separate IAM role because it can change AWS resources.

Step 1: Make EC2 Updates Eligible for Remediation

The classifier already grouped resource types into HIGH, MEDIUM, and LOW categories.

An EC2 instance belongs to MEDIUM_RISK_TYPES. Initially, that meant even a tag update was classified as MEDIUM and stopped before remediation.

The resource type alone was not enough. The action mattered too.

I updated the classifier so an in-place update on a MEDIUM resource becomes LOW, while creates remain MEDIUM and delete or replacement actions become HIGH:

def classify_change(resource_type, actions):
    if resource_type in HIGH_RISK_TYPES:
        return "HIGH"
    elif resource_type in MEDIUM_RISK_TYPES:
        if "delete" in actions or "replace" in actions:
            return "HIGH"
        elif "update" in actions:
            return "LOW"
        return "MEDIUM"
    else:
        return "LOW"

For this test, Terraform reported the manually added EC2 tag as an update. That moved it into the LOW path.

Step 2: Keep Deletions Out of the Automatic Path

LOW did not automatically mean “start Terraform.” I also filtered out deletions:

low_changes_to_remediate = [
    change for change in classified["LOW"]
    if "delete" not in change.get("actions", [])
]

This distinction matters. Updating an existing resource to match the code is different from recreating something that a person deliberately removed. Deletions stay in the manual-review path.

If an eligible LOW change remains, Lambda starts the remediation CodeBuild project:

if low_changes_to_remediate:
    codebuild.start_build(
        projectName=os.environ.get(
            "REMEDIATION_PROJECT_NAME",
            "terraform-drift-remediation",
        )
    )

The Lambda role only needs permission to start that specific project:

{
  Effect   = "Allow"
  Action   = ["codebuild:StartBuild"]
  Resource = aws_codebuild_project.remediation.arn
}

Lambda makes the routing decision. CodeBuild performs the repair.

Step 3: Run Remediation Separately

I did not add terraform apply to the existing drift detector. Detection should remain read-only even when remediation fails or is disabled.

The second build starts in a clean environment, installs Terraform, clones the Three-Tier repository, and applies the declared configuration:

version: 0.2

phases:
  install:
    commands:
      - curl -o terraform.zip https://releases.hashicorp.com/terraform/1.10.0/terraform_1.10.0_linux_amd64.zip
      - unzip terraform.zip -d /usr/local/bin
  pre_build:
    commands:
      - git clone https://github.com/lalitbagga/Three-Tier-Infra.git /tmp/Three-Tier-Infra
      - cd /tmp/Three-Tier-Infra
      - terraform init
  build:
    commands:
      - cd /tmp/Three-Tier-Infra
      - terraform apply -auto-approve -lock=false

The remediation project uses its own CodeBuild role with write access to the AWS services managed by the Three-Tier project. During implementation, Terraform exposed several missing permissions as it evaluated and changed resources, so I expanded the experimental role to cover the project.

That role is intentionally separate from the detector. A job that only runs terraform plan should not inherit the permissions required for terraform apply.

The Repository Was Part of the Control Loop

One remediation run tried to include unrelated resource actions because CodeBuild was cloning a version of the Three-Tier repository that did not match the configuration I intended to operate.

The problem was not the classifier. Lambda correctly identified the triggering drift. The remediation build was applying the complete repository it cloned from GitHub.

I pushed the current Three-Tier configuration before running the test again.

This is easy to miss in CI. Terraform does not know what exists only on a developer's laptop. If CodeBuild clones GitHub, the committed branch is the configuration Terraform will enforce.

The Tag Disappeared

With the current repository in GitHub, I repeated the test:

Opened the EC2 instance in the AWS console.
Added a tag manually and saved it.
Started the Terraform drift detector.
The plan reported an update to the EC2 instance.
Lambda classified the update as LOW.
Lambda started the remediation CodeBuild project.
CodeBuild ran Terraform apply.
I returned to the EC2 Tags view and confirmed that the manually added tag was gone.

Terraform restored the infrastructure to what was declared in Git.

Git configuration: tag absent
AWS resource:       tag added manually
Terraform result:   tag removed

The system was no longer only telling me about drift. For the LOW update I tested, it detected the change, classified it, and repaired it.

MEDIUM, HIGH, and deletion paths still stop for human review. That boundary is intentional. Automatic remediation is useful only when the system can distinguish a small correction from a change that deserves a person.

What Comes Next

The infrastructure is back in the expected state, but the evidence is spread across CodeBuild and CloudWatch logs.

The next part of the system will keep a remediation history in a database, expose it through an API, and visualize it in Grafana. I want to see what changed, how it was classified, whether remediation ran, and whether it succeeded without reconstructing the story from separate AWS services.

Detection found the tag. Classification decided it was LOW. Remediation removed it.

Now the system needs to remember that it happened.

My Terraform Drift Alert Could Not Explain the Drift

Lalit Bagga — Wed, 22 Jul 2026 02:40:34 +0000

My Terraform drift detector had a problem.

It could run terraform plan, detect drift, and send me an email through SNS. But the message only told me that drift existed.

It did not tell me which resources changed. It did not tell me whether the change affected IAM, networking, or something less critical.

So I changed the alert into a structured event and sent it to two places: SQS for audit processing and Lambda for severity classification.

The Stack

CodeBuild                  → runs terraform plan
        ↓
terraform show -json       → extracts the changed resources
        ↓
SNS                        → publishes one structured event
        ↓
        ├── SQS            → keeps the event for audit processing
        └── Lambda         → classifies changes as HIGH, MEDIUM, or LOW
                 ↓
            CloudWatch     → stores the classification logs

Step 1: Publish Structured Drift Details

The build already saved the Terraform plan:

terraform plan -detailed-exitcode -out=plan.tfplan -lock=false || EXIT_CODE=$?

Exit code 2 means Terraform found changes. When that happens, I convert the saved plan to JSON:

terraform show -json plan.tfplan > /tmp/plan.json

Terraform plan JSON is large. SNS and Lambda do not need all of it, so I used jq to keep only the useful fields:

DRIFT_CHANGES=$(jq -c '[
  .resource_changes[] |
  select(.change.actions != ["no-op"]) |
  {
    address: .address,
    type: .type,
    actions: .change.actions
  }
]' /tmp/plan.json)

Each change now contains:

{
  "address": "module.compute.aws_iam_role.ec2_role",
  "type": "aws_iam_role",
  "actions": ["create"]
}

The important part here is what I left out. I did not publish the complete before-and-after resource values. Resource address, type, and action are enough for classification without sending the entire Terraform plan through SNS.

Step 2: Send the Event to SQS

I created an SQS queue called terraform-drift-audit and subscribed it to the SNS topic.

resource "aws_sqs_queue" "drift_audit" {
  name = "terraform-drift-audit"
}

resource "aws_sns_topic_subscription" "sqs" {
  topic_arn = aws_sns_topic.drift.arn
  protocol  = "sqs"
  endpoint  = aws_sqs_queue.drift_audit.arn
}

I also added the queue URL as a Terraform output:

output "sqs_queue_url" {
  description = "URL of the drift audit SQS queue"
  value       = aws_sqs_queue.drift_audit.url
}

After terraform apply, I can retrieve the URL without opening the AWS console:

terraform output -raw sqs_queue_url

That URL is what I used with the AWS CLI to check whether SNS delivered the message.

SQS also needs permission to receive messages from SNS. I restricted the queue policy to the drift topic instead of allowing every SNS topic:

Condition = {
  ArnEquals = {
    "aws:SourceArn" = aws_sns_topic.drift.arn
  }
}

SQS is not the final audit database. It keeps the event available for a future consumer. Permanent drift history belongs to a later phase.

The Queue Existed in the Wrong Region

I copied the queue URL into this command:

aws sqs receive-message \
  --queue-url https://sqs.us-east-2.amazonaws.com/<account-id>/terraform-drift-audit \
  --max-number-of-messages 10

AWS returned NonExistentQueue.

The queue existed. My AWS CLI was using us-east-1, while the Terraform project created the queue in us-east-2.

The fix was adding the region explicitly to the same command:

aws sqs receive-message \
  --queue-url https://sqs.us-east-2.amazonaws.com/<account-id>/terraform-drift-audit \
  --max-number-of-messages 10 \
  --region us-east-2

The resource name was correct. The CLI was asking the wrong regional endpoint.

After fixing the region, I could see the structured drift message in SQS.

Step 3: Classify Every Change with Lambda

The Lambda function receives the same SNS event that went to SQS.

SNS does not send my drift JSON as the top-level Lambda event. It wraps the message inside Records[].Sns.Message. The function must first extract that string and parse it back into JSON.

This is the complete classifier:

import json
import os
from datetime import datetime, timezone

# Severity definitions
HIGH_RISK_TYPES = {
    "aws_security_group",
    "aws_security_group_rule",
    "aws_iam_role",
    "aws_iam_policy",
    "aws_iam_role_policy",
    "aws_iam_user",
    "aws_iam_group",
    "aws_iam_access_key",
}

MEDIUM_RISK_TYPES = {
    "aws_instance",
    "aws_db_instance",
    "aws_ecs_cluster",
    "aws_ecs_service",
    "aws_ecs_task_definition",
    "aws_lb",
    "aws_lb_listener",
    "aws_lb_target_group",
    "aws_nat_gateway",
    "aws_route_table",
    "aws_network_acl",
}


def classify_change(resource_type, actions):
    """Classify a single resource change by severity."""
    if resource_type in HIGH_RISK_TYPES:
        return "HIGH"
    elif resource_type in MEDIUM_RISK_TYPES:
        # Delete or replace is more concerning than create
        if "delete" in actions or "replace" in actions:
            return "HIGH"
        return "MEDIUM"
    else:
        return "LOW"


def lambda_handler(event, context):
    """Process SNS message with drift details and classify severity."""
    print(f"Received event: {json.dumps(event)}")

    for record in event.get("Records", []):
        # SNS message is in the SNS record
        sns_message = record.get("Sns", {}).get("Message", "{}")

        try:
            drift_data = json.loads(sns_message)
        except json.JSONDecodeError:
            print(f"Failed to parse SNS message: {sns_message}")
            continue

        timestamp = drift_data.get(
            "timestamp",
            datetime.now(timezone.utc).isoformat()
        )
        project = drift_data.get("project", "unknown")
        changes = drift_data.get("changes", [])

        # Classify each change
        classified = {"HIGH": [], "MEDIUM": [], "LOW": []}

        for change in changes:
            address = change.get("address", "unknown")
            resource_type = change.get("type", "unknown")
            actions = change.get("actions", [])

            severity = classify_change(resource_type, actions)
            classified[severity].append({
                "address": address,
                "type": resource_type,
                "actions": actions,
            })

        # Build summary
        summary = {
            "timestamp": timestamp,
            "project": project,
            "total_changes": len(changes),
            "high_count": len(classified["HIGH"]),
            "medium_count": len(classified["MEDIUM"]),
            "low_count": len(classified["LOW"]),
            "classified": classified,
        }

        # Log the classified drift
        print(
            f"Drift classification: {json.dumps(summary, indent=2)}"
        )

        # TODO Phase 3: Auto-remediate HIGH severity changes
        # TODO Phase 3: Alert on MEDIUM severity changes

    return {"statusCode": 200, "body": "Drift classified"}

The code does four things:

Extracts the SNS message from the Lambda event.
Parses the structured drift JSON.
Classifies each resource by type and action.
Writes the complete summary to CloudWatch Logs.

IAM and security group resources are HIGH. Resources such as EC2, RDS, ECS, load balancers, NAT gateways, and route tables start as MEDIUM. Everything else starts as LOW.

A delete or replacement on a MEDIUM resource becomes HIGH. This classifier is deliberately simple. It is a deterministic first version that I can read, test, and improve later.

The Lambda Package Needed Another Provider

Terraform packages the Python file using archive_file:

data "archive_file" "severity_lambda" {
  type        = "zip"
  source_file = "${path.module}/lambda/severity_classifier.py"
  output_path = "${path.module}/lambda/severity_classifier.zip"
}

The first run failed because the archive provider was missing from .terraform.lock.hcl.

The fix was:

terraform init -upgrade

That downloaded hashicorp/archive and updated the lock file.

The generated ZIP does not need to be committed. Terraform creates it when the configuration runs.

Verification

I triggered a real drift event and checked both destinations from the terminal.

First I read up to 10 messages from SQS:

aws sqs receive-message \
  --queue-url https://sqs.us-east-2.amazonaws.com/<account-id>/terraform-drift-audit \
  --max-number-of-messages 10 \
  --region us-east-2

The structured message appeared in the SQS queue. Lambda received the event and wrote this summary to CloudWatch:

aws logs tail /aws/lambda/terraform-drift-severity \
  --follow \
  --region us-east-2

--follow keeps the terminal attached to the log group. When the next drift event invokes Lambda, the classification appears directly in the terminal.

{
  "high_count": 8,
  "medium_count": 13,
  "low_count": 38
}

The HIGH results included an IAM role and an ALB security group:

module.compute.aws_iam_role.ec2_role
module.security.aws_security_group.alb_sg

Lambda classified all 59 resource changes reported by that Terraform plan. This confirmed that the structured SNS event reached Lambda and that each change was assigned a severity.

What Is Next

The system can now tell me what changed and how serious each change might be.

It still does not remediate anything.

That is intentional. The next step is deciding what should happen after classification. I will start with LOW-severity updates that are safe to test automatically. MEDIUM, HIGH, and deletion paths will remain outside automatic remediation and will later require human review.

That is Phase 3.

The Security Group Change Terraform Never Told Me About

Lalit Bagga — Wed, 15 Jul 2026 14:06:02 +0000

Terraform had a blind spot in my three-tier project.

The code could create the infrastructure. The state lived in S3. But if someone changed a security group in the AWS console, Terraform would not tell me when it happened.

I would discover it the next time I ran terraform plan.

That could be tomorrow or six months later.

So I built a drift detector that runs the same plan inside CodeBuild and sends an SNS email when Terraform finds a difference.

I made CodeBuild the entry point. One run compares the deployed infrastructure with Terraform and reports any difference by email.

The Stack

CodeBuild run                 → starts the detector
        ↓
CodeBuild                     → installs Terraform and clones the repo
        ↓
terraform plan                → compares code, state, and AWS
        ↓
SNS                           → sends an email when drift is detected

Why I Used CodeBuild Instead of Lambda

My first idea was Lambda.

A scheduled function could run terraform plan, then SNS could send the result.

The problem was packaging Terraform. I could build a Lambda container image and store it in ECR, but that adds a Dockerfile, image builds, image updates, and another repository to maintain.

CodeBuild already does what this job needs: start a clean Linux environment, run a script, stream logs to CloudWatch, and exit.

For a Terraform plan job, that was the simpler choice.

Step 1: Build the CodeBuild Job

The CodeBuild project does not build application source. Its source type is NO_SOURCE, and Terraform injects the buildspec directly:

resource "aws_codebuild_project" "drift" {
  name          = "terraform-drift"
  description   = "Runs terraform plan against Three-Tier-Infra to detect drift"
  build_timeout = 10
  service_role  = aws_iam_role.codebuild.arn

  artifacts {
    type = "NO_ARTIFACTS"
  }

  environment {
    compute_type = "BUILD_GENERAL1_SMALL"
    image        = "aws/codebuild/amazonlinux2-x86_64-standard:5.0"
    type         = "LINUX_CONTAINER"

    environment_variable {
      name  = "SNS_TOPIC_ARN"
      value = aws_sns_topic.drift.arn
    }
  }

  source {
    type      = "NO_SOURCE"
    buildspec = file("${path.module}/buildspec.yml")
  }

  logs_config {
    cloudwatch_logs {
      group_name = "/codebuild/terraform-drift"
    }
  }
}

Passing SNS_TOPIC_ARN as an environment variable keeps the generated ARN out of the buildspec. If Terraform recreates the topic, the next apply updates CodeBuild with the new ARN.

Step 2: Run Terraform Plan and Capture Drift

This was the buildspec used by the detector:

version: 0.2

phases:
  install:
    commands:
      - curl -o terraform.zip https://releases.hashicorp.com/terraform/1.10.0/terraform_1.10.0_linux_amd64.zip
      - unzip terraform.zip -d /usr/local/bin

  pre_build:
    commands:
      - git clone https://github.com/lalitbagga/Three-Tier-Infra.git /tmp/Three-Tier-Infra
      - cd /tmp/Three-Tier-Infra
      - terraform init

  build:
    commands:
      - cd /tmp/Three-Tier-Infra
      - EXIT_CODE=0
      - terraform plan -detailed-exitcode -out=plan.tfplan -lock=false || EXIT_CODE=$?
      - |
        if [ "$EXIT_CODE" -eq 2 ]; then
          echo "Drift detected. Publishing details to SNS."
          aws sns publish \
            --topic-arn "$SNS_TOPIC_ARN" \
            --subject "Terraform Drift Detected" \
            --message "Drift found in Three-Tier-Infra"
        elif [ "$EXIT_CODE" -eq 1 ]; then
          echo "Error during terraform plan."
          exit 1
        else
          echo "No drift detected."
        fi

The important part is -detailed-exitcode.

terraform plan -detailed-exitcode returns:

0 → plan succeeded, no changes
1 → Terraform failed
2 → plan succeeded, changes detected

The buildspec captures that result before CodeBuild decides whether the job succeeded:

EXIT_CODE=0
terraform plan -detailed-exitcode -out=plan.tfplan -lock=false || EXIT_CODE=$?

That gives the pipeline three clear paths: do nothing when the infrastructure matches, stop when Terraform fails, and notify when Terraform finds drift.

The SSH Key Only Existed on My Laptop

The Three-Tier project originally created its EC2 key pair from a local file:

public_key = file("~/.ssh/bastion-key.pub")

That worked on my machine. CodeBuild starts in an ephemeral container where that file does not exist.

I moved the public key to SSM Parameter Store and changed the Three-Tier repository:

data "aws_ssm_parameter" "bastion_key" {
  name = "/threeTier/bastionKeyPublic"
}

resource "aws_key_pair" "bastion_key" {
  key_name   = "bastion-key"
  public_key = data.aws_ssm_parameter.bastion_key.value
}

The CodeBuild role received ssm:GetParameter for the /threeTier/* path.

This removed the developer-laptop dependency. The same Terraform plan can now run locally or in CodeBuild.

Step 3: Send the Email Through SNS

The SNS topic and email subscription are managed by Terraform:

resource "aws_sns_topic" "drift" {
  name = "terraform-drift-alerts"
}

resource "aws_sns_topic_subscription" "drift" {
  topic_arn = aws_sns_topic.drift.arn
  protocol  = "email"
  endpoint  = var.alert_email
}

CodeBuild also needs sns:Publish permission scoped to that topic. CloudWatch permissions allow the build to create its log group and write the execution logs.

The role has four jobs:

S3 read              → read Terraform state
SSM read             → retrieve shared parameters
SNS publish          → send the drift alert
CloudWatch Logs      → record the build

It does not have S3 write permissions and it cannot apply Terraform changes.

Verification

I started the CodeBuild project directly:

aws codebuild start-build \
  --project-name terraform-drift \
  --region us-east-2

The command returns a build ID. Use it to inspect the result:

aws codebuild batch-get-builds \
  --ids <build-id> \
  --region us-east-2

The successful drift path produced three pieces of evidence:

CodeBuild completed the Terraform plan
        ↓
The build published to terraform-drift-alerts
        ↓
The subscribed email received "Terraform Drift Detected"

The detected change included the SSM refactor made in the Three-Tier repository.

That confirmed the complete scope of this version: start the detector, compare the infrastructure, identify drift, and deliver the result by email.

What Is Next

Running the detector on demand proved the idea, but an email that says "drift happened" is not enough.

The next phase converts the Terraform plan to JSON, sends the event to SQS, and uses Lambda to classify every change as HIGH, MEDIUM, or LOW.

EventBridge will be added in an upcoming blog.

Observability for ECS: Prometheus, Grafana, and Crash Testing

Lalit Bagga — Thu, 09 Jul 2026 14:10:47 +0000

CloudWatch logs tell you what happened after something goes wrong. They are reactive. What I wanted was something that shows me what is happening right now. CPU, memory, request rates, the shape of the system while it is running. That is observability, and CloudWatch alone is not enough for it.

So I added Prometheus and Grafana, ran a load test, and then deliberately crashed the app to watch ECS recover on its own.

The Stack

Node.js app on ECS Fargate  → exposes /metrics endpoint
        ↓
Prometheus on EC2            → scrapes /metrics every 15 seconds
        ↓
Grafana on EC2               → visualizes the data from Prometheus
        ↓
Uptime Robot                 → monitors public URL availability

Prometheus and Grafana run together on a t3.micro EC2 instance using Docker Compose. Free tier eligible. No managed services needed.

Step 1: Expose Metrics from the Node App

Prometheus works by scraping an HTTP endpoint that exposes metrics in a specific format. The app needs to provide that endpoint.

Install the Prometheus client library:

npm install prom-client

Add the metrics endpoint to index.js:

const express = require('express')
const client = require('prom-client')

const app = express()
const port = 3000

const collectDefaultMetrics = client.collectDefaultMetrics
collectDefaultMetrics()

app.get('/', (req, res) => {
  res.json({ message: 'Welcome to Three Tier App' })
})

app.get('/health', (req, res) => {
  res.json({ status: 'ok' })
})

app.get('/metrics', async (req, res) => {
  res.set('Content-Type', client.register.contentType)
  res.end(await client.register.metrics())
})

app.listen(port, () => {
  console.log(`App listening on port ${port}`)
})

collectDefaultMetrics() automatically collects Node.js runtime metrics, heap memory, CPU time, event loop lag, garbage collection. No manual instrumentation needed for the basics.

Hitting /metrics now returns something like:

# HELP nodejs_heap_size_used_bytes Process heap size used from Node.js in bytes.
# TYPE nodejs_heap_size_used_bytes gauge
nodejs_heap_size_used_bytes 4567890

Push this change through the GitHub Actions pipeline and the running ECS task will expose metrics at /metrics through the ALB.

Step 2: Run Prometheus and Grafana on EC2

I added a monitoring EC2 instance to my existing compute Terraform module. The user_data script installs Docker, installs Docker Compose, creates the configuration files, and starts both containers on boot.

Prometheus config:

global:
  scrape_interval: 15s

scrape_configs:
  - job_name: 'three-tier-app'
    static_configs:
      - targets: ['your-alb-dns-name']
    metrics_path: '/metrics'

Prometheus scrapes the ALB URL at /metrics every 15 seconds. The ALB forwards requests to the healthy ECS task.

Docker Compose:

services:
  prometheus:
    image: prom/prometheus
    ports:
      - "9090:9090"
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
      - prometheus-data:/prometheus

  grafana:
    image: grafana/grafana
    ports:
      - "3000:3000"
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=admin123
    volumes:
      - grafana-data:/var/lib/grafana

volumes:
  prometheus-data:
  grafana-data:

The named volumes mean Prometheus data and Grafana dashboards survive container restarts. If the containers restart the historical data is still there.

Security group for the monitoring instance:

Port 3000 → open to 0.0.0.0/0  (Grafana, public)
Port 9090 → not open            (Prometheus, internal only)
Port 22   → open to my IP only  (SSH management)

Prometheus is not exposed publicly. It contains raw infrastructure metrics you do not want the public reading. Grafana talks to Prometheus internally using the Docker Compose service name, not localhost.

The Docker Networking Lesson

When I first tried to connect Grafana to Prometheus I used http://localhost:9090 as the URL. It failed immediately.

The reason is that inside a Docker container localhost refers to that container itself, not the host machine and not other containers. Prometheus is a separate container. To reach it from Grafana you use the service name defined in Docker Compose:

http://prometheus:9090

Docker Compose creates an internal network where containers can reference each other by service name. This is fundamental Docker networking and easy to miss the first time.

Step 3: Connect Grafana to Prometheus

Once both containers are running:

Open Grafana at http://your-ec2-ip:3000
Go to Connections → Data Sources → Add data source → Prometheus
URL: http://prometheus:9090
Save and Test

The response should say the Prometheus API was queried successfully.

Then create a dashboard. The most useful metrics to start with:

nodejs_heap_size_used_bytes   → memory usage over time
process_cpu_seconds_total     → CPU consumption

Step 4: Load Testing

With the dashboard open I ran a load test using hey:

hey -n 10000 -c 50 http://your-alb-url/

This sends 10,000 requests with 50 concurrent connections. Watching the Grafana dashboard during the test showed memory climbing and CPU spiking in real time. The metrics endpoint was doing its job.

Step 5: Crash Testing

The most interesting part of observability is proving your system recovers from failure. I added a crash endpoint to the app:

app.get('/crash', (req, res) => {
  res.json({ message: 'Crashing...' })
  process.exit(1)
})

Deployed it through the pipeline, then triggered it:

curl http://your-alb-url/crash

What happened next:

Container process exited
        ↓
ECS detected unhealthy task
        ↓
Task deregistered from ALB target group
        ↓
ECS started a new task automatically
        ↓
New task passed health checks
        ↓
Task registered with ALB
        ↓
App responding again

Recovery took about 1 to 2 minutes. During that window the ALB returned 503s. Uptime Robot sent a down alert within 5 minutes, then a recovery alert once the new task was healthy.

The Grafana dashboard showed a gap in metrics during the crash window, then metrics resuming when the new task came up.

What This Proves

Nobody manually intervened. The container crashed, ECS detected it, a new task started, and traffic resumed. That is self-healing infrastructure working as designed.

The combination of Prometheus metrics, Grafana dashboards, and Uptime Robot gives three different views of the system:

Uptime Robot  → is it up or down right now
Grafana       → what is the system doing over time
CloudWatch    → what did the application log

Each answers a different question. Together they give you enough visibility to understand your system under normal load and under failure.

#aws#ecs#prometheus#grafana#devops

Making My ECS App Public with an Application Load Balancer

Lalit Bagga — Fri, 03 Jul 2026 14:46:57 +0000

My Node.js app was running on ECS Fargate and the CloudWatch logs confirmed it was healthy. But there was no way to actually reach it. No public URL. No way to test it from a browser. The container was alive but invisible.

The fix is an Application Load Balancer. Here is how I wired it up.

Why the App Was Unreachable

ECS Fargate tasks run in private subnets by design. No public IP, no direct internet access. That is the correct security posture for an application server. You never want your app directly exposed to the internet.

The ALB sits in the public subnet and acts as the entry point. Traffic flows like this:

Internet
        ↓
ALB (public subnet, internet facing)
        ↓
ECS Fargate task (private subnet, no public IP)

The app never needs a public IP. Only the ALB does.

What You Need to Wire Together

An ALB setup has three moving parts that work together:

Security Group controls who can connect to the ALB. Port 80 open to the world.

Target Group is where the ALB sends traffic after it is allowed in. It holds a list of healthy ECS tasks and runs health checks against them. If a task fails health checks it gets removed from rotation.

Listener watches a specific port on the ALB and decides what to do with incoming traffic. In this case listen on port 80 and forward to the target group.

A common mistake is thinking the security group is enough. It is not. The security group decides whether traffic is allowed. The listener decides what to do with it. The target group decides where it goes. All three are required.

One Thing ALB Requires

An ALB must span at least two subnets in different Availability Zones. This is an AWS requirement, not optional.

The reason is high availability. If one AZ goes down the ALB continues serving traffic through the other. You get one ALB, one DNS name, one entry point. AWS handles the redundancy behind the scenes.

This means your VPC needs public subnets in at least two different AZs. If you are defining subnets in Terraform always specify the availability zone explicitly:

resource "aws_subnet" "main_subnet_public_1" {
  vpc_id            = aws_vpc.main.id
  cidr_block        = "10.0.1.0/24"
  availability_zone = "us-east-2a"
}

resource "aws_subnet" "main_subnet_public_2" {
  vpc_id            = aws_vpc.main.id
  cidr_block        = "10.0.2.0/24"
  availability_zone = "us-east-2b"
}

If you leave out availability_zone AWS picks one automatically. You may end up with both subnets in the same AZ and the ALB will refuse to deploy.

Terraform for the ALB

I added an ALB module to my existing three tier Terraform project.

Security group for the ALB:

resource "aws_security_group" "alb_sg" {
  name   = "alb_sg"
  vpc_id = var.vpc_id

  tags = {
    Name = "alb_sg"
  }
}

resource "aws_vpc_security_group_ingress_rule" "alb_sg_ingress" {
  security_group_id = aws_security_group.alb_sg.id
  from_port         = 80
  to_port           = 80
  ip_protocol       = "tcp"
  cidr_ipv4         = "0.0.0.0/0"
}

resource "aws_vpc_security_group_egress_rule" "alb_sg_egress" {
  security_group_id = aws_security_group.alb_sg.id
  ip_protocol       = "-1"
  cidr_ipv4         = "0.0.0.0/0"
}

The ALB, target group, and listener:

resource "aws_lb" "main" {
  name               = "three-tier-alb"
  internal           = false
  load_balancer_type = "application"
  security_groups    = [var.alb_sg_id]
  subnets            = [var.public_subnet_1_id, var.public_subnet_2_id]

  tags = {
    Name = "three-tier-alb"
  }
}

resource "aws_lb_target_group" "main" {
  name        = "three-tier-tg"
  port        = 3000
  protocol    = "HTTP"
  vpc_id      = var.vpc_id
  target_type = "ip"

  health_check {
    path                = "/health"
    healthy_threshold   = 2
    unhealthy_threshold = 3
    interval            = 30
  }
}

resource "aws_lb_listener" "main" {
  load_balancer_arn = aws_lb.main.arn
  port              = 80
  protocol          = "HTTP"

  default_action {
    type             = "forward"
    target_group_arn = aws_lb_target_group.main.arn
  }
}

One thing worth noting on the target group: target_type = "ip" is required for Fargate. Fargate tasks do not have EC2 instance IDs. They have IP addresses. If you use instance as the target type the target group will never register any healthy targets and you will get 503s.

Locking Down the ECS Security Group

Before adding the ALB my ECS security group allowed inbound traffic from anywhere on port 3000. Once the ALB is in place that should change. Only the ALB should be able to talk to the ECS tasks. Not the public internet.

resource "aws_vpc_security_group_ingress_rule" "ecs_sg_ingress" {
  security_group_id            = aws_security_group.ecs_sg.id
  from_port                    = 3000
  to_port                      = 3000
  ip_protocol                  = "tcp"
  referenced_security_group_id = aws_security_group.alb_sg.id
}

Instead of cidr_ipv4 = "0.0.0.0/0" the ingress rule now references the ALB security group directly. Only traffic coming from the ALB is allowed to reach the ECS task on port 3000.

Wiring ECS to the Target Group

The ECS service needs to know about the target group so it can register tasks automatically. When a new task starts ECS registers its IP with the target group. When a task stops it deregisters.

resource "aws_ecs_service" "app" {
  name            = "three-tier-app-service"
  cluster         = aws_ecs_cluster.main.id
  task_definition = aws_ecs_task_definition.app.arn
  desired_count   = 1
  launch_type     = "FARGATE"

  network_configuration {
    subnets         = [var.private_subnet_id]
    security_groups = [var.ecs_sg_id]
  }

  load_balancer {
    target_group_arn = var.target_group_arn
    container_name   = "app-container"
    container_port   = 3000
  }

  depends_on = [var.alb_listener_arn]
}

The depends_on on the listener matters. ECS should not start registering tasks with the target group until the listener exists. Without this Terraform might try to create the ECS service before the listener is ready and health checks will fail in a loop.

The Result

After terraform apply the output shows the ALB DNS name:

alb_dns_name = "three-tier-alb-119067097.us-east-2.elb.amazonaws.com"

Hitting that URL in the browser returns the app response. Hitting /health returns a healthy status. The app is now publicly accessible through the load balancer with the ECS task completely hidden in the private subnet.

What Is Next

The app is live but there is no visibility into what it is actually doing. The next step is adding Prometheus and Grafana to collect real metrics and build dashboards, along with load testing and crash recovery to prove the infrastructure holds up under pressure.

aws#ecs#terraform#devops#loadbalancer

From EC2 to ECS: Containerizing My Three Tier App with GitHub Actions

Lalit Bagga — Mon, 22 Jun 2026 13:30:35 +0000

My three tier architecture had a problem. The app server was a plain EC2 instance sitting in a private subnet doing absolutely nothing. No application running on it. No way to deploy to it without SSHing in manually. That is not how modern infrastructure works.

So I containerized the app, pushed it to ECR, deployed it to ECS Fargate, and wired up a GitHub Actions pipeline that handles every deployment automatically. Here is how it went.

The App

I kept the application simple on purpose. The goal was never to build a complex application. The goal was to prove the full deployment pipeline works end to end. A Node.js Express API with two endpoints:

const express = require('express')
const app = express()
const port = 3000

app.get('/', (req, res) => {
  res.json({ message: 'Welcome to Three Tier App' })
})

app.get('/health', (req, res) => {
  res.json({ status: 'ok' })
})

app.listen(port, () => {
  console.log(`App listening on port ${port}`)
})

The health endpoint matters more than the root endpoint. ECS uses it to determine whether the container is healthy and ready to serve traffic. If health checks fail the task gets killed and restarted.

The Dockerfile

GitHub Copilot suggested a multi stage build and it was the right call:

FROM node:22-alpine AS deps

WORKDIR /app

COPY package*.json ./
RUN npm ci --omit=dev && npm cache clean --force

FROM node:22-alpine AS runtime

ENV NODE_ENV=production
WORKDIR /app

COPY --from=deps /app/node_modules ./node_modules
COPY index.js ./
COPY package.json ./

USER node

EXPOSE 3000

CMD ["node", "index.js"]

Two stages. The first stage installs dependencies. The second stage is the actual runtime image and only copies what it needs from the first stage. Build tools and npm cache never make it into the final image. Smaller image, faster pulls, better security.

USER node runs the process as a non root user. If the container gets compromised an attacker does not get root access to the underlying system.

The Architecture

Before writing any Terraform, it helps to understand what ECS actually needs:

Cluster — a logical boundary for your tasks. Think of it as the environment.

Task Definition — a blueprint. It tells ECS which image to run, how much CPU and memory to give it, which port to expose, and how to check if the container is healthy.

Service — keeps your task running. If the container crashes the service restarts it. You define how many copies should be running at any time.

Fargate — the launch type. Instead of managing EC2 instances as worker nodes, Fargate handles the underlying infrastructure. You only think about the container.

Execution Role — an IAM role that gives ECS permission to pull your image from ECR and write logs to CloudWatch.

Terraform for ECS

I added an ECS module to my existing three tier infrastructure:

resource "aws_ecs_cluster" "main" {
  name = "three-tier-app-cluster"
}

resource "aws_ecs_task_definition" "app" {
  family                   = "three-tier-app-task"
  network_mode             = "awsvpc"
  requires_compatibilities = ["FARGATE"]
  cpu                      = "256"
  memory                   = "512"
  execution_role_arn       = aws_iam_role.ecs_execution_role.arn

  container_definitions = jsonencode([
    {
      name      = "app-container"
      image     = var.ecr_repository_url
      essential = true
      portMappings = [
        {
          containerPort = 3000
          protocol      = "tcp"
        }
      ]
      healthCheck = {
        command     = ["CMD-SHELL", "wget -q -O /dev/null http://localhost:3000/health || exit 1"]
        interval    = 30
        timeout     = 5
        retries     = 3
        startPeriod = 10
      }
      logConfiguration = {
        logDriver = "awslogs"
        options = {
          "awslogs-group"         = "/ecs/three-tier-app"
          "awslogs-region"        = "us-east-2"
          "awslogs-stream-prefix" = "ecs"
        }
      }
    }
  ])
}

resource "aws_ecs_service" "app" {
  name            = "three-tier-app-service"
  cluster         = aws_ecs_cluster.main.id
  task_definition = aws_ecs_task_definition.app.arn
  desired_count   = 1
  launch_type     = "FARGATE"

  network_configuration {
    subnets         = [var.private_subnet_id]
    security_groups = [var.ecs_sg_id]
  }
}

The Errors

Two things broke and both were worth understanding.

Exit code 255 with no logs

The container kept exiting with code 255 and there was nothing in CloudWatch to explain why. The task definition had no logging configuration so ECS was swallowing all the output.

Added CloudWatch logging to the container definition and the actual error became visible immediately.

exec format error

exec /usr/local/bin/docker-entrypoint.sh: exec format error

I built the Docker image on a Mac with Apple Silicon which uses ARM architecture. ECS Fargate runs on AMD64. The image architecture did not match the platform and the container could not start.

The fix is building with an explicit platform flag:

docker buildx build --platform linux/amd64 -t three-tier-app .

This is easy to miss when developing locally on a Mac. The container runs fine locally and only fails when deployed to ECS.

The Assumption That Was Wrong

I assumed ECS would automatically pick up a new image when I pushed to ECR. Push a new image with the latest tag, ECS detects it, redeploys. That is not how it works.

ECS pins to the exact image digest that was running when the task started. Even if you push a new latest tag to ECR, the running task keeps using the old image until you explicitly trigger a new deployment.

This is actually intentional. You do not want your production containers randomly restarting because someone pushed a new image. You want deployments to be deliberate and controlled.

The way to trigger a new deployment is:

aws ecs update-service --cluster three-tier-app-cluster \
  --service three-tier-app-service \
  --force-new-deployment

Which is exactly what the GitHub Actions pipeline automates.

The GitHub Actions Pipeline

The pipeline lives in .github/workflows/deploy.yml and triggers on every push to main:

name: Deploy to ECS

on:
  push:
    branches:
      - main

env:
  AWS_REGION: us-east-2
  ECR_REPOSITORY: three-tier-app
  ECS_CLUSTER: three-tier-app-cluster
  ECS_SERVICE: three-tier-app-service

jobs:
  deploy:
    name: Build and Deploy
    runs-on: ubuntu-latest

    steps:
      - name: Checkout code
        uses: actions/checkout@v4

      - name: Configure AWS credentials
        uses: aws-actions/configure-aws-credentials@v4
        with:
          aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
          aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
          aws-region: ${{ env.AWS_REGION }}

      - name: Login to Amazon ECR
        id: login-ecr
        uses: aws-actions/amazon-ecr-login@v2

      - name: Build, tag, and push image to ECR
        env:
          ECR_REGISTRY: ${{ steps.login-ecr.outputs.registry }}
          IMAGE_TAG: ${{ github.sha }}
        run: |
          docker buildx build --platform linux/amd64 \
            -t $ECR_REGISTRY/$ECR_REPOSITORY:$IMAGE_TAG \
            -t $ECR_REGISTRY/$ECR_REPOSITORY:latest \
            --push .

      - name: Deploy to ECS
        run: |
          aws ecs update-service \
            --cluster ${{ env.ECS_CLUSTER }} \
            --service ${{ env.ECS_SERVICE }} \
            --force-new-deployment \
            --region ${{ env.AWS_REGION }}

Each image gets tagged twice. Once with the git commit SHA for traceability and once with latest for convenience. If something goes wrong you can identify exactly which commit is running in production.

AWS credentials are stored as GitHub secrets and never appear in the workflow file.

Verifying It Works

The app is running in a private subnet with no public IP. Without an Application Load Balancer there is no public URL to hit. But two things confirm everything is working.

CloudWatch logs show the app started successfully:

App listening on port 3000

And the ECS console shows the task as running and healthy with a recent timestamp after every pipeline run.

The pipeline going green means the image built, pushed to ECR, and ECS started a new deployment. CloudWatch logs confirm the new container came up healthy.

What Is Next

The app is running but invisible. The next step is adding an Application Load Balancer to expose it publicly. The ALB sits in the public subnet, accepts traffic from the internet, and forwards it to the ECS task in the private subnet.

Once that is in place the full URL will be accessible and the three tier story is complete.

#aws #ecs #docker #devops #githubactions

Refactoring Terraform: From One File to Modules

Lalit Bagga — Mon, 08 Jun 2026 13:31:33 +0000

My three-tier AWS architecture worked. VPC, subnets, bastion host, app server, RDS, all deployed and running. But my main.tf was a flat file with everything mixed together. Security groups next to route tables next to RDS instances next to IAM roles.

It worked for a learning project. It would not work in a real team environment where multiple people need to understand, maintain, and extend the infrastructure.

So I refactored it into modules. Here is what I learned.

What Is a Module

A module is just a folder with its own Terraform files. Nothing magic about it. You move related resources into that folder, define what it needs as inputs, define what it exposes as outputs, and then call it from your root configuration.

The root main.tf becomes an orchestrator, it calls each module and wires them together by passing outputs from one into inputs of another.

The Final Structure

Before refactoring everything lived in one file. After:

three-tier/
├── main.tf               ← calls all modules, wires them together
├── variables.tf
├── outputs.tf
└── module/
    ├── networking/
    │   ├── main.tf
    │   ├── variable.tf
    │   └── outputs.tf
    ├── security/
    │   ├── main.tf
    │   ├── variable.tf
    │   └── outputs.tf
    ├── compute/
    │   ├── main.tf
    │   ├── variable.tf
    │   └── output.tf
    └── database/
        ├── main.tf
        ├── variable.tf
        └── output.tf

Each module owns one concern:

networking  → VPC, subnets, IGW, NAT gateway, route tables
security    → security groups and all ingress/egress rules
compute     → IAM roles, instance profile, SSM, key pair, EC2 instances
database    → RDS instance, DB subnet group

The Core Pattern: Outputs and Variables

This is the most important thing to understand before you start. Modules cannot reach outside themselves. If the compute module needs the VPC ID, it cannot just reference aws_vpc.main.id that resource lives in the networking module now.

The pattern is always three steps:

Step 1 Output it from the source module:

# module/networking/outputs.tf
output "vpc_id" {
  value = aws_vpc.main.id
}

Step 2 Declare it as a variable in the receiving module:

# module/security/variable.tf
variable "vpc_id" {
  description = "VPC ID from networking module"
  type        = string
}

Step 3 Pass it through the root main.tf:

# main.tf
module "security" {
  source = "./module/security"
  vpc_id = module.networking.vpc_id
}

Every cross-module reference follows this exact pattern. Once you internalize it the errors stop being confusing.

The Dependency Order

Modules depend on each other in a specific order. Networking has no dependencies so it goes first. Security needs the VPC ID from networking. Compute and database both need outputs from networking and security.

networking  → no dependencies
    ↓
security    → needs vpc_id from networking
    ↓
compute     → needs subnet IDs from networking
            → needs bastion_sg_id, private_sg_id from security
database    → needs db subnet IDs from networking
            → needs db_sg_id from security

Terraform figures out the order automatically based on these references. You do not need to use depends_on explicitly as soon as you reference module.networking.vpc_id, Terraform knows networking must complete before security starts.

How I Approached the Refactor

I did it one module at a time, starting with networking. The process for each module was:

Create the module folder and files
Move the relevant resources into module/networking/main.tf
Add a module "networking" call in root main.tf
Run terraform plan
Fix the errors — usually missing outputs or undeclared variables
Repeat for next module

The errors I kept hitting all looked like this:

Error: Reference to undeclared resource
  on main.tf line 38, in resource "aws_security_group" "bastion_sg":
  vpc_id = aws_vpc.main.id

A managed resource "aws_vpc" "main" has not been declared in the root module.

This means a resource is trying to reference something that has moved into a module. The fix is always the same , output it from the module, declare a variable in the receiving module, pass it through root.

The State Migration Problem

Here is something nobody warns you about when refactoring Terraform into modules.

When you move a resource from root into a module, its address in the state file changes. What was aws_vpc.main becomes module.networking.aws_vpc.main. Terraform sees this as a different resource, it thinks the old one was deleted and a new one needs to be created.

Running terraform plan after the refactor showed this:

Plan: 27 to add, 0 to change, 27 to destroy.

That is not what you want. It would destroy and recreate all your infrastructure.

The proper fix for a production environment is terraform state mv , a command that tells Terraform a resource just moved, it was not deleted. You run one command per resource:

terraform state mv aws_vpc.main module.networking.aws_vpc.main
terraform state mv aws_subnet.main_subnet_public_1 module.networking.aws_subnet.main_subnet_public_1
# ... one for every resource

For a learning project with no real traffic or data at risk, the simpler path is:

terraform destroy
terraform apply

Destroy everything, apply fresh from the new module structure. Same end result, no manual state migration required.

The apply completed cleanly:

Apply complete! Resources: 35 added, 0 changed, 0 destroyed.

What the Root main.tf Looks Like Now

The root main.tf went from a flat list of 43+ resources to a clean orchestration file:

module "networking" {
  source     = "./module/networking"
  aws_region = var.aws_region
}

module "security" {
  source = "./module/security"
  vpc_id = module.networking.vpc_id
}

module "compute" {
  source                = "./module/compute"
  public_subnet_id      = module.networking.public_subnet_id
  private_subnet_id     = module.networking.private_subnet_id
  bastion_sg_id         = module.security.bastion_sg_id
  private_sg_id         = module.security.private_sg_id
}

module "database" {
  source         = "./module/database"
  db_subnet_1_id = module.networking.db_subnet_1_id
  db_subnet_2_id = module.networking.db_subnet_2_id
  db_sg_id       = module.security.db_sg_id
}

You can read this and immediately understand the infrastructure. Four modules, clear dependencies, no hunting through hundreds of lines to find what you need.

What I Learned

Modules are just folders. There is no magic. The mental shift is understanding that resources can no longer reference each other directly once they live in different modules. Everything goes through outputs and variables.

Start with networking. It has no dependencies so there are no wiring errors to debug. Get networking working first, then add security, then compute and database.

The state migration problem is real. In production you would never destroy and recreate. You would use terraform state mv or moved blocks to migrate state without downtime. For a learning project, destroy and recreate is fine, but knowing why the problem exists is important.

The root main.tf should be an orchestrator, not a resource file. If you have resource blocks in your root main.tf alongside module calls, that is a signal something belongs in a module.

What Is Next

The next step is enabling RDS IAM Authentication, replacing the hardcoded database password with token-based access. Storing credentials directly in Terraform is a bad practice and there is a cleaner way to handle it.

#aws #terraform #devops #infrastructureascode #modules