Learn how to build a complete containerized deployment pipeline on AWS— VPC, ECR, ECS Fargate, an Application Load Balancer with HTTPS, CodePipeline/CodeBuild CI/CD, and CloudWatch alarting — using Terraform, across two isolated environments, and how to actually recover when things go wrong along the way.
Author: Adedeji Adesanya
Difficulty: Intermediate
Estimated Time: 6–8 Hours (across two sessions)
AWS Services Used:
Amazon VPC
Amazon ECR
AWS IAM
Application Load Balancer (ALB) + AWS Certificate Manager (ACM)
Amazon ECS (Fargate)
AWS CodePipeline + AWS CodeBuild
Amazon CloudWatch + Amazon SNS
Amazon S3 + DynamoDB (Terraform remote state)
Table of Contents
Introduction
What You Will Build
Solution Architecture
Prerequisites
Building the Foundation: VPC, ECR, IAM
Fronting the App: ALB and ECS Fargate
Getting the App Actually Running (and What Went Wrong)
Wiring Up CI/CD with CodePipeline
Adding HTTPS to the Load Balancer
The Terraform vs. Pipeline Conflict (and How to Fix It)
Monitoring and Alerting
Governance: Branch Protection and Remote State
Lessons Learned
Production Considerations
Conclusion
Introduction
Most Terraform tutorials show you a clean, linear path: write some HCL, run terraform apply, watch resources appear, done. That's useful for learning syntax, but it's not what real infrastructure work actually feels like.
This project started as a straightforward goal — deploy a containerized web app to AWS using Terraform, with a proper CI/CD pipeline behind it. What it turned into was a two-day exercise in debugging the kind of problems that don't show up in tutorials: stale module variables that didn't match between files, an empty .tf file that looked complete in an editor but was actually zero bytes on disk, a Dockerfile silently missing a COPY instruction, a PowerShell pipe corrupting an authentication token, and — the most interesting one — Terraform and a CI/CD pipeline fighting over who controls the same resource.
None of these are exotic problems. They're the ordinary friction of building something real. This article walks through the architecture we ended up with, and — more usefully — the specific failures we hit and how each one was actually diagnosed and fixed, not just patched over.
What You Will Build
By the end of this project, you'll have two fully isolated environments (dev and prod), each with:
A custom VPC with public/private subnets across two Availability Zones, a NAT Gateway, and proper routing
A private ECR repository with image scanning and a lifecycle policy
IAM roles scoped to least privilege for both ECS task execution and the running application
An Application Load Balancer with an HTTPS listener (redirecting from HTTP) in front of an ECS Fargate service
A CodePipeline + CodeBuild pipeline that builds, tags, and deploys a new container image automatically on every push to the branch
CloudWatch alarms for 5xx errors, unhealthy targets, low running task count, and high CPU — all notifying an SNS topic by email
Terraform state stored remotely in S3 with DynamoDB locking, instead of local state files
Branch protection on the production branch, requiring a pull request before anything reaches prod
Solution Architecture
At a high level:
Each environment (dev, prod) is a completely separate Terraform root module with its own state file, its own VPC, its own pipeline, and its own alarms, sharing only the underlying Terraform modules (the reusable vpc, ecr, iam, alb, ecs, codepipeline, sns, and cloudwatch modules live once, and both environments call them with different variables).
Prerequisites
RequirementStatusAWS Account with admin-equivalent IAM permissions✅Terraform installed (we used 1.12.x)✅Docker Desktop✅A GitHub repository✅Basic familiarity with VPC, IAM, and containers✅
Building the Foundation: VPC, ECR, IAM
We started with the standard networking layer: a VPC, public and private subnets across two AZs, an Internet Gateway, a NAT Gateway for the private subnets, and route tables. Nothing unusual here — except that this is exactly where our first real problem showed up (more on that in Lessons Learned).
ECR came next: a single private repository with IMMUTABLE tag mutability (once you push a tag, it can never be overwritten — this matters a lot once CI/CD enters the picture, since it forces every build to have a genuinely unique tag), image scanning on push, and a lifecycle policy expiring untagged images after 14 days and capping tagged images at 10.
IAM followed the standard ECS pattern of two roles:
A task execution role, used by ECS itself to pull the image from ECR and write logs to CloudWatch (this one just attaches the AWS-managed AmazonECSTaskExecutionRolePolicy)
A task role, used by the application code at runtime, scoped narrowly to only the DynamoDB, SQS, SNS, S3, and CloudWatch Logs resources matching our naming prefix — not *
Fronting the App: ALB and ECS Fargate
The ALB module creates a security group (initially just port 80), the load balancer itself across the public subnets, a target group pointed at port 3000 (our app's port), and an HTTP listener. The ECS module creates a Fargate cluster with Container Insights enabled, a CloudWatch log group, a security group that only accepts traffic from the ALB's security group (not the open internet), a task definition, and the service itself.
Getting the App Actually Running (and What Went Wrong)
We built a simple Express app — an intentionally over-designed storefront landing page for "GlobalMart," styled around an airport-departures-board concept (navy and saffron palette, a flip-board ticker showing product "arrivals" from different countries) rather than a generic template. Dockerized it, pushed it to ECR, pointed the ECS task definition at the image, and applied.
The service came up with zero healthy tasks. ALB health checks were failing with a 404.
The cause: the Dockerfile copied server.js into the image but never copied the public/ folder where index.html actually lived. The container was running, Express was serving requests, and every single one of them correctly 404'd — because there was nothing to serve. Adding one line (COPY public ./public), rebuilding, and pushing a new tag fixed it immediately. This is the kind of bug that's invisible in terraform plan and invisible in docker build output — it only shows up once real traffic hits the container, which is exactly why health checks and monitoring matter.
Wiring Up CI/CD with CodePipeline
Manually running docker build && docker tag && docker push and then hand-editing a Terraform variable every time you want to deploy is fine for a demo, but it's not how real teams ship code. We built a codepipeline Terraform module with:
An S3 bucket for pipeline artifacts (versioned, encrypted, public access blocked)
A CodeStar Connection to GitHub (this is the one piece Terraform can't fully automate — AWS requires a one-time manual OAuth authorization click in the console after the connection resource is created)
A CodeBuild project running in privileged mode (required for Docker-in-Docker), with a buildspec.yml that logs into ECR, builds the image, tags it with the first 8 characters of the git commit SHA (guaranteeing a unique tag every time, which plays correctly with our IMMUTABLE repository), pushes it, and writes an imagedefinitions.json file
A three-stage CodePipeline: Source (GitHub via the connection) → Build (CodeBuild) → Deploy (native ECS deploy action, which reads imagedefinitions.json and updates the service)
Once the GitHub connection was authorized in the console, a push to the branch triggered the full pipeline automatically, and it succeeded end-to-end.
Adding HTTPS to the Load Balancer
Since we didn't own a domain (ACM certificates require domain validation via DNS or email, which isn't possible against a bare AWS-generated ALB hostname), we generated a self-signed certificate with OpenSSL and imported it directly into ACM — skipping domain validation entirely. This isn't a certificate a browser will trust, but it's genuinely useful for demonstrating correct TLS termination architecture: an HTTP listener on port 80 that redirects (301) to HTTPS on port 443, and an HTTPS listener that terminates TLS at the load balancer and forwards plain HTTP to the target group.
Getting there involved one Windows-specific detour worth mentioning: aws ecr get-login-password | docker login ... piped through native PowerShell silently corrupted the long base64 authentication token and failed with a 400 Bad Request. Routing the exact same command through cmd /c "..." instead fixed it immediately — a reminder that shell pipeline behavior isn't universal, even for commands copied directly from AWS's own documentation.
The Terraform vs. Pipeline Conflict (and How to Fix It)
This was the most instructive bug in the whole project.
Once CodePipeline started deploying new task definition revisions on every push (bypassing Terraform entirely — it calls the ECS API directly), Terraform's local state fell out of sync with reality. The next time we ran terraform plan for an unrelated change, Terraform saw that the "current" task definition according to its state was several revisions behind what was actually running, and proposed to forcibly roll the service back — silently undoing every deploy the pipeline had made since the last terraform apply.
This is a structural conflict, not a bug in either tool: Terraform assumes it's the sole owner of everything in its state, and a CI/CD pipeline managing the same resource violates that assumption. The fix is to explicitly tell Terraform to stop caring about that one attribute:
hclresource "aws_ecs_service" "app" {
# ...
lifecycle {
ignore_changes = [task_definition]
}
}
With this in place, Terraform continues managing the infrastructure around the service (cluster, networking, security groups) but defers entirely to the pipeline for which image is currently deployed. This is a pattern worth knowing before you combine Terraform-managed ECS with any CI/CD tool that also touches the task definition — without it, your next unrelated terraform apply can quietly roll back production.
Monitoring and Alerting
Four CloudWatch alarms per environment, all notifying a single SNS topic subscribed by email:
ALB 5xx errors — more than 10 target-side 5xx responses in a 5-minute window
Unhealthy target count — any target behind the ALB reporting unhealthy
Low running task count — fewer running ECS tasks than desired (using the ECS/ContainerInsights namespace, which requires Container Insights to be enabled on the cluster)
High CPU — ECS service CPU utilization above 80% sustained for 15 minutes
The SNS email subscription needs a manual confirmation click after terraform apply — Terraform can create the subscription, but AWS requires the recipient to actively confirm it, the same way it requires manual authorization for the GitHub connection.
Governance: Branch Protection and Remote State
Two changes made this feel less like a personal sandbox and more like a real team setup:
Branch protection on prod — configured directly in GitHub (Settings → Branches), requiring a pull request before anything merges into the branch that triggers the production pipeline. No more direct pushes to prod, even by an admin, without a deliberate PR.
Migrating Terraform state from local files to S3 with DynamoDB locking — local terraform.tfstate files are fine for solo experimentation but are a real liability the moment more than one person (or more than one machine) might run terraform apply against the same infrastructure: concurrent applies can corrupt state, and a deleted or lost local file means losing track of everything Terraform manages. Remote state in S3, combined with a DynamoDB table for locking (so two applies can't run simultaneously), is the standard production pattern.
Lessons Learned
Empty files look identical to complete files in an editor. At one point, several .tf files inside a module directory were showing valid, complete content in the code editor — but were actually zero bytes on disk. The content only existed in an unsaved editor buffer. terraform validate correctly reported the module's variables as undeclared, which looked like a logic error but was actually a save error. Always verify file sizes on disk (Get-ChildItem, or ls -la) when a .tf file's content doesn't match what Terraform is complaining about.
Duplicate resource declarations fail loudly, which is a feature. Copy-pasting a resource block into the wrong file (in our case, aws_subnet blocks ended up duplicated across two files) produces an immediate, unambiguous "Duplicate resource configuration" error — this is one of the few Terraform failure modes that's actually easy to diagnose, because the error names the exact resource and the exact second location.
PowerShell's pipeline isn't always transparent. Long strings piped between commands (the ECR auth token, in our case) can be silently mangled by PowerShell in ways that produce a downstream error with no obvious connection to the actual cause. When a command that should obviously work fails with a generic error, try routing it through cmd /c before assuming the problem is on the remote end.
A working terraform plan and a working application are two different claims. Every piece of infrastructure can apply cleanly and still serve a 404 for every request, because Terraform has no visibility into what's actually inside your container image. Health checks, and genuinely hitting the live endpoint after every deploy, are not optional steps.
Recycle Bin is a legitimate disaster recovery tool. An accidental folder deletion mid-session turned out to be completely recoverable because Windows' standard delete (rather than a hard Remove-Item -Force) sends files to the Recycle Bin first. Combined with git for the source code and the fact that AWS resources don't disappear just because local files do, this turned a moment of genuine panic into a five-minute non-event. It's still worth treating your Terraform state file with real care — it was the one thing that mattered here and wasn't backed up anywhere except that Recycle Bin.
Branches don't protect you from typing on the wrong one. A git checkout prod earlier in a session, followed by continuing to work without checking git branch again, meant several commits meant for main landed on prod instead. Git doesn't warn you about this, git commit and git push work identically regardless of which branch is checked out. git status at the top of every output tells you the current branch; it's worth reading that line every time, not just the "nothing to commit" part.
Production Considerations
This project is intentionally a solid foundation, not a finished production system. Before treating something like this as production-ready, we'd still want:
A real domain name and a properly validated ACM certificate (our self-signed cert demonstrates the architecture but isn't trusted by any browser)
Auto-scaling policies on the ECS service, instead of a hardcoded desired count
The DynamoDB, SQS, SNS, and S3 resources the IAM task role is already scoped for, but which don't exist yet
A WAF in front of the ALB for basic protection against common web exploits
Centralizing logs and metrics further with CloudWatch Dashboards, rather than checking alarms individually
Conclusion
The infrastructure in this project (a VPC, a container registry, a load balancer, a Fargate service, a CI/CD pipeline, and monitoring) isn't architecturally unusual. What made it worth writing up wasn't the happy path; it was everything that went wrong along the way, and the fact that every failure had a specific, diagnosable cause rather than being a mystery to work around.
That's the actual shape of infrastructure work: not a sequence of commands that always succeed, but a sequence of hypotheses, verifications, and fixes. The terraform plan that shows "No changes" after a scare is more valuable than the one that shows a clean create on the first try, because it proves the system actually recovered correctly rather than just looking fine.
If you're building something similar, the advice that mattered most in this project wasn't about any specific AWS service, it was to verify state directly rather than assume it (Get-ChildItem before trusting a file exists, terraform plan before trusting an apply succeeded, curl the actual endpoint before trusting a health check passed), and to fix the root cause of a failure rather than the symptom in front of you.
Top comments (0)