Ever wondered what actually happens when a server crashes in production? Not in theory, but for real. Does your app just die? Does anyone notice? Does anything fix itself?
I decided to stop wondering and build the answer myself. This article walks through a project where I built a small, genuinely self-healing web application on AWS: containers that survive being deliberately broken, a load balancer that routes around failure automatically, a CI/CD pipeline that ships code safely, and alerts that tell me when things go wrong.
By the end of this article, you'll have replicated the whole thing yourself, with your own hands on the keyboard, not just read about it.
Let's get into it.
What We're Actually Building
Here's the bird's eye view before we touch a single line of code:
- A small Node.js app with a real health check endpoint (not just "am I alive", but "am I actually okay")
- A VPC spread across two AWS Availability Zones, with public and private subnets
- The app running as containers on AWS Fargate (serverless containers, no EC2 servers to babysit)
- An Application Load Balancer that only sends traffic to healthy containers
- A CI/CD pipeline with GitHub Actions that builds and deploys automatically on every push
- CloudWatch dashboards and alarms, wired to email alerts via SNS
- A chaos endpoint to deliberately break things on purpose, so we can watch the system heal itself
All of the infrastructure is written in Terraform, so it's repeatable, version-controlled, and destroyable with a single command when you're done experimenting.
You can find the full, finished code in the repo here: View the GitHub Repository
Prerequisites
Before we start, make sure you've got:
- An AWS account (with a non-root IAM user set up, please don't use root credentials day to day)
- Terraform installed locally
- Docker Desktop installed and running
- The AWS CLI installed and configured (aws configure)
- A GitHub account and a new empty repo to push your code to
- Basic comfort with the command line (we'll be doing a lot of terraform apply)
Quick note throughout this article: anywhere you see the account ID 123456789012, swap it for your own AWS account ID. That number is a placeholder, not a real account.
Step 1: The Skeleton
Start with a simple folder structure to keep the application code and the infrastructure code cleanly separated:
resilient-aws-infra/
├── app/ # The Node.js application + Dockerfile
├── infra/ # All the Terraform code
└── .github/
└── workflows/ # CI/CD pipeline definition
_Keeping app/ and infra/ separate is a small thing, but it pays off. It means your CI/CD pipeline can be told "only rebuild when the app changes", which we'll use later.
_
Step 2: The App
Nothing fancy here. Just a tiny Express app with three routes:
- / — a homepage that says hello
- /health — a real health check, one that actually checks something (memory usage), not just "the process is running"
- /chaos/toggle-unhealthy — a hidden switch we'll use later to deliberately break the app on purpose
const express = require('express');
const app = express();
const PORT = process.env.PORT || 3000;
let forceUnhealthy = false;
app.get('/', (req, res) => {
res.send('<h1>Resilient AWS Infra - it works</h1>');
});
// A REAL health check - not just "am I running", but "am I actually okay"
app.get('/health', (req, res) => {
const memoryUsedMB = process.memoryUsage().heapUsed / 1024 / 1024;
const isHealthy = !forceUnhealthy && memoryUsedMB < 200;
if (isHealthy) {
res.status(200).json({ status: 'healthy', memoryMB: memoryUsedMB.toFixed(1) });
} else {
res.status(503).json({ status: 'unhealthy', memoryMB: memoryUsedMB.toFixed(1), forced: forceUnhealthy });
}
});
// A deliberate "break glass" switch, locked behind a secret header
const CHAOS_SECRET = process.env.CHAOS_SECRET || 'change-me';
app.get('/chaos/toggle-unhealthy', (req, res) => {
const providedSecret = req.header('X-Chaos-Secret');
if (providedSecret !== CHAOS_SECRET) {
return res.status(403).json({ error: 'Forbidden' });
}
forceUnhealthy = !forceUnhealthy;
res.json({ forceUnhealthy });
});
app.listen(PORT, () => {
console.log(`Server running on port ${PORT}`);
});
And the matching package.json:
{
"name": "resilient-aws-infra-app",
"version": "1.0.0",
"description": "Small demo app for practicing incident response on AWS",
"main": "server.js",
"scripts": {
"start": "node server.js"
},
"dependencies": {
"express": "^4.19.2"
}
}
That /chaos endpoint might look reckless, but notice it's locked behind a secret header. We'll wire that secret in properly through Terraform later, so it's never sitting around in plain text.
Step 3: Dockerize It
Next, we package the app into a container. Create a Dockerfile inside app/:
FROM node:20-alpine
WORKDIR /app
COPY package.json ./
RUN npm install --production
COPY server.js ./
EXPOSE 3000
CMD ["node", "server.js"]
Quick breakdown:
- FROM node:20-alpine starts from a small, lightweight image that already has Node.js
- WORKDIR /app sets our working folder inside the container
- We copy package.json first and install dependencies before copying the actual code. This is a small trick that speeds up rebuilds, since Docker can reuse the cached dependency layer if only your code changed
- EXPOSE 3000 documents which port the app listens on
- CMD is what runs when the container starts
Heads up on Windows: if you're using Notepad to create this file, it'll often save it as Dockerfile.txt instead of Dockerfile. Docker specifically looks for a file named exactly Dockerfile, no extension. Rename it if needed:
Rename-Item Dockerfile.txt Dockerfile
Test the build locally before moving on:
docker build -t resilient-aws-infra-app .
Step 4: The Networking Foundation (VPC)
This is where we start writing Terraform. Everything from here on lives in the infra/ folder.
First, tell Terraform which cloud provider and region we're using:
# providers.tf
terraform {
required_version = ">= 1.5.0"
required_providers {
aws = {
source = "hashicorp/aws"
version = "~> 5.0"
}
}
}
provider "aws" {
region = "eu-west-1"
}
Then define some reusable variables so we're not hardcoding values everywhere:
# variables.tf
variable "project_name" {
type = string
default = "resilient-aws-infra"
}
variable "aws_region" {
type = string
default = "eu-west-1"
}
variable "vpc_cidr" {
type = string
default = "10.0.0.0/16"
}
variable "public_subnet_cidrs" {
type = list(string)
default = ["10.0.1.0/24", "10.0.2.0/24"]
}
variable "private_subnet_cidrs" {
type = list(string)
default = ["10.0.11.0/24", "10.0.12.0/24"]
}
variable "availability_zones" {
type = list(string)
default = ["eu-west-1a", "eu-west-1b"]
}
Now the actual network. This creates a VPC (your own private slice of AWS), spread across two Availability Zones for resilience, with public subnets (reachable from the internet) and private subnets (where our containers will actually live):
# main.tf
resource "aws_vpc" "main" {
cidr_block = var.vpc_cidr
enable_dns_support = true
enable_dns_hostnames = true
tags = {
Name = "${var.project_name}-vpc"
}
}
resource "aws_internet_gateway" "main" {
vpc_id = aws_vpc.main.id
tags = {
Name = "${var.project_name}-igw"
}
}
resource "aws_subnet" "public" {
count = length(var.public_subnet_cidrs)
vpc_id = aws_vpc.main.id
cidr_block = var.public_subnet_cidrs[count.index]
availability_zone = var.availability_zones[count.index]
map_public_ip_on_launch = true
tags = {
Name = "${var.project_name}-public-${var.availability_zones[count.index]}"
}
}
resource "aws_subnet" "private" {
count = length(var.private_subnet_cidrs)
vpc_id = aws_vpc.main.id
cidr_block = var.private_subnet_cidrs[count.index]
availability_zone = var.availability_zones[count.index]
tags = {
Name = "${var.project_name}-private-${var.availability_zones[count.index]}"
}
}
resource "aws_eip" "nat" {
domain = "vpc"
}
resource "aws_nat_gateway" "main" {
allocation_id = aws_eip.nat.id
subnet_id = aws_subnet.public[0].id
depends_on = [aws_internet_gateway.main]
}
resource "aws_route_table" "public" {
vpc_id = aws_vpc.main.id
route {
cidr_block = "0.0.0.0/0"
gateway_id = aws_internet_gateway.main.id
}
}
resource "aws_route_table" "private" {
vpc_id = aws_vpc.main.id
route {
cidr_block = "0.0.0.0/0"
nat_gateway_id = aws_nat_gateway.main.id
}
}
resource "aws_route_table_association" "public" {
count = length(aws_subnet.public)
subnet_id = aws_subnet.public[count.index].id
route_table_id = aws_route_table.public.id
}
resource "aws_route_table_association" "private" {
count = length(aws_subnet.private)
subnet_id = aws_subnet.private[count.index].id
route_table_id = aws_route_table.private.id
}
A quick mental model for the pieces above:
- Public subnets are the front door. Things here can be reached from the internet, like our load balancer.
- Private subnets are the back room. Our actual containers live here, never directly reachable from the outside.
- The Internet Gateway lets public subnets talk to the internet.
- The NAT Gateway lets private subnets reach out (to pull Docker images, for example) without letting the internet reach in.
Run terraform init, then terraform plan to preview it, and terraform apply to build it for real.
terraform init
terraform plan
terraform apply
Step 5: A Home for Your Docker Image (ECR)
Before we can run our container in AWS, it needs somewhere to live. That's ECR (Elastic Container Registry), basically a private Docker Hub inside your own AWS account.
# ecr.tf
resource "aws_ecr_repository" "app" {
name = "${var.project_name}-app"
image_tag_mutability = "MUTABLE"
image_scanning_configuration {
scan_on_push = true
}
}
output "ecr_repository_url" {
value = aws_ecr_repository.app.repository_url
}
scan_on_push = true is a nice little freebie: AWS automatically scans every image you push for known vulnerabilities, at no extra cost.
Apply it, then build and push your image:
terraform apply
# Authenticate Docker to your new ECR repo
aws ecr get-login-password --region eu-west-1 | docker login --username AWS --password-stdin 123456789012.dkr.ecr.eu-west-1.amazonaws.com
# Build, tag, and push
docker build -t resilient-aws-infra-app .
docker tag resilient-aws-infra-app:latest 123456789012.dkr.ecr.eu-west-1.amazonaws.com/resilient-aws-infra-app:latest
docker push 123456789012.dkr.ecr.eu-west-1.amazonaws.com/resilient-aws-infra-app:latest
Step 6: Security Groups (The Firewall Rules)
Before we build the load balancer and the containers, let's set up the security groups that control exactly what can talk to what. Think of these as a strict guest list.
# security-groups.tf
# The load balancer can be reached by anyone, on port 80
resource "aws_security_group" "alb" {
name = "${var.project_name}-alb-sg"
description = "Allow inbound HTTP from the internet"
vpc_id = aws_vpc.main.id
ingress {
from_port = 80
to_port = 80
protocol = "tcp"
cidr_blocks = ["0.0.0.0/0"]
}
egress {
from_port = 0
to_port = 0
protocol = "-1"
cidr_blocks = ["0.0.0.0/0"]
}
}
# The containers can ONLY be reached by the load balancer, on port 3000
resource "aws_security_group" "ecs_service" {
name = "${var.project_name}-ecs-sg"
description = "Allow inbound traffic only from the ALB"
vpc_id = aws_vpc.main.id
ingress {
from_port = 3000
to_port = 3000
protocol = "tcp"
security_groups = [aws_security_group.alb.id]
}
egress {
from_port = 0
to_port = 0
protocol = "-1"
cidr_blocks = ["0.0.0.0/0"]
}
}
Notice the container security group doesn't open port 3000 to the whole internet, it only trusts traffic coming from the load balancer's own security group. Your containers are never directly exposed.
Step 7: The Load Balancer
This is the piece that actually makes resilience possible. The Application Load Balancer (ALB) sits in front of your containers, continuously checking their health, and only sends traffic to the ones that pass.
# alb.tf
resource "aws_lb" "main" {
name = "${var.project_name}-alb"
internal = false
load_balancer_type = "application"
security_groups = [aws_security_group.alb.id]
subnets = aws_subnet.public[*].id
}
resource "aws_lb_target_group" "app" {
name = "${var.project_name}-tg"
port = 3000
protocol = "HTTP"
vpc_id = aws_vpc.main.id
target_type = "ip"
health_check {
path = "/health"
port = "traffic-port"
protocol = "HTTP"
healthy_threshold = 2
unhealthy_threshold = 3
timeout = 5
interval = 15
matcher = "200"
}
}
resource "aws_lb_listener" "http" {
load_balancer_arn = aws_lb.main.arn
port = 80
protocol = "HTTP"
default_action {
type = "forward"
target_group_arn = aws_lb_target_group.app.arn
}
}
output "alb_dns_name" {
value = aws_lb.main.dns_name
}
That health_check block is doing the real work here. Every 15 seconds, it hits /health. If it doesn't get a 200 response within 5 seconds, and this happens 3 times in a row, the ALB marks that specific container "unhealthy" and quietly stops sending it traffic. No drama, no downtime for users, just a clean handoff to the containers that are actually working.
Step 8: ECS (Where the Container Actually Runs)
Now for the piece that runs your container. We're using AWS Fargate, which is "serverless containers": you don't manage any EC2 instances, AWS just runs your container for you.
First, the cluster, the IAM role ECS needs, and somewhere for logs to go:
# ecs.tf
resource "aws_ecs_cluster" "main" {
name = "${var.project_name}-cluster"
}
resource "aws_iam_role" "ecs_task_execution" {
name = "${var.project_name}-ecs-execution-role"
assume_role_policy = jsonencode({
Version = "2012-10-17"
Statement = [
{
Action = "sts:AssumeRole"
Effect = "Allow"
Principal = { Service = "ecs-tasks.amazonaws.com" }
}
]
})
}
resource "aws_iam_role_policy_attachment" "ecs_task_execution" {
role = aws_iam_role.ecs_task_execution.name
policy_arn = "arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy"
}
resource "aws_cloudwatch_log_group" "app" {
name = "/ecs/${var.project_name}"
retention_in_days = 7
}
Next, the Task Definition, essentially the recipe for your container: which image to use, how much CPU and memory it needs, and which port it listens on.
resource "aws_ecs_task_definition" "app" {
family = "${var.project_name}-task"
requires_compatibilities = ["FARGATE"]
network_mode = "awsvpc"
cpu = "256"
memory = "512"
execution_role_arn = aws_iam_role.ecs_task_execution.arn
container_definitions = jsonencode([
{
name = "${var.project_name}-container"
image = "${aws_ecr_repository.app.repository_url}:latest"
essential = true
portMappings = [
{ containerPort = 3000, protocol = "tcp" }
]
environment = [
{ name = "CHAOS_SECRET", value = var.chaos_secret }
]
logConfiguration = {
logDriver = "awslogs"
options = {
"awslogs-group" = aws_cloudwatch_log_group.app.name
"awslogs-region" = var.aws_region
"awslogs-stream-prefix" = "ecs"
}
}
}
])
}
A quick note on that CHAOS_SECRET environment variable: it comes from a Terraform variable with no default value, which forces you to supply it yourself in a local terraform.tfvars file, one that's excluded from git with a .gitignore rule. That way the secret never ends up committed to your repo.
# variables.tf (add this)
variable "chaos_secret" {
type = string
sensitive = true
}
# terraform.tfvars (never commit this file!)
chaos_secret = "some-long-random-string-only-you-know"
# .gitignore
.terraform/
*.tfstate
*.tfstate.*
*.tfvars
Finally, the ECS Service, the piece that actually keeps containers running, places them in the private subnets, and registers them with the load balancer:
resource "aws_ecs_service" "app" {
name = "${var.project_name}-service"
cluster = aws_ecs_cluster.main.id
task_definition = aws_ecs_task_definition.app.arn
desired_count = 2
launch_type = "FARGATE"
deployment_circuit_breaker {
enable = true
rollback = true
}
network_configuration {
subnets = aws_subnet.private[*].id
security_groups = [aws_security_group.ecs_service.id]
assign_public_ip = false
}
load_balancer {
target_group_arn = aws_lb_target_group.app.arn
container_name = "${var.project_name}-container"
container_port = 3000
}
depends_on = [aws_lb_listener.http]
}
Two details worth calling out:
- desired_count = 2 is the actual resilience piece. If one container crashes or an entire Availability Zone has a bad day, the other one keeps serving traffic while ECS quietly replaces the failed one.
- deployment_circuit_breaker protects you from a different kind of failure: a genuinely broken deploy. If a new version of your code crashes on startup, ECS will detect that the deployment isn't working and automatically roll back to the last known-good version. We'll test this properly later on.
Apply everything and check that the app responds:
terraform apply
curl http://<your-alb-dns-name>
Step 9: Watching It Heal Itself
Here's the fun part. Let's actually break something on purpose and watch the system fix itself.
Open a second terminal and poll the app continuously:
while ($true) {
try {
$r = Invoke-WebRequest -UseBasicParsing -Uri "http://<your-alb-dns-name>" -TimeoutSec 3
Write-Host "$(Get-Date -Format 'HH:mm:ss') - Status: $($r.StatusCode)" -ForegroundColor Green
} catch {
Write-Host "$(Get-Date -Format 'HH:mm:ss') - FAILED" -ForegroundColor Red
}
Start-Sleep -Seconds 1
}
Then, in your original terminal, trigger the chaos endpoint:
curl -H "X-Chaos-Secret: your-actual-secret" http://<your-alb-dns-name>/chaos/toggle-unhealthy
This flips one container into "I am not okay" mode. Watch your polling window, and check the target health directly:
aws elbv2 describe-target-health --target-group-arn <your-target-group-arn>
Here's what happens behind the scenes, roughly on this timeline:
- The broken container starts failing its /health checks
- After 3 failed checks in a row (about 45 seconds), the ALB marks it unhealthy and stops routing to it
- ECS notices the task is unhealthy too, and replaces it with a fresh container
- The new container boots up healthy and rejoins the group
Meanwhile, your other container has been serving every single request the entire time. That's the whole point of running more than one replica behind a health-checked load balancer.
If you're feeling bold, you can even break both containers at the same time by calling the chaos endpoint twice in a row. In my own testing, ECS launched two replacement containers in parallel and had a healthy one back online in under 20 seconds. Genuinely zero visible downtime, even under a double failure.
Step 10: Seeing What's Happening (CloudWatch)
Breaking things is fun, but you also want to actually see it happening without digging through CLI commands every time. Let's add a CloudWatch dashboard.
# monitoring.tf
resource "aws_cloudwatch_dashboard" "main" {
dashboard_name = "${var.project_name}-dashboard"
dashboard_body = jsonencode({
widgets = [
{
type = "metric"
x = 0, y = 0, width = 12, height = 6
properties = {
title = "Healthy vs Unhealthy Targets"
view = "timeSeries"
region = var.aws_region
metrics = [
["AWS/ApplicationELB", "HealthyHostCount", "TargetGroup", aws_lb_target_group.app.arn_suffix, "LoadBalancer", aws_lb.main.arn_suffix, { label = "Healthy" }],
["AWS/ApplicationELB", "UnHealthyHostCount", "TargetGroup", aws_lb_target_group.app.arn_suffix, "LoadBalancer", aws_lb.main.arn_suffix, { label = "Unhealthy" }]
]
period = 60
}
}
]
})
}
Next, let's get alerted the moment something actually breaks, rather than having to notice it ourselves. This uses an SNS topic (basically a notification channel) with an email subscription, plus a CloudWatch alarm that watches for unhealthy targets.
resource "aws_sns_topic" "alerts" {
name = "${var.project_name}-alerts"
}
resource "aws_sns_topic_subscription" "email" {
topic_arn = aws_sns_topic.alerts.arn
protocol = "email"
endpoint = var.alert_email
}
resource "aws_cloudwatch_metric_alarm" "unhealthy_targets" {
alarm_name = "${var.project_name}-unhealthy-targets"
comparison_operator = "GreaterThanThreshold"
evaluation_periods = 1
metric_name = "UnHealthyHostCount"
namespace = "AWS/ApplicationELB"
period = 60
statistic = "Maximum"
threshold = 0
treat_missing_data = "notBreaching"
alarm_actions = [aws_sns_topic.alerts.arn]
ok_actions = [aws_sns_topic.alerts.arn]
dimensions = {
TargetGroup = aws_lb_target_group.app.arn_suffix
LoadBalancer = aws_lb.main.arn_suffix
}
}
After applying, check your inbox for a subscription confirmation email from AWS and click it, alerts won't fire until you do.
Step 11: Shipping Code Automatically (CI/CD)
Manually building, tagging, and pushing Docker images gets old fast. Let's automate it with GitHub Actions, so every push to main builds a fresh image and deploys it.
We'll authenticate using OIDC (OpenID Connect) instead of storing long-lived AWS access keys inside GitHub. In plain terms, GitHub proves its identity to AWS using a short-lived, automatically rotating token, so there's no static secret sitting around that could ever leak.
First, tell AWS to trust GitHub as an identity provider, and create a tightly-scoped role that only your specific repo can assume:
# cicd.tf
resource "aws_iam_openid_connect_provider" "github" {
url = "https://token.actions.githubusercontent.com"
client_id_list = ["sts.amazonaws.com"]
thumbprint_list = ["6938fd4d98bab03faadb97b34396831e3780aea1"]
}
resource "aws_iam_role" "github_actions" {
name = "${var.project_name}-github-actions"
assume_role_policy = jsonencode({
Version = "2012-10-17"
Statement = [
{
Effect = "Allow"
Principal = { Federated = aws_iam_openid_connect_provider.github.arn }
Action = "sts:AssumeRoleWithWebIdentity"
Condition = {
StringEquals = {
"token.actions.githubusercontent.com:aud" = "sts.amazonaws.com"
}
StringLike = {
"token.actions.githubusercontent.com:sub" = "repo:${split("/", var.github_repo)[0]}@*/${split("/", var.github_repo)[1]}@*:*"
}
}
}
]
})
}
resource "aws_iam_role_policy" "github_actions" {
name = "${var.project_name}-github-actions-policy"
role = aws_iam_role.github_actions.id
policy = jsonencode({
Version = "2012-10-17"
Statement = [
{ Effect = "Allow", Action = ["ecr:GetAuthorizationToken"], Resource = "*" },
{
Effect = "Allow"
Action = [
"ecr:BatchCheckLayerAvailability", "ecr:GetDownloadUrlForLayer",
"ecr:BatchGetImage", "ecr:PutImage", "ecr:InitiateLayerUpload",
"ecr:UploadLayerPart", "ecr:CompleteLayerUpload"
]
Resource = aws_ecr_repository.app.arn
},
{
Effect = "Allow"
Action = ["ecs:UpdateService", "ecs:DescribeServices", "ecs:DescribeTaskDefinition", "ecs:RegisterTaskDefinition"]
Resource = "*"
},
{
Effect = "Allow"
Action = ["iam:PassRole"]
Resource = aws_iam_role.ecs_task_execution.arn
}
]
})
}
A small but important detail: GitHub's OIDC tokens actually include the numeric account and repo IDs baked into the subject claim (something like repo:owner@12345/repo-name@67890:ref:refs/heads/main), not just the plain owner/repo text. The @* wildcards in the condition above account for that. Worth knowing before you find yourself puzzled over an "access denied" error that seems to make no sense.
Notice how narrow that policy is: it can push to this specific ECR repository, update this specific ECS service, and nothing else. Not admin access, can't touch your VPC, can't delete anything. If this token ever leaked somehow, the blast radius is tiny.
Now the actual workflow file, at .github/workflows/deploy.yml:
name: Build and Deploy
on:
push:
branches: [main]
paths:
- 'app/**'
permissions:
id-token: write
contents: read
env:
AWS_REGION: eu-west-1
ECR_REPOSITORY: resilient-aws-infra-app
ECS_CLUSTER: resilient-aws-infra-cluster
ECS_SERVICE: resilient-aws-infra-service
ECS_TASK_FAMILY: resilient-aws-infra-task
CONTAINER_NAME: resilient-aws-infra-container
jobs:
build-and-deploy:
runs-on: ubuntu-latest
steps:
- name: Checkout code
uses: actions/checkout@v4
- name: Configure AWS credentials via OIDC
uses: aws-actions/configure-aws-credentials@v4
with:
role-to-assume: arn:aws:iam::123456789012:role/resilient-aws-infra-github-actions
aws-region: ${{ env.AWS_REGION }}
- name: Login to Amazon ECR
id: login-ecr
uses: aws-actions/amazon-ecr-login@v2
- name: Build, tag, and push image
id: build-image
env:
ECR_REGISTRY: ${{ steps.login-ecr.outputs.registry }}
IMAGE_TAG: ${{ github.sha }}
run: |
docker build -t $ECR_REGISTRY/$ECR_REPOSITORY:$IMAGE_TAG -t $ECR_REGISTRY/$ECR_REPOSITORY:latest ./app
docker push $ECR_REGISTRY/$ECR_REPOSITORY:$IMAGE_TAG
docker push $ECR_REGISTRY/$ECR_REPOSITORY:latest
echo "image=$ECR_REGISTRY/$ECR_REPOSITORY:$IMAGE_TAG" >> $GITHUB_OUTPUT
- name: Download current task definition
run: |
aws ecs describe-task-definition --task-definition $ECS_TASK_FAMILY \
--query taskDefinition > task-definition.json
- name: Fill in new image ID in task definition
id: task-def
uses: aws-actions/amazon-ecs-render-task-definition@v1
with:
task-definition: task-definition.json
container-name: ${{ env.CONTAINER_NAME }}
image: ${{ steps.build-image.outputs.image }}
- name: Deploy to ECS
uses: aws-actions/amazon-ecs-deploy-task-definition@v2
with:
task-definition: ${{ steps.task-def.outputs.task-definition }}
service: ${{ env.ECS_SERVICE }}
cluster: ${{ env.ECS_CLUSTER }}
wait-for-service-stability: true
Notice the paths: - 'app/**' filter near the top. This means the pipeline only triggers when files inside app/ change, so a pure infrastructure tweak doesn't kick off an unnecessary rebuild.
Push a small change to app/server.js, and watch the magic happen in your repo's Actions tab.
Step 12: Testing a Bad Deploy on Purpose
Here's a scenario every developer has lived through: you ship code that's subtly broken, and it crashes on startup. What happens?
Let's simulate it. Add this to server.js, right after the chaos secret check:
// Simulate a bad deploy: crash on startup if this "required" config is missing.
if (!process.env.REQUIRED_CONFIG_VALUE) {
console.error('FATAL: REQUIRED_CONFIG_VALUE is not set. Exiting.');
process.exit(1);
}
We deliberately haven't set that environment variable anywhere, so this simulates a classic misconfigured deploy. Push it to main and watch what happens.
Here's the sequence I watched play out on my own deploy:
- GitHub Actions builds and pushes the broken image, then asks ECS to deploy it
- ECS tries to start new containers with the broken image, they crash almost immediately
- ECS retries a few times (it's patient, giving the deployment a fair shot)
- After enough failures, the deployment circuit breaker kicks in: "deployment failed: tasks failed to start", immediately followed by "rolling back to deployment..."
- ECS automatically reverts to the last known-good version
- Your users never see a thing, the old, working containers kept serving traffic the entire time
That's the deployment circuit breaker doing exactly what it's for. Once you're done proving the point, fix the "bug" properly by actually supplying the config value in your task definition, and push a clean deploy:
environment = [
{ name = "CHAOS_SECRET", value = var.chaos_secret },
{ name = "REQUIRED_CONFIG_VALUE", value = "production" }
]
Cleaning Up
Once you're done experimenting, don't forget to tear everything down. The NAT Gateway in particular bills a small hourly fee for as long as it exists.
cd infra
terraform destroy
One small gotcha: if you've pushed images to ECR, Terraform will refuse to delete the (non-empty) repository. Clear the images first:
aws ecr batch-delete-image \
--repository-name resilient-aws-infra-app \
--image-ids "$(aws ecr list-images --repository-name resilient-aws-infra-app --query 'imageIds[*]' --output json)"
terraform destroy
What This Project Actually Proves
Stepping back, here's what got genuinely tested, not just built and left sitting there:
- Single container failure — one container breaks, the other keeps serving traffic, ECS replaces the broken one automatically
- Double container failure — both containers break at once, ECS replaces both in parallel, still zero visible downtime
- Bad deploy protection — genuinely broken code gets caught before it ever reaches real users, and rolls back on its own
- Alerting — the moment something breaks, an email lands in your inbox, no manual checking required
- Automated, secure deploys — every push to main ships safely, with no long-lived credentials sitting in GitHub
None of this was "trust me, it should work in theory". Every single piece was deliberately broken and watched recover, with logs and screenshots to prove it.
Wrapping Up
Building something like this yourself teaches you more in an afternoon than reading ten articles about "cloud resilience" ever will. Break things on purpose. Watch what actually happens. Read the logs when it doesn't go the way you expected.
The full code for everything covered here is available on GitHub:
Clone it, terraform apply it, break it, and see for yourself.
If you build on this or spot something worth improving, I'd love to hear about it. Happy building!
Top comments (0)