DEV Community: Adesanya Adedeji Stephen

Building Self-Healing Infrastructure on AWS: A Hands-On Guide

Adesanya Adedeji Stephen — Wed, 12 Aug 2026 15:20:22 +0000

Ever wondered what actually happens when a server crashes in production? Not in theory, but for real. Does your app just die? Does anyone notice? Does anything fix itself?
I decided to stop wondering and build the answer myself. This article walks through a project where I built a small, genuinely self-healing web application on AWS: containers that survive being deliberately broken, a load balancer that routes around failure automatically, a CI/CD pipeline that ships code safely, and alerts that tell me when things go wrong.
By the end of this article, you'll have replicated the whole thing yourself, with your own hands on the keyboard, not just read about it.
Let's get into it.

What We're Actually Building

Here's the bird's eye view before we touch a single line of code:

A small Node.js app with a real health check endpoint (not just "am I alive", but "am I actually okay")
A VPC spread across two AWS Availability Zones, with public and private subnets
The app running as containers on AWS Fargate (serverless containers, no EC2 servers to babysit)
An Application Load Balancer that only sends traffic to healthy containers
A CI/CD pipeline with GitHub Actions that builds and deploys automatically on every push
CloudWatch dashboards and alarms, wired to email alerts via SNS
A chaos endpoint to deliberately break things on purpose, so we can watch the system heal itself

All of the infrastructure is written in Terraform, so it's repeatable, version-controlled, and destroyable with a single command when you're done experimenting.

You can find the full, finished code in the repo here: View the GitHub Repository

Prerequisites

Before we start, make sure you've got:

An AWS account (with a non-root IAM user set up, please don't use root credentials day to day)
Terraform installed locally
Docker Desktop installed and running
The AWS CLI installed and configured (aws configure)
A GitHub account and a new empty repo to push your code to
Basic comfort with the command line (we'll be doing a lot of terraform apply)

Quick note throughout this article: anywhere you see the account ID 123456789012, swap it for your own AWS account ID. That number is a placeholder, not a real account.

Step 1: The Skeleton

Start with a simple folder structure to keep the application code and the infrastructure code cleanly separated:

resilient-aws-infra/
├── app/ # The Node.js application + Dockerfile
├── infra/ # All the Terraform code
└── .github/
└── workflows/ # CI/CD pipeline definition

_Keeping app/ and infra/ separate is a small thing, but it pays off. It means your CI/CD pipeline can be told "only rebuild when the app changes", which we'll use later.
_

Step 2: The App

Nothing fancy here. Just a tiny Express app with three routes:

/ — a homepage that says hello
/health — a real health check, one that actually checks something (memory usage), not just "the process is running"
/chaos/toggle-unhealthy — a hidden switch we'll use later to deliberately break the app on purpose

const express = require('express');
const app = express();
const PORT = process.env.PORT || 3000;

let forceUnhealthy = false;

app.get('/', (req, res) => {
  res.send('<h1>Resilient AWS Infra - it works</h1>');
});

// A REAL health check - not just "am I running", but "am I actually okay"
app.get('/health', (req, res) => {
  const memoryUsedMB = process.memoryUsage().heapUsed / 1024 / 1024;
  const isHealthy = !forceUnhealthy && memoryUsedMB < 200;

  if (isHealthy) {
    res.status(200).json({ status: 'healthy', memoryMB: memoryUsedMB.toFixed(1) });
  } else {
    res.status(503).json({ status: 'unhealthy', memoryMB: memoryUsedMB.toFixed(1), forced: forceUnhealthy });
  }
});

// A deliberate "break glass" switch, locked behind a secret header
const CHAOS_SECRET = process.env.CHAOS_SECRET || 'change-me';

app.get('/chaos/toggle-unhealthy', (req, res) => {
  const providedSecret = req.header('X-Chaos-Secret');

  if (providedSecret !== CHAOS_SECRET) {
    return res.status(403).json({ error: 'Forbidden' });
  }

  forceUnhealthy = !forceUnhealthy;
  res.json({ forceUnhealthy });
});

app.listen(PORT, () => {
  console.log(`Server running on port ${PORT}`);
});

And the matching package.json:

{
  "name": "resilient-aws-infra-app",
  "version": "1.0.0",
  "description": "Small demo app for practicing incident response on AWS",
  "main": "server.js",
  "scripts": {
    "start": "node server.js"
  },
  "dependencies": {
    "express": "^4.19.2"
  }
}

That /chaos endpoint might look reckless, but notice it's locked behind a secret header. We'll wire that secret in properly through Terraform later, so it's never sitting around in plain text.

Step 3: Dockerize It

Next, we package the app into a container. Create a Dockerfile inside app/:

FROM node:20-alpine

WORKDIR /app

COPY package.json ./
RUN npm install --production

COPY server.js ./

EXPOSE 3000

CMD ["node", "server.js"]

Quick breakdown:

FROM node:20-alpine starts from a small, lightweight image that already has Node.js
WORKDIR /app sets our working folder inside the container
We copy package.json first and install dependencies before copying the actual code. This is a small trick that speeds up rebuilds, since Docker can reuse the cached dependency layer if only your code changed
EXPOSE 3000 documents which port the app listens on
CMD is what runs when the container starts

Heads up on Windows: if you're using Notepad to create this file, it'll often save it as Dockerfile.txt instead of Dockerfile. Docker specifically looks for a file named exactly Dockerfile, no extension. Rename it if needed:

Rename-Item Dockerfile.txt Dockerfile

Test the build locally before moving on:

docker build -t resilient-aws-infra-app .

Step 4: The Networking Foundation (VPC)

This is where we start writing Terraform. Everything from here on lives in the infra/ folder.

First, tell Terraform which cloud provider and region we're using:

# providers.tf
terraform {
  required_version = ">= 1.5.0"

  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.0"
    }
  }
}

provider "aws" {
  region = "eu-west-1"
}

Then define some reusable variables so we're not hardcoding values everywhere:

# variables.tf
variable "project_name" {
  type    = string
  default = "resilient-aws-infra"
}

variable "aws_region" {
  type    = string
  default = "eu-west-1"
}

variable "vpc_cidr" {
  type    = string
  default = "10.0.0.0/16"
}

variable "public_subnet_cidrs" {
  type    = list(string)
  default = ["10.0.1.0/24", "10.0.2.0/24"]
}

variable "private_subnet_cidrs" {
  type    = list(string)
  default = ["10.0.11.0/24", "10.0.12.0/24"]
}

variable "availability_zones" {
  type    = list(string)
  default = ["eu-west-1a", "eu-west-1b"]
}

Now the actual network. This creates a VPC (your own private slice of AWS), spread across two Availability Zones for resilience, with public subnets (reachable from the internet) and private subnets (where our containers will actually live):

# main.tf
resource "aws_vpc" "main" {
  cidr_block           = var.vpc_cidr
  enable_dns_support   = true
  enable_dns_hostnames = true

  tags = {
    Name = "${var.project_name}-vpc"
  }
}

resource "aws_internet_gateway" "main" {
  vpc_id = aws_vpc.main.id

  tags = {
    Name = "${var.project_name}-igw"
  }
}

resource "aws_subnet" "public" {
  count                   = length(var.public_subnet_cidrs)
  vpc_id                  = aws_vpc.main.id
  cidr_block              = var.public_subnet_cidrs[count.index]
  availability_zone       = var.availability_zones[count.index]
  map_public_ip_on_launch = true

  tags = {
    Name = "${var.project_name}-public-${var.availability_zones[count.index]}"
  }
}

resource "aws_subnet" "private" {
  count             = length(var.private_subnet_cidrs)
  vpc_id            = aws_vpc.main.id
  cidr_block        = var.private_subnet_cidrs[count.index]
  availability_zone = var.availability_zones[count.index]

  tags = {
    Name = "${var.project_name}-private-${var.availability_zones[count.index]}"
  }
}

resource "aws_eip" "nat" {
  domain = "vpc"
}

resource "aws_nat_gateway" "main" {
  allocation_id = aws_eip.nat.id
  subnet_id     = aws_subnet.public[0].id

  depends_on = [aws_internet_gateway.main]
}

resource "aws_route_table" "public" {
  vpc_id = aws_vpc.main.id

  route {
    cidr_block = "0.0.0.0/0"
    gateway_id = aws_internet_gateway.main.id
  }
}

resource "aws_route_table" "private" {
  vpc_id = aws_vpc.main.id

  route {
    cidr_block     = "0.0.0.0/0"
    nat_gateway_id = aws_nat_gateway.main.id
  }
}

resource "aws_route_table_association" "public" {
  count          = length(aws_subnet.public)
  subnet_id      = aws_subnet.public[count.index].id
  route_table_id = aws_route_table.public.id
}

resource "aws_route_table_association" "private" {
  count          = length(aws_subnet.private)
  subnet_id      = aws_subnet.private[count.index].id
  route_table_id = aws_route_table.private.id
}

A quick mental model for the pieces above:

Public subnets are the front door. Things here can be reached from the internet, like our load balancer.
Private subnets are the back room. Our actual containers live here, never directly reachable from the outside.
The Internet Gateway lets public subnets talk to the internet.
The NAT Gateway lets private subnets reach out (to pull Docker images, for example) without letting the internet reach in.

Run terraform init, then terraform plan to preview it, and terraform apply to build it for real.

terraform init
terraform plan
terraform apply

Step 5: A Home for Your Docker Image (ECR)

Before we can run our container in AWS, it needs somewhere to live. That's ECR (Elastic Container Registry), basically a private Docker Hub inside your own AWS account.

# ecr.tf
resource "aws_ecr_repository" "app" {
  name                 = "${var.project_name}-app"
  image_tag_mutability = "MUTABLE"

  image_scanning_configuration {
    scan_on_push = true
  }
}

output "ecr_repository_url" {
  value = aws_ecr_repository.app.repository_url
}

scan_on_push = true is a nice little freebie: AWS automatically scans every image you push for known vulnerabilities, at no extra cost.

Apply it, then build and push your image:

terraform apply

# Authenticate Docker to your new ECR repo
aws ecr get-login-password --region eu-west-1 | docker login --username AWS --password-stdin 123456789012.dkr.ecr.eu-west-1.amazonaws.com

# Build, tag, and push
docker build -t resilient-aws-infra-app .
docker tag resilient-aws-infra-app:latest 123456789012.dkr.ecr.eu-west-1.amazonaws.com/resilient-aws-infra-app:latest
docker push 123456789012.dkr.ecr.eu-west-1.amazonaws.com/resilient-aws-infra-app:latest

Step 6: Security Groups (The Firewall Rules)

Before we build the load balancer and the containers, let's set up the security groups that control exactly what can talk to what. Think of these as a strict guest list.

# security-groups.tf

# The load balancer can be reached by anyone, on port 80
resource "aws_security_group" "alb" {
  name        = "${var.project_name}-alb-sg"
  description = "Allow inbound HTTP from the internet"
  vpc_id      = aws_vpc.main.id

  ingress {
    from_port   = 80
    to_port     = 80
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }

  egress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }
}

# The containers can ONLY be reached by the load balancer, on port 3000
resource "aws_security_group" "ecs_service" {
  name        = "${var.project_name}-ecs-sg"
  description = "Allow inbound traffic only from the ALB"
  vpc_id      = aws_vpc.main.id

  ingress {
    from_port       = 3000
    to_port         = 3000
    protocol        = "tcp"
    security_groups = [aws_security_group.alb.id]
  }

  egress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }
}

Notice the container security group doesn't open port 3000 to the whole internet, it only trusts traffic coming from the load balancer's own security group. Your containers are never directly exposed.

Step 7: The Load Balancer

This is the piece that actually makes resilience possible. The Application Load Balancer (ALB) sits in front of your containers, continuously checking their health, and only sends traffic to the ones that pass.

# alb.tf
resource "aws_lb" "main" {
  name               = "${var.project_name}-alb"
  internal           = false
  load_balancer_type = "application"
  security_groups    = [aws_security_group.alb.id]
  subnets            = aws_subnet.public[*].id
}

resource "aws_lb_target_group" "app" {
  name        = "${var.project_name}-tg"
  port        = 3000
  protocol    = "HTTP"
  vpc_id      = aws_vpc.main.id
  target_type = "ip"

  health_check {
    path                = "/health"
    port                = "traffic-port"
    protocol            = "HTTP"
    healthy_threshold   = 2
    unhealthy_threshold = 3
    timeout             = 5
    interval            = 15
    matcher             = "200"
  }
}

resource "aws_lb_listener" "http" {
  load_balancer_arn = aws_lb.main.arn
  port               = 80
  protocol           = "HTTP"

  default_action {
    type             = "forward"
    target_group_arn = aws_lb_target_group.app.arn
  }
}

output "alb_dns_name" {
  value = aws_lb.main.dns_name
}

That health_check block is doing the real work here. Every 15 seconds, it hits /health. If it doesn't get a 200 response within 5 seconds, and this happens 3 times in a row, the ALB marks that specific container "unhealthy" and quietly stops sending it traffic. No drama, no downtime for users, just a clean handoff to the containers that are actually working.

Step 8: ECS (Where the Container Actually Runs)

Now for the piece that runs your container. We're using AWS Fargate, which is "serverless containers": you don't manage any EC2 instances, AWS just runs your container for you.

First, the cluster, the IAM role ECS needs, and somewhere for logs to go:

# ecs.tf
resource "aws_ecs_cluster" "main" {
  name = "${var.project_name}-cluster"
}

resource "aws_iam_role" "ecs_task_execution" {
  name = "${var.project_name}-ecs-execution-role"

  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Action    = "sts:AssumeRole"
        Effect    = "Allow"
        Principal = { Service = "ecs-tasks.amazonaws.com" }
      }
    ]
  })
}

resource "aws_iam_role_policy_attachment" "ecs_task_execution" {
  role       = aws_iam_role.ecs_task_execution.name
  policy_arn = "arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy"
}

resource "aws_cloudwatch_log_group" "app" {
  name              = "/ecs/${var.project_name}"
  retention_in_days = 7
}

Next, the Task Definition, essentially the recipe for your container: which image to use, how much CPU and memory it needs, and which port it listens on.

resource "aws_ecs_task_definition" "app" {
  family                   = "${var.project_name}-task"
  requires_compatibilities = ["FARGATE"]
  network_mode             = "awsvpc"
  cpu                      = "256"
  memory                   = "512"
  execution_role_arn       = aws_iam_role.ecs_task_execution.arn

  container_definitions = jsonencode([
    {
      name      = "${var.project_name}-container"
      image     = "${aws_ecr_repository.app.repository_url}:latest"
      essential = true
      portMappings = [
        { containerPort = 3000, protocol = "tcp" }
      ]
      environment = [
        { name = "CHAOS_SECRET", value = var.chaos_secret }
      ]
      logConfiguration = {
        logDriver = "awslogs"
        options = {
          "awslogs-group"         = aws_cloudwatch_log_group.app.name
          "awslogs-region"        = var.aws_region
          "awslogs-stream-prefix" = "ecs"
        }
      }
    }
  ])
}

A quick note on that CHAOS_SECRET environment variable: it comes from a Terraform variable with no default value, which forces you to supply it yourself in a local terraform.tfvars file, one that's excluded from git with a .gitignore rule. That way the secret never ends up committed to your repo.

# variables.tf (add this)
variable "chaos_secret" {
  type      = string
  sensitive = true
}

# terraform.tfvars (never commit this file!)
chaos_secret = "some-long-random-string-only-you-know"

# .gitignore
.terraform/
*.tfstate
*.tfstate.*
*.tfvars

Finally, the ECS Service, the piece that actually keeps containers running, places them in the private subnets, and registers them with the load balancer:

resource "aws_ecs_service" "app" {
  name            = "${var.project_name}-service"
  cluster         = aws_ecs_cluster.main.id
  task_definition = aws_ecs_task_definition.app.arn
  desired_count   = 2
  launch_type     = "FARGATE"

  deployment_circuit_breaker {
    enable   = true
    rollback = true
  }

  network_configuration {
    subnets          = aws_subnet.private[*].id
    security_groups  = [aws_security_group.ecs_service.id]
    assign_public_ip = false
  }

  load_balancer {
    target_group_arn = aws_lb_target_group.app.arn
    container_name    = "${var.project_name}-container"
    container_port    = 3000
  }

  depends_on = [aws_lb_listener.http]
}

Two details worth calling out:

desired_count = 2 is the actual resilience piece. If one container crashes or an entire Availability Zone has a bad day, the other one keeps serving traffic while ECS quietly replaces the failed one.
deployment_circuit_breaker protects you from a different kind of failure: a genuinely broken deploy. If a new version of your code crashes on startup, ECS will detect that the deployment isn't working and automatically roll back to the last known-good version. We'll test this properly later on.

Apply everything and check that the app responds:

terraform apply

curl http://<your-alb-dns-name>

Step 9: Watching It Heal Itself

Here's the fun part. Let's actually break something on purpose and watch the system fix itself.

Open a second terminal and poll the app continuously:

while ($true) {
    try {
        $r = Invoke-WebRequest -UseBasicParsing -Uri "http://<your-alb-dns-name>" -TimeoutSec 3
        Write-Host "$(Get-Date -Format 'HH:mm:ss') - Status: $($r.StatusCode)" -ForegroundColor Green
    } catch {
        Write-Host "$(Get-Date -Format 'HH:mm:ss') - FAILED" -ForegroundColor Red
    }
    Start-Sleep -Seconds 1
}

Then, in your original terminal, trigger the chaos endpoint:

curl -H "X-Chaos-Secret: your-actual-secret" http://<your-alb-dns-name>/chaos/toggle-unhealthy

This flips one container into "I am not okay" mode. Watch your polling window, and check the target health directly:

aws elbv2 describe-target-health --target-group-arn <your-target-group-arn>

Here's what happens behind the scenes, roughly on this timeline:

The broken container starts failing its /health checks
After 3 failed checks in a row (about 45 seconds), the ALB marks it unhealthy and stops routing to it
ECS notices the task is unhealthy too, and replaces it with a fresh container
The new container boots up healthy and rejoins the group

Meanwhile, your other container has been serving every single request the entire time. That's the whole point of running more than one replica behind a health-checked load balancer.

If you're feeling bold, you can even break both containers at the same time by calling the chaos endpoint twice in a row. In my own testing, ECS launched two replacement containers in parallel and had a healthy one back online in under 20 seconds. Genuinely zero visible downtime, even under a double failure.

Step 10: Seeing What's Happening (CloudWatch)

Breaking things is fun, but you also want to actually see it happening without digging through CLI commands every time. Let's add a CloudWatch dashboard.

# monitoring.tf
resource "aws_cloudwatch_dashboard" "main" {
  dashboard_name = "${var.project_name}-dashboard"

  dashboard_body = jsonencode({
    widgets = [
      {
        type   = "metric"
        x = 0, y = 0, width = 12, height = 6
        properties = {
          title  = "Healthy vs Unhealthy Targets"
          view   = "timeSeries"
          region = var.aws_region
          metrics = [
            ["AWS/ApplicationELB", "HealthyHostCount", "TargetGroup", aws_lb_target_group.app.arn_suffix, "LoadBalancer", aws_lb.main.arn_suffix, { label = "Healthy" }],
            ["AWS/ApplicationELB", "UnHealthyHostCount", "TargetGroup", aws_lb_target_group.app.arn_suffix, "LoadBalancer", aws_lb.main.arn_suffix, { label = "Unhealthy" }]
          ]
          period = 60
        }
      }
    ]
  })
}

Next, let's get alerted the moment something actually breaks, rather than having to notice it ourselves. This uses an SNS topic (basically a notification channel) with an email subscription, plus a CloudWatch alarm that watches for unhealthy targets.

resource "aws_sns_topic" "alerts" {
  name = "${var.project_name}-alerts"
}

resource "aws_sns_topic_subscription" "email" {
  topic_arn = aws_sns_topic.alerts.arn
  protocol  = "email"
  endpoint  = var.alert_email
}

resource "aws_cloudwatch_metric_alarm" "unhealthy_targets" {
  alarm_name          = "${var.project_name}-unhealthy-targets"
  comparison_operator = "GreaterThanThreshold"
  evaluation_periods  = 1
  metric_name         = "UnHealthyHostCount"
  namespace           = "AWS/ApplicationELB"
  period              = 60
  statistic           = "Maximum"
  threshold           = 0
  treat_missing_data  = "notBreaching"
  alarm_actions       = [aws_sns_topic.alerts.arn]
  ok_actions          = [aws_sns_topic.alerts.arn]

  dimensions = {
    TargetGroup  = aws_lb_target_group.app.arn_suffix
    LoadBalancer = aws_lb.main.arn_suffix
  }
}

After applying, check your inbox for a subscription confirmation email from AWS and click it, alerts won't fire until you do.

Step 11: Shipping Code Automatically (CI/CD)

Manually building, tagging, and pushing Docker images gets old fast. Let's automate it with GitHub Actions, so every push to main builds a fresh image and deploys it.

We'll authenticate using OIDC (OpenID Connect) instead of storing long-lived AWS access keys inside GitHub. In plain terms, GitHub proves its identity to AWS using a short-lived, automatically rotating token, so there's no static secret sitting around that could ever leak.

First, tell AWS to trust GitHub as an identity provider, and create a tightly-scoped role that only your specific repo can assume:

# cicd.tf
resource "aws_iam_openid_connect_provider" "github" {
  url             = "https://token.actions.githubusercontent.com"
  client_id_list  = ["sts.amazonaws.com"]
  thumbprint_list = ["6938fd4d98bab03faadb97b34396831e3780aea1"]
}

resource "aws_iam_role" "github_actions" {
  name = "${var.project_name}-github-actions"

  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Effect    = "Allow"
        Principal = { Federated = aws_iam_openid_connect_provider.github.arn }
        Action    = "sts:AssumeRoleWithWebIdentity"
        Condition = {
          StringEquals = {
            "token.actions.githubusercontent.com:aud" = "sts.amazonaws.com"
          }
          StringLike = {
            "token.actions.githubusercontent.com:sub" = "repo:${split("/", var.github_repo)[0]}@*/${split("/", var.github_repo)[1]}@*:*"
          }
        }
      }
    ]
  })
}

resource "aws_iam_role_policy" "github_actions" {
  name = "${var.project_name}-github-actions-policy"
  role = aws_iam_role.github_actions.id

  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      { Effect = "Allow", Action = ["ecr:GetAuthorizationToken"], Resource = "*" },
      {
        Effect = "Allow"
        Action = [
          "ecr:BatchCheckLayerAvailability", "ecr:GetDownloadUrlForLayer",
          "ecr:BatchGetImage", "ecr:PutImage", "ecr:InitiateLayerUpload",
          "ecr:UploadLayerPart", "ecr:CompleteLayerUpload"
        ]
        Resource = aws_ecr_repository.app.arn
      },
      {
        Effect   = "Allow"
        Action   = ["ecs:UpdateService", "ecs:DescribeServices", "ecs:DescribeTaskDefinition", "ecs:RegisterTaskDefinition"]
        Resource = "*"
      },
      {
        Effect   = "Allow"
        Action   = ["iam:PassRole"]
        Resource = aws_iam_role.ecs_task_execution.arn
      }
    ]
  })
}

A small but important detail: GitHub's OIDC tokens actually include the numeric account and repo IDs baked into the subject claim (something like repo:owner@12345/repo-name@67890:ref:refs/heads/main), not just the plain owner/repo text. The @* wildcards in the condition above account for that. Worth knowing before you find yourself puzzled over an "access denied" error that seems to make no sense.

Notice how narrow that policy is: it can push to this specific ECR repository, update this specific ECS service, and nothing else. Not admin access, can't touch your VPC, can't delete anything. If this token ever leaked somehow, the blast radius is tiny.

Now the actual workflow file, at .github/workflows/deploy.yml:

name: Build and Deploy

on:
  push:
    branches: [main]
    paths:
      - 'app/**'

permissions:
  id-token: write
  contents: read

env:
  AWS_REGION: eu-west-1
  ECR_REPOSITORY: resilient-aws-infra-app
  ECS_CLUSTER: resilient-aws-infra-cluster
  ECS_SERVICE: resilient-aws-infra-service
  ECS_TASK_FAMILY: resilient-aws-infra-task
  CONTAINER_NAME: resilient-aws-infra-container

jobs:
  build-and-deploy:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout code
        uses: actions/checkout@v4

      - name: Configure AWS credentials via OIDC
        uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: arn:aws:iam::123456789012:role/resilient-aws-infra-github-actions
          aws-region: ${{ env.AWS_REGION }}

      - name: Login to Amazon ECR
        id: login-ecr
        uses: aws-actions/amazon-ecr-login@v2

      - name: Build, tag, and push image
        id: build-image
        env:
          ECR_REGISTRY: ${{ steps.login-ecr.outputs.registry }}
          IMAGE_TAG: ${{ github.sha }}
        run: |
          docker build -t $ECR_REGISTRY/$ECR_REPOSITORY:$IMAGE_TAG -t $ECR_REGISTRY/$ECR_REPOSITORY:latest ./app
          docker push $ECR_REGISTRY/$ECR_REPOSITORY:$IMAGE_TAG
          docker push $ECR_REGISTRY/$ECR_REPOSITORY:latest
          echo "image=$ECR_REGISTRY/$ECR_REPOSITORY:$IMAGE_TAG" >> $GITHUB_OUTPUT

      - name: Download current task definition
        run: |
          aws ecs describe-task-definition --task-definition $ECS_TASK_FAMILY \
            --query taskDefinition > task-definition.json

      - name: Fill in new image ID in task definition
        id: task-def
        uses: aws-actions/amazon-ecs-render-task-definition@v1
        with:
          task-definition: task-definition.json
          container-name: ${{ env.CONTAINER_NAME }}
          image: ${{ steps.build-image.outputs.image }}

      - name: Deploy to ECS
        uses: aws-actions/amazon-ecs-deploy-task-definition@v2
        with:
          task-definition: ${{ steps.task-def.outputs.task-definition }}
          service: ${{ env.ECS_SERVICE }}
          cluster: ${{ env.ECS_CLUSTER }}
          wait-for-service-stability: true

Notice the paths: - 'app/**' filter near the top. This means the pipeline only triggers when files inside app/ change, so a pure infrastructure tweak doesn't kick off an unnecessary rebuild.

Push a small change to app/server.js, and watch the magic happen in your repo's Actions tab.

Step 12: Testing a Bad Deploy on Purpose

Here's a scenario every developer has lived through: you ship code that's subtly broken, and it crashes on startup. What happens?

Let's simulate it. Add this to server.js, right after the chaos secret check:

// Simulate a bad deploy: crash on startup if this "required" config is missing.
if (!process.env.REQUIRED_CONFIG_VALUE) {
  console.error('FATAL: REQUIRED_CONFIG_VALUE is not set. Exiting.');
  process.exit(1);
}

We deliberately haven't set that environment variable anywhere, so this simulates a classic misconfigured deploy. Push it to main and watch what happens.

Here's the sequence I watched play out on my own deploy:

GitHub Actions builds and pushes the broken image, then asks ECS to deploy it
ECS tries to start new containers with the broken image, they crash almost immediately
ECS retries a few times (it's patient, giving the deployment a fair shot)
After enough failures, the deployment circuit breaker kicks in: "deployment failed: tasks failed to start", immediately followed by "rolling back to deployment..."
ECS automatically reverts to the last known-good version
Your users never see a thing, the old, working containers kept serving traffic the entire time

That's the deployment circuit breaker doing exactly what it's for. Once you're done proving the point, fix the "bug" properly by actually supplying the config value in your task definition, and push a clean deploy:

environment = [
  { name = "CHAOS_SECRET", value = var.chaos_secret },
  { name = "REQUIRED_CONFIG_VALUE", value = "production" }
]

Cleaning Up

Once you're done experimenting, don't forget to tear everything down. The NAT Gateway in particular bills a small hourly fee for as long as it exists.

cd infra
terraform destroy

One small gotcha: if you've pushed images to ECR, Terraform will refuse to delete the (non-empty) repository. Clear the images first:

aws ecr batch-delete-image \
  --repository-name resilient-aws-infra-app \
  --image-ids "$(aws ecr list-images --repository-name resilient-aws-infra-app --query 'imageIds[*]' --output json)"

terraform destroy

What This Project Actually Proves

Stepping back, here's what got genuinely tested, not just built and left sitting there:

Single container failure — one container breaks, the other keeps serving traffic, ECS replaces the broken one automatically
Double container failure — both containers break at once, ECS replaces both in parallel, still zero visible downtime
Bad deploy protection — genuinely broken code gets caught before it ever reaches real users, and rolls back on its own
Alerting — the moment something breaks, an email lands in your inbox, no manual checking required
Automated, secure deploys — every push to main ships safely, with no long-lived credentials sitting in GitHub

None of this was "trust me, it should work in theory". Every single piece was deliberately broken and watched recover, with logs and screenshots to prove it.

Wrapping Up

Building something like this yourself teaches you more in an afternoon than reading ten articles about "cloud resilience" ever will. Break things on purpose. Watch what actually happens. Read the logs when it doesn't go the way you expected.

The full code for everything covered here is available on GitHub:

View the GitHub Repository

Clone it, terraform apply it, break it, and see for yourself.

If you build on this or spot something worth improving, I'd love to hear about it. Happy building!

Building a Production-Ready CI/CD Pipeline on AWS: From Empty Terraform Files to a Live, Monitored, Auto-Deploying App

Adesanya Adedeji Stephen — Sun, 19 Jul 2026 21:38:22 +0000

Learn how to build a complete containerized deployment pipeline on AWS— VPC, ECR, ECS Fargate, an Application Load Balancer with HTTPS, CodePipeline/CodeBuild CI/CD, and CloudWatch alarting — using Terraform, across two isolated environments, and how to actually recover when things go wrong along the way.

Author: Adedeji Adesanya

Difficulty: Intermediate

Estimated Time: 6–8 Hours (across two sessions)

AWS Services Used:

Amazon VPC
Amazon ECR
AWS IAM
Application Load Balancer (ALB) + AWS Certificate Manager (ACM)
Amazon ECS (Fargate)
AWS CodePipeline + AWS CodeBuild
Amazon CloudWatch + Amazon SNS
Amazon S3 + DynamoDB (Terraform remote state)

Table of Contents

Introduction
What You Will Build
Solution Architecture
Prerequisites
Building the Foundation: VPC, ECR, IAM
Fronting the App: ALB and ECS Fargate
Getting the App Actually Running (and What Went Wrong)
Wiring Up CI/CD with CodePipeline
Adding HTTPS to the Load Balancer
The Terraform vs. Pipeline Conflict (and How to Fix It)
Monitoring and Alerting
Governance: Branch Protection and Remote State
Lessons Learned
Production Considerations
Conclusion

Introduction

Most Terraform tutorials show you a clean, linear path: write some HCL, run terraform apply, watch resources appear, done. That's useful for learning syntax, but it's not what real infrastructure work actually feels like.

This project started as a straightforward goal — deploy a containerized web app to AWS using Terraform, with a proper CI/CD pipeline behind it. What it turned into was a two-day exercise in debugging the kind of problems that don't show up in tutorials: stale module variables that didn't match between files, an empty .tf file that looked complete in an editor but was actually zero bytes on disk, a Dockerfile silently missing a COPY instruction, a PowerShell pipe corrupting an authentication token, and — the most interesting one — Terraform and a CI/CD pipeline fighting over who controls the same resource.

None of these are exotic problems. They're the ordinary friction of building something real. This article walks through the architecture we ended up with, and — more usefully — the specific failures we hit and how each one was actually diagnosed and fixed, not just patched over.

What You Will Build

By the end of this project, you'll have two fully isolated environments (dev and prod), each with:

A custom VPC with public/private subnets across two Availability Zones, a NAT Gateway, and proper routing
A private ECR repository with image scanning and a lifecycle policy
IAM roles scoped to least privilege for both ECS task execution and the running application
An Application Load Balancer with an HTTPS listener (redirecting from HTTP) in front of an ECS Fargate service
A CodePipeline + CodeBuild pipeline that builds, tags, and deploys a new container image automatically on every push to the branch
CloudWatch alarms for 5xx errors, unhealthy targets, low running task count, and high CPU — all notifying an SNS topic by email
Terraform state stored remotely in S3 with DynamoDB locking, instead of local state files
Branch protection on the production branch, requiring a pull request before anything reaches prod

Solution Architecture

At a high level:

Each environment (dev, prod) is a completely separate Terraform root module with its own state file, its own VPC, its own pipeline, and its own alarms, sharing only the underlying Terraform modules (the reusable vpc, ecr, iam, alb, ecs, codepipeline, sns, and cloudwatch modules live once, and both environments call them with different variables).

Prerequisites

RequirementStatusAWS Account with admin-equivalent IAM permissions✅Terraform installed (we used 1.12.x)✅Docker Desktop✅A GitHub repository✅Basic familiarity with VPC, IAM, and containers✅

Building the Foundation: VPC, ECR, IAM

We started with the standard networking layer: a VPC, public and private subnets across two AZs, an Internet Gateway, a NAT Gateway for the private subnets, and route tables. Nothing unusual here — except that this is exactly where our first real problem showed up (more on that in Lessons Learned).

ECR came next: a single private repository with IMMUTABLE tag mutability (once you push a tag, it can never be overwritten — this matters a lot once CI/CD enters the picture, since it forces every build to have a genuinely unique tag), image scanning on push, and a lifecycle policy expiring untagged images after 14 days and capping tagged images at 10.

IAM followed the standard ECS pattern of two roles:

A task execution role, used by ECS itself to pull the image from ECR and write logs to CloudWatch (this one just attaches the AWS-managed AmazonECSTaskExecutionRolePolicy)
A task role, used by the application code at runtime, scoped narrowly to only the DynamoDB, SQS, SNS, S3, and CloudWatch Logs resources matching our naming prefix — not *

Fronting the App: ALB and ECS Fargate

The ALB module creates a security group (initially just port 80), the load balancer itself across the public subnets, a target group pointed at port 3000 (our app's port), and an HTTP listener. The ECS module creates a Fargate cluster with Container Insights enabled, a CloudWatch log group, a security group that only accepts traffic from the ALB's security group (not the open internet), a task definition, and the service itself.

Getting the App Actually Running (and What Went Wrong)

We built a simple Express app — an intentionally over-designed storefront landing page for "GlobalMart," styled around an airport-departures-board concept (navy and saffron palette, a flip-board ticker showing product "arrivals" from different countries) rather than a generic template. Dockerized it, pushed it to ECR, pointed the ECS task definition at the image, and applied.

The service came up with zero healthy tasks. ALB health checks were failing with a 404.

The cause: the Dockerfile copied server.js into the image but never copied the public/ folder where index.html actually lived. The container was running, Express was serving requests, and every single one of them correctly 404'd — because there was nothing to serve. Adding one line (COPY public ./public), rebuilding, and pushing a new tag fixed it immediately. This is the kind of bug that's invisible in terraform plan and invisible in docker build output — it only shows up once real traffic hits the container, which is exactly why health checks and monitoring matter.

Wiring Up CI/CD with CodePipeline

Manually running docker build && docker tag && docker push and then hand-editing a Terraform variable every time you want to deploy is fine for a demo, but it's not how real teams ship code. We built a codepipeline Terraform module with:

An S3 bucket for pipeline artifacts (versioned, encrypted, public access blocked)
A CodeStar Connection to GitHub (this is the one piece Terraform can't fully automate — AWS requires a one-time manual OAuth authorization click in the console after the connection resource is created)
A CodeBuild project running in privileged mode (required for Docker-in-Docker), with a buildspec.yml that logs into ECR, builds the image, tags it with the first 8 characters of the git commit SHA (guaranteeing a unique tag every time, which plays correctly with our IMMUTABLE repository), pushes it, and writes an imagedefinitions.json file
A three-stage CodePipeline: Source (GitHub via the connection) → Build (CodeBuild) → Deploy (native ECS deploy action, which reads imagedefinitions.json and updates the service)

Once the GitHub connection was authorized in the console, a push to the branch triggered the full pipeline automatically, and it succeeded end-to-end.

Adding HTTPS to the Load Balancer

Since we didn't own a domain (ACM certificates require domain validation via DNS or email, which isn't possible against a bare AWS-generated ALB hostname), we generated a self-signed certificate with OpenSSL and imported it directly into ACM — skipping domain validation entirely. This isn't a certificate a browser will trust, but it's genuinely useful for demonstrating correct TLS termination architecture: an HTTP listener on port 80 that redirects (301) to HTTPS on port 443, and an HTTPS listener that terminates TLS at the load balancer and forwards plain HTTP to the target group.

Getting there involved one Windows-specific detour worth mentioning: aws ecr get-login-password | docker login ... piped through native PowerShell silently corrupted the long base64 authentication token and failed with a 400 Bad Request. Routing the exact same command through cmd /c "..." instead fixed it immediately — a reminder that shell pipeline behavior isn't universal, even for commands copied directly from AWS's own documentation.

The Terraform vs. Pipeline Conflict (and How to Fix It)

This was the most instructive bug in the whole project.

Once CodePipeline started deploying new task definition revisions on every push (bypassing Terraform entirely — it calls the ECS API directly), Terraform's local state fell out of sync with reality. The next time we ran terraform plan for an unrelated change, Terraform saw that the "current" task definition according to its state was several revisions behind what was actually running, and proposed to forcibly roll the service back — silently undoing every deploy the pipeline had made since the last terraform apply.

This is a structural conflict, not a bug in either tool: Terraform assumes it's the sole owner of everything in its state, and a CI/CD pipeline managing the same resource violates that assumption. The fix is to explicitly tell Terraform to stop caring about that one attribute:

hclresource "aws_ecs_service" "app" {
# ...

lifecycle {
ignore_changes = [task_definition]
}
}

With this in place, Terraform continues managing the infrastructure around the service (cluster, networking, security groups) but defers entirely to the pipeline for which image is currently deployed. This is a pattern worth knowing before you combine Terraform-managed ECS with any CI/CD tool that also touches the task definition — without it, your next unrelated terraform apply can quietly roll back production.

Monitoring and Alerting

Four CloudWatch alarms per environment, all notifying a single SNS topic subscribed by email:

ALB 5xx errors — more than 10 target-side 5xx responses in a 5-minute window
Unhealthy target count — any target behind the ALB reporting unhealthy
Low running task count — fewer running ECS tasks than desired (using the ECS/ContainerInsights namespace, which requires Container Insights to be enabled on the cluster)
High CPU — ECS service CPU utilization above 80% sustained for 15 minutes

The SNS email subscription needs a manual confirmation click after terraform apply — Terraform can create the subscription, but AWS requires the recipient to actively confirm it, the same way it requires manual authorization for the GitHub connection.

Governance: Branch Protection and Remote State

Two changes made this feel less like a personal sandbox and more like a real team setup:

Branch protection on prod — configured directly in GitHub (Settings → Branches), requiring a pull request before anything merges into the branch that triggers the production pipeline. No more direct pushes to prod, even by an admin, without a deliberate PR.

Migrating Terraform state from local files to S3 with DynamoDB locking — local terraform.tfstate files are fine for solo experimentation but are a real liability the moment more than one person (or more than one machine) might run terraform apply against the same infrastructure: concurrent applies can corrupt state, and a deleted or lost local file means losing track of everything Terraform manages. Remote state in S3, combined with a DynamoDB table for locking (so two applies can't run simultaneously), is the standard production pattern.

Lessons Learned

Empty files look identical to complete files in an editor. At one point, several .tf files inside a module directory were showing valid, complete content in the code editor — but were actually zero bytes on disk. The content only existed in an unsaved editor buffer. terraform validate correctly reported the module's variables as undeclared, which looked like a logic error but was actually a save error. Always verify file sizes on disk (Get-ChildItem, or ls -la) when a .tf file's content doesn't match what Terraform is complaining about.

Duplicate resource declarations fail loudly, which is a feature. Copy-pasting a resource block into the wrong file (in our case, aws_subnet blocks ended up duplicated across two files) produces an immediate, unambiguous "Duplicate resource configuration" error — this is one of the few Terraform failure modes that's actually easy to diagnose, because the error names the exact resource and the exact second location.

PowerShell's pipeline isn't always transparent. Long strings piped between commands (the ECR auth token, in our case) can be silently mangled by PowerShell in ways that produce a downstream error with no obvious connection to the actual cause. When a command that should obviously work fails with a generic error, try routing it through cmd /c before assuming the problem is on the remote end.

A working terraform plan and a working application are two different claims. Every piece of infrastructure can apply cleanly and still serve a 404 for every request, because Terraform has no visibility into what's actually inside your container image. Health checks, and genuinely hitting the live endpoint after every deploy, are not optional steps.

Recycle Bin is a legitimate disaster recovery tool. An accidental folder deletion mid-session turned out to be completely recoverable because Windows' standard delete (rather than a hard Remove-Item -Force) sends files to the Recycle Bin first. Combined with git for the source code and the fact that AWS resources don't disappear just because local files do, this turned a moment of genuine panic into a five-minute non-event. It's still worth treating your Terraform state file with real care — it was the one thing that mattered here and wasn't backed up anywhere except that Recycle Bin.

Branches don't protect you from typing on the wrong one. A git checkout prod earlier in a session, followed by continuing to work without checking git branch again, meant several commits meant for main landed on prod instead. Git doesn't warn you about this, git commit and git push work identically regardless of which branch is checked out. git status at the top of every output tells you the current branch; it's worth reading that line every time, not just the "nothing to commit" part.

Production Considerations

This project is intentionally a solid foundation, not a finished production system. Before treating something like this as production-ready, we'd still want:

A real domain name and a properly validated ACM certificate (our self-signed cert demonstrates the architecture but isn't trusted by any browser)
Auto-scaling policies on the ECS service, instead of a hardcoded desired count
The DynamoDB, SQS, SNS, and S3 resources the IAM task role is already scoped for, but which don't exist yet
A WAF in front of the ALB for basic protection against common web exploits
Centralizing logs and metrics further with CloudWatch Dashboards, rather than checking alarms individually

Conclusion

The infrastructure in this project (a VPC, a container registry, a load balancer, a Fargate service, a CI/CD pipeline, and monitoring) isn't architecturally unusual. What made it worth writing up wasn't the happy path; it was everything that went wrong along the way, and the fact that every failure had a specific, diagnosable cause rather than being a mystery to work around.

That's the actual shape of infrastructure work: not a sequence of commands that always succeed, but a sequence of hypotheses, verifications, and fixes. The terraform plan that shows "No changes" after a scare is more valuable than the one that shows a clean create on the first try, because it proves the system actually recovered correctly rather than just looking fine.

If you're building something similar, the advice that mattered most in this project wasn't about any specific AWS service, it was to verify state directly rather than assume it (Get-ChildItem before trusting a file exists, terraform plan before trusting an apply succeeded, curl the actual endpoint before trusting a health check passed), and to fix the root cause of a failure rather than the symptom in front of you.

Building a Highly Available Multi-Region Web Application on AWS with Global Accelerator

Adesanya Adedeji Stephen — Tue, 14 Jul 2026 20:14:25 +0000

Learn how to build a fault-tolerant web application that automatically redirects users to a healthy AWS Region during regional outages using Amazon EC2, Application Load Balancer, Route 53 Health Checks, and AWS Global Accelerator.

Author: Adedeji Adesanya

Difficulty: Intermediate

Estimated Time: 2–3 Hours

AWS Services Used:

Amazon VPC
Amazon EC2
Application Load Balancer (ALB)
Amazon Route 53 Health Checks
AWS Global Accelerator

Introduction
What You Will Build
Solution Architecture
Prerequisites
Understanding the Architecture
Project Overview

Introduction

Imagine you've built an e-commerce website hosted in the us-east-1 AWS Region.

Everything works perfectly—until an unexpected regional outage occurs.

Suddenly, your application becomes unreachable. Customers can't browse products, place orders, or make payments. Every minute of downtime affects user experience and can result in lost revenue.

This is why modern cloud applications are designed with high availability and disaster recovery in mind.

Instead of relying on a single Region, organizations deploy applications across multiple AWS Regions so that if one becomes unavailable, traffic can automatically be redirected to another healthy Region.

In this tutorial, you'll build exactly that.

By the end of this guide, you'll have a multi-region web application capable of automatically failing over between two AWS Regions using AWS Global Accelerator and Amazon Route 53 Health Checks.

Rather than simply deploying infrastructure, you'll learn how multiple AWS services work together to create a resilient architecture suitable for production environments.

What You Will Build

By the end of this tutorial, you'll have an architecture that looks like this:

At a high level, the architecture consists of:

Two independent AWS Regions
Separate networking infrastructure in each Region
One EC2 instance running the application in each Region
One Application Load Balancer per Region
Route 53 Health Checks for continuous monitoring
AWS Global Accelerator providing a single global entry point
Automatic traffic failover during outages

The end result is a web application that continues serving users even when an entire AWS Region becomes unavailable.

Why AWS Global Accelerator?

Many engineers assume Amazon Route 53 is the only solution for routing users globally.

While Route 53 is an excellent DNS service, DNS-based failover has one important limitation: DNS caching.

Because DNS records are cached by clients and Internet Service Providers, changes may not take effect immediately.

AWS Global Accelerator solves this differently.

Instead of relying on DNS propagation, it provides:

Two static Anycast IP addresses
Intelligent traffic routing through the AWS Global Network
Faster failover
Improved latency
Better user experience

For applications requiring high availability and low latency, Global Accelerator is often the preferred choice.

Prerequisites

Before beginning this project, ensure you have the following:

Requirement	Status
AWS Account	✅
Basic understanding of EC2	✅
Basic understanding of Amazon VPC	✅
Two AWS Regions enabled	✅
IAM User with AdministratorAccess (or equivalent permissions)	✅

Services Used

Throughout this project, you'll work with several AWS services.

Amazon VPC

Amazon VPC provides an isolated virtual network where your AWS resources will run. Each Region will have its own dedicated VPC to ensure complete isolation.

Amazon EC2

Amazon EC2 hosts the web application in both Regions.

Each EC2 instance runs the same application, allowing either Region to serve user requests independently.

Application Load Balancer

The Application Load Balancer distributes incoming traffic to healthy EC2 instances.

It also serves as the endpoint monitored by Route 53 Health Checks.

Amazon Route 53 Health Checks

Health Checks continuously monitor the application's availability.

If a Region becomes unhealthy, Route 53 reports the failure, allowing Global Accelerator to redirect traffic automatically.

AWS Global Accelerator

Global Accelerator acts as the application's single global entry point.

Instead of users connecting directly to a specific Region, they connect to Global Accelerator, which routes traffic to the nearest healthy endpoint.

This makes regional failover seamless and transparent to users.

Building the Network Foundation

Before we can deploy our application, we need to build the network that will host it.

In AWS, every resource lives inside a Virtual Private Cloud (VPC). Since we're building a multi-region architecture, we'll create a separate VPC in each AWS Region.

This ensures that each Region operates independently, eliminating a single point of failure.

Step 1: Create the Primary VPC (us-east-1)

Switch your AWS Region to US East (N. Virginia) – us-east-1.

From the AWS Management Console, navigate to:

VPC
→ Your VPCs
→ Create VPC

Choose VPC only and configure it as shown below.

Setting Value
Name primary-vpc
IPv4 CIDR Block 10.0.0.0/16
IPv6 None
Tenancy Default

After entering the values, click Create VPC.

Once the VPC has been created successfully, you should see it listed in the VPC dashboard.

Why a /16 CIDR Block?

You might be wondering why we chose 10.0.0.0/16.

A /16 network provides 65,536 IP addresses, giving plenty of room for future expansion.

Although our lab only uses a few EC2 instances, production environments often require additional subnets for:

Application servers
Databases
NAT Gateways
Private services
Kubernetes clusters

Planning your IP address space early helps avoid costly network redesigns later.

💡 Pro Tip: Always leave room for growth when designing cloud networks.

Step 2: Create the Secondary VPC (us-west-2)

Next, switch your AWS Region to US West (Oregon) – us-west-2.

Repeat the same process.

Navigate to:
VPC
→ Your VPCs
→ Create VPC

Use the following configuration.

Setting Value
Name secondary-vpc
IPv4 CIDR Block 10.1.0.0/16
IPv6 None
Tenancy Default

Notice that we used a different CIDR block (10.1.0.0/16) instead of 10.0.0.0/16.

This is intentional.

Each VPC must have a unique address range to avoid overlapping IP addresses. Using distinct CIDR blocks makes future networking tasks—such as VPC Peering or AWS Transit Gateway—much simpler.

Step 3: Create Public Subnets

With both VPCs in place, the next step is to create public subnets.

A subnet divides a VPC into smaller, manageable networks.

Our EC2 instances and Application Load Balancers will reside in these public subnets so they can receive traffic from the internet.

Primary Region (us-east-1)

Navigate to:

VPC
→ Subnets
→ Create subnet

Select primary-vpc and create the following subnet.

Setting Value
Name primary-public-subnet
Availability Zone us-east-1a
CIDR Block 10.0.1.0/24

Click Create subnet

Secondary Region (us-west-2)

Repeat the process in us-west-2.

Setting Value
Name secondary-public-subnet
Availability Zone us-west-2a
CIDR Block 10.1.1.0/24

Why Use a /24 Subnet?

Each /24 subnet provides 256 IP addresses.

This is more than enough for our project while leaving room for additional EC2 instances, load balancers, or other resources in the future.

Separating the VPC into smaller subnets also improves network organization and scalability.

Step 4: Attach Internet Gateways

By default, a VPC has no internet connectivity.

To allow resources inside the VPC to communicate with the internet, we need an Internet Gateway (IGW).

We'll create one for each VPC.

Navigate to:

VPC
→ Internet Gateways
→ Create Internet Gateway

For the primary Region:

Setting Value
Name primary-igw

After creating it, choose Actions → Attach to VPC, then select primary-vpc.

*Repeat the same process in us-west-2.
*
Setting Value
Name secondary-igw

What's Next?

we'll configure Route Tables, launch our EC2 instances, install the web application, and prepare the environment for load balancing.

Deploying the Web Application and Configuring the Load Balancer

At this stage, our networking foundation is complete.

Each AWS Region now has its own VPC, public subnet, and Internet Gateway. The next step is to deploy the application that users will access.

To keep the architecture consistent, we'll deploy an identical web application in both Regions. Each application will run on an Amazon EC2 instance and sit behind an Application Load Balancer (ALB).

Using identical deployments ensures that either Region can serve traffic if the other becomes unavailable.

Step 5: Create a Security Group for the Web Server

Before launching the EC2 instances, we'll create a security group that controls inbound traffic.

In the AWS Management Console, navigate to:

VPC
→ Security Groups
→ Create Security Group

Use the following configuration for the Primary Region (us-east-1).

Setting Value
Name web-server-sg
Description Security group for EC2 web servers
VPC primary-vpc

Inbound Rules
Type Protocol Port Source
HTTP TCP 80 0.0.0.0/0
SSH TCP 22 Your Public IP

Why restrict SSH?

Allowing SSH from 0.0.0.0/0 exposes your server to the entire internet. Restricting it to your public IP significantly improves security while still allowing you to administer the instance.

Repeat the same process in us-west-2, ensuring the security group is associated with secondary-vpc.

Step 6: Launch the EC2 Instances

Now we'll deploy the application servers.

Navigate to:

EC2
→ Instances
→ Launch Instance
Primary Region (us-east-1)

Use the following configuration.

Setting Value
Name primary-web-server
AMI Amazon Linux 2023
Instance Type t2.micro
Key Pair Select your existing key pair
Network primary-vpc
Subnet primary-public-subnet
Auto-assign Public IP Enabled
Security Group web-server-sg

Repeat the same steps in us-west-2, changing only the instance name to:
secondary-web-server

Step 7: Install Apache and Deploy a Simple Web Page

Once the EC2 instances are running, connect to each instance using SSH.

The output should display:

Active: active (running)

Step 8: Create the Application Homepage

Instead of using Apache's default page, we'll create a simple HTML page so it's easy to identify which Region is serving requests.

Step 9: Verify the Web Servers

Open a web browser and enter the public IP address of each EC2 instance.

You should see the custom page indicating which Region is serving the request.

This simple verification confirms that Apache is running correctly and the application is reachable before introducing the load balancer.

⚠️ Don't skip this step.

Verifying the application directly helps isolate problems early. If the page doesn't load now, it certainly won't work behind the load balancer later.

Step 10: Create an Application Load Balancer

Now we'll place each EC2 instance behind an Application Load Balancer.

Navigate to:

EC2
→ Load Balancers
→ Create Load Balancer
→ Application Load Balancer

For the Primary Region, use the following configuration.

Setting Value
Name primary-alb
Scheme Internet-facing
IP Address Type IPv4
VPC primary-vpc
Subnets Select the public subnet(s)

Listener
Protocol Port
HTTP 80

Step 11: Create a Target Group

Before the ALB can forward traffic, it needs a target group.

Configure it as follows.

Setting Value
Target Type Instances
Name primary-target-group
Protocol HTTP
Port 80
Health Check Path /

*Repeat the process in us-west-2, creating:
*
secondary-alb
secondary-target-group

Step 12: Verify Load Balancer Health

Once the Application Load Balancer has been created, wait a few minutes while it performs health checks against the registered EC2 instance.

When everything is configured correctly, the target status should change to:

Healthy

If the target remains Unhealthy, check the following:

Is Apache running?
Does the security group allow inbound HTTP (port 80)?
Is the health check path set to /?
Is the EC2 instance registered with the correct target group?

Configuring Amazon Route 53 Health Checks

So far, we've built two identical application environments:

A Primary Region in us-east-1
A Secondary Region in us-west-2

Both applications are running successfully behind their respective Application Load Balancers.

However, our architecture still has one major limitation.

If the primary application goes offline, AWS has no way of knowing that it should stop sending traffic to it.

This is where Amazon Route 53 Health Checks come into play.

They continuously monitor the health of your application's endpoints. If an endpoint becomes unhealthy, other AWS services—such as Global Accelerator—can use that information to automatically redirect traffic to a healthy endpoint.

How Route 53 Health Checks Work

At regular intervals, Route 53 sends requests to your application's endpoint.

If the application responds successfully, the endpoint is considered healthy.

If it fails to respond after a specified number of consecutive checks, Route 53 marks it as unhealthy.

In our project, we'll configure a health check for each Region's Application Load Balancer.

Step 13: Copy the Load Balancer DNS Names

Before creating the health checks, we need the DNS name of each Application Load Balancer.

Navigate to:

EC2
→ Load Balancers

Select primary-alb.

Copy the value under DNS name.

It should look similar to:

primary-alb-123456789.us-east-1.elb.amazonaws.com

Repeat the process for secondary-alb.

Step 14: Create the Primary Health Check

Navigate to:

Route 53
→ Health Checks
→ Create Health Check

Configure it using the following settings.

Setting Value
Name Primary-ALB-Health-Check
What to monitor Endpoint
Specify endpoint by Domain Name
Domain Name Primary ALB DNS Name
Protocol HTTP
Port 80
Path /
Request Interval 30 Seconds
Failure Threshold 3

Leave all other settings at their default values.

Click Create Health Check.

Step 15: Create the Secondary Health Check

Repeat the same process for the Application Load Balancer in us-west-2.

Use the following configuration.

Setting Value
Name Secondary-ALB-Health-Check
Domain Name Secondary ALB DNS
Protocol HTTP
Port 80
Path /
Request Interval 30 Seconds
Failure Threshold 3

Step 16: Verify the Health Checks

Once both health checks have been created, wait a few minutes.

Route 53 needs time to send its first requests and determine the health of each endpoint.

Initially, the status may display:

Checking...

After a short period, both health checks should report:

Healthy

Understanding the Health Check Results

When Route 53 marks an endpoint as Healthy, it means:

The endpoint is reachable.
The application is responding successfully.
The configured path (/) returned a successful HTTP response.

If an endpoint becomes Unhealthy, Route 53 will stop considering it available.

This health information becomes especially important in the next part of the tutorial, where AWS Global Accelerator uses it to decide where incoming traffic should be routed.

Troubleshooting Health Checks

If your health check doesn't become healthy after several minutes, don't panic. The issue is usually one of the following:

Problem Solution
Apache is not running Start the Apache service using sudo systemctl start httpd

Port 80 is blocked Verify the EC2 security group allows inbound HTTP traffic

Incorrect health check path Ensure the path is /

Wrong DNS name Confirm you're using the ALB DNS name, not the EC2 public IP

Target group is unhealthy Verify the EC2 instance is healthy behind the ALB

Taking a few minutes to troubleshoot now will save time later when configuring Global Accelerator.

What We Accomplished

At this point, we've successfully:

Created health checks for both Application Load Balancers.
Verified that both application endpoints are healthy.
Enabled continuous monitoring of our application.
Prepared the architecture for automatic failover.

In the next part, we'll configure AWS Global Accelerator and connect it to both Application Load Balancers. This will provide users with a single global entry point and enable automatic traffic failover whenever a Region becomes unavailable.

Configuring AWS Global Accelerator

Up to this point, we've built two independent application environments:

Primary Region (us-east-1)
Secondary Region (us-west-2)

Each environment has its own:

Amazon VPC
EC2 Instance
Application Load Balancer
Route 53 Health Check

The final piece is giving users one consistent entry point regardless of which Region is serving the application.

This is where AWS Global Accelerator comes in.

Unlike DNS-based routing, Global Accelerator provides two static Anycast IP addresses and uses the AWS global network to direct users to the best available endpoint. If a Region becomes unhealthy, traffic is automatically routed to another healthy Region with minimal disruption.

In this section, we'll configure AWS Global Accelerator and connect it to the Application Load Balancers we created earlier.

How AWS Global Accelerator Works

When a user accesses your application, they don't connect directly to an EC2 instance or an Application Load Balancer.

Instead, they connect to the Global Accelerator endpoint.

Global Accelerator evaluates the health of each configured endpoint and routes traffic to the optimal Region.

If an endpoint becomes unavailable, traffic is redirected automatically without requiring DNS updates.

Step 17: Create a Global Accelerator

From the AWS Management Console, navigate to:

AWS Global Accelerator
→ Create Accelerator

Configure the accelerator using the following settings.

Setting Value
Name Multi-Region-Web-App
IP Address Type IPv4
Enabled Yes

Leave the remaining settings at their default values.

Click Create Accelerator.

After a few moments, AWS will provision your accelerator and assign two static Anycast IP addresses.

These IP addresses remain the same even if the backend infrastructure changes.

Step 18: Configure the Listener

Once the accelerator has been created, the next step is to configure a listener.

A listener defines which ports Global Accelerator accepts from clients.

Use the following configuration.

Setting Value
Protocol TCP
Port 80

Click Next.

Step 19: Create an Endpoint Group

Global Accelerator organizes endpoints into Endpoint Groups.

Each endpoint group represents an AWS Region.

We'll create one endpoint group for each Region.

Primary Endpoint Group
Setting Value
Region us-east-1
Traffic Dial 100%
Health Check Protocol HTTP
Health Check Port 80
Health Check Path /

Repeat the process for the secondary Region.

Secondary Endpoint Group
Setting Value
Region us-west-2
Traffic Dial 100%
Health Check Protocol HTTP
Health Check Port 80
Health Check Path /

Step 20: Add the Application Load Balancers

Within each endpoint group, add the corresponding Application Load Balancer.

Endpoint Group 1
Endpoint Weight
primary-alb 128
Endpoint Group 2
Endpoint Weight
secondary-alb 128

The endpoint weight determines how traffic is distributed among endpoints within the same group. Since each group contains only one ALB, the default weight is appropriate.

Step 21: Deploy the Accelerator

Review your configuration and click Create Accelerator.

Provisioning typically takes a few minutes.

Once complete, you'll receive:

A Global Accelerator DNS name
Two static Anycast IP addresses

Your application can now be accessed through a single global endpoint instead of directly through either Application Load Balancer.

Verifying the Configuration

Open a browser and navigate to the Global Accelerator DNS name.

You should see your application load successfully.

Depending on your location and routing policies, the request may be served from either the primary or secondary Region.

At this point, everything appears normal—but the real value of this architecture becomes evident when a Region fails.

We'll test that in the next section.

Why Use AWS Global Accelerator Instead of DNS Failover?

This is a common question among engineers.

Traditional DNS failover relies on DNS record updates and caching.

Because DNS responses are cached by clients and Internet Service Providers, failover may not happen immediately.

AWS Global Accelerator takes a different approach.

Instead of relying on DNS changes, it uses the AWS global network and continuously monitors endpoint health.

This allows traffic to be redirected much faster, improving both availability and user experience.

For applications where uptime is critical, Global Accelerator is often a better choice than DNS-only failover.

What We Accomplished

At this point, we've successfully:

Created an AWS Global Accelerator
Configured a listener
Added endpoint groups for two AWS Regions
Registered both Application Load Balancers
Verified that the accelerator is healthy and ready to serve traffic The architecture is now fully configured.

In the next section, we'll intentionally simulate a regional outage to validate the failover process and observe how Global Accelerator automatically redirects traffic to the healthy Region.

Testing Automatic Failover

A highly available architecture isn't complete until it's been tested.

Up to this point, we've deployed our application across two AWS Regions, configured health checks, and set up AWS Global Accelerator to route traffic intelligently.

Now it's time to answer the most important question:

What happens if the primary Region becomes unavailable?

Instead of assuming the architecture will work as expected, we'll intentionally simulate a failure and observe how AWS responds.

Testing failover is a critical part of any disaster recovery strategy. It provides confidence that your application can continue serving users even when unexpected failures occur.

Step 22: Verify the Application Before Testing

Before simulating a failure, open the AWS Global Accelerator DNS name in your browser.

The application should load successfully.

Depending on your location and Global Accelerator's routing decisions, you should see the page served from your primary Region.

This confirms that the application is healthy and ready for testing.

Step 23: Simulate a Regional Failure

There are several ways to simulate a failure.

For this lab, we'll stop the Apache web server running on the EC2 instance in the Primary Region.

Connect to the primary EC2 instance using SSH.

At this point, the Application Load Balancer in the primary Region will no longer receive successful responses from the web server.

After several failed health checks, the target will be marked as Unhealthy.

Step 24: Observe Automatic Failover

Wait a few moments while Route 53 Health Checks detect the failure.

Once the endpoint is marked as unhealthy, AWS Global Accelerator automatically stops routing traffic to the primary Region.

Refresh your browser using the same Global Accelerator DNS name.

You should now see the application being served from the Secondary Region (us-west-2).

Notice that the URL remains exactly the same.

No DNS changes are required.

No manual intervention is needed.

From the user's perspective, the application remains available despite the failure of one Region.

This seamless transition is one of the key advantages of AWS Global Accelerator.

Step 25: Restore the Primary Region

Now let's bring the primary application back online.

Reconnect to the EC2 instance in us-east-1 and start the Apache service.

After a few successful health checks, Route 53 will mark the endpoint as healthy again.

Global Accelerator will automatically resume routing traffic to the primary Region based on its endpoint health.

What We Observed

This test demonstrates several important concepts about highly available architectures:

Application health is continuously monitored.
Failed endpoints are automatically removed from service.
Traffic is redirected without requiring DNS changes.
Users continue accessing the application through the same global endpoint.
Once the failed Region recovers, it is automatically added back into service.

These behaviors are essential for building resilient cloud applications that can withstand infrastructure failures with minimal impact on end users.

Key Takeaway

One of the biggest lessons from this project is that high availability isn't just about deploying resources across multiple Regions.

It's about designing an architecture that can detect failures, respond automatically, and recover without requiring manual intervention.

That's exactly what we've accomplished by combining Route 53 Health Checks with AWS Global Accelerator.

Cleaning Up Your AWS Resources

Congratulations! You've successfully built and tested a highly available multi-region web application on AWS.

If you created these resources in your own AWS account, it's important to remove them once you're done with the lab. Leaving resources running can result in unexpected charges, especially for services such as AWS Global Accelerator and Application Load Balancers.

In this section, we'll clean up the environment in a logical order.

Step 26: Delete the AWS Global Accelerator

Navigate to:

AWS Global Accelerator
→ Accelerators

Select the accelerator you created and choose Delete.

Deleting the accelerator first ensures that it no longer references your Application Load Balancers.

Step 27: Delete the Route 53 Health Checks

Next, navigate to:

Route 53
→ Health Checks

Delete both health checks:

Primary-ALB-Health-Check
Secondary-ALB-Health-Check

Since these health checks are no longer needed, removing them prevents unnecessary monitoring costs.

Step 28: Delete the Application Load Balancers

Navigate to:

EC2
→ Load Balancers

Delete:

primary-alb
secondary-alb

Keep in mind that AWS may take a few minutes to completely remove each load balancer.

Step 29: Delete the Target Groups

After the load balancers have been deleted, navigate to:

EC2
→ Target Groups

Delete:

primary-target-group
secondary-target-group

Target groups cannot be deleted while they are still associated with an active load balancer, which is why this step comes after deleting the ALBs.

Step 30: Terminate the EC2 Instances

Navigate to:

EC2
→ Instances

Select both EC2 instances and choose:

Instance State
→ Terminate Instance

Wait until their status changes to Terminated before proceeding.

Step 31: Delete the Security Groups

Navigate to:

VPC
→ Security Groups

Delete the custom security groups you created for this project.

If AWS prevents you from deleting a security group, double-check that no resources are still attached to it.

Step 32: Delete the Internet Gateways

Navigate to:

VPC
→ Internet Gateways

Before deleting each Internet Gateway, you'll need to detach it from its VPC.

Select the Internet Gateway, then choose:

Actions
→ Detach from VPC

Once detached, delete the Internet Gateway.

Repeat this process for both Regions.

Step 33: Delete the Subnets

Navigate to:

VPC
→ Subnets

Delete the public subnet in each Region.

AWS will prevent subnet deletion if resources still exist within it, so ensure all EC2 instances have been terminated first.

Step 34: Delete the VPCs

Finally, navigate to:

VPC
→ Your VPCs

Delete:

primary-vpc
secondary-vpc

At this point, all resources created during this project should have been removed from your AWS account.

Cleanup Checklist

Use this checklist to verify that you've removed everything:

✅ AWS Global Accelerator deleted
✅ Route 53 Health Checks deleted
✅ Application Load Balancers deleted
✅ Target Groups deleted
✅ EC2 Instances terminated
✅ Security Groups deleted
✅ Internet Gateways detached and deleted
✅ Public Subnets deleted
✅ VPCs deleted

Completing this checklist helps ensure you don't incur unnecessary AWS charges after finishing the lab.

Conclusion and Lessons Learned

Building cloud infrastructure is one thing.

Building infrastructure that can withstand failure is another.

_In this tutorial, we went beyond deploying a simple web application. We designed and implemented a highly available architecture capable of automatically redirecting traffic when an AWS Region becomes unavailable.
_
Along the way, we explored how several AWS services work together to deliver a resilient solution:

Amazon VPC provided isolated networking environments.
Amazon EC2 hosted the application in each Region.
Application Load Balancers distributed traffic and performed health checks on application instances.
Amazon Route 53 Health Checks continuously monitored endpoint availability.
AWS Global Accelerator provided a single global entry point and intelligent traffic routing.

The most valuable part of this project wasn't creating the infrastructure—it was validating that the architecture behaved as expected during a simulated regional outage. Watching traffic automatically shift to the healthy Region reinforced an important principle of cloud engineering:

A resilient architecture isn't defined by the resources you deploy. It's defined by how those resources respond when failures occur.

This project also highlighted the importance of testing disaster recovery plans. Designing for failure is only part of the process; regularly validating your recovery strategy is what builds confidence in production systems.

Production Considerations

Although this project demonstrates the core concepts of high availability and multi-region failover, there are several ways it could be enhanced for a production environment:

Replace standalone EC2 instances with Auto Scaling Groups to improve scalability and resilience.
Use Amazon RDS Multi-AZ or Amazon Aurora Global Database for highly available databases.
Secure the application with HTTPS using AWS Certificate Manager (ACM).
Protect the application using AWS WAF and AWS Shield.
Improve global performance with Amazon CloudFront.
Provision the infrastructure using Terraform or AWS CloudFormation instead of manually creating resources.
Centralize monitoring and alerting with Amazon CloudWatch and AWS CloudTrail.

These enhancements reflect how similar architectures are commonly implemented in production environments.

Final Thoughts

This project deepened my understanding of high availability, disaster recovery, and global traffic management on AWS. More importantly, it reinforced the value of designing systems with failure in mind rather than treating resilience as an afterthought.

I hope this guide has helped you understand not only how to build a multi-region architecture, but also why each component plays a critical role in delivering a reliable application.

If you followed along, I'd love to hear about your experience. Did you encounter any challenges? Were there any improvements or alternative approaches you explored? Feel free to share your thoughts in the comments, I'd be interested to learn how you approached the project.