How I Built a Self-Healing, Multi-AZ Infrastructure on AWS — And What Broke Along the Way

Knirl Amboy — Thu, 17 Sep 2026 05:29:40 +0000

The Problem I Set Out to Solve

AWS can projects look the same: spin up one EC2 instance, install a web server, call it done. The moment that instance crashes or traffic spikes, the whole thing falls over.

I wanted to build something that actually answers the question when it comes to "high availability": what happens when something fails, and does the system recover without a human stepping in

So I set out to build a web architecture with no single point of failure — spanning two Availability Zones, load-balanced, auto-scaling, with a self-healing database — and then, critically, to actually break it on purpose to prove it recovers, instead of just trusting the console said everything was configured correctly.

The Architecture, Briefly

A VPC spanning 2 Availability Zones, with public subnets for the load balancer and private subnets for everything else
An Application Load Balancer distributing traffic to an Auto Scaling Group of EC2 instances
RDS running MySQL in Multi-AZ mode, with a live standby ready to take over
CloudWatch monitoring the whole thing, with an alarm wired to actually notify me

That's the "what." The interesting part — the part that actually taught me something — is everything that didn't work on the first try.

The Debugging Journey

Incident 1: The ALB That Wouldn't Respond

After wiring up the VPC, security groups, target group, and load balancer, I hit the ALB's DNS name in my browser and got:

ERR_CONNECTION_TIMED_OUT

My first assumption was a security group misconfiguration. So I worked through it methodically:

Checked the ALB's security group — inbound HTTP 80 from 0.0.0.0/0 was correctly set
Checked subnet placement — the ALB was correctly deployed across both public subnets
Checked the ALB's scheme — confirmed "Internet-facing," not "Internal"
Checked the listener — HTTP:80 was correctly forwarding to my target group
Checked Network ACLs at the subnet level — default allow-all rules were untouched

Every single AWS-side configuration was correct. That's the moment it clicked that the problem probably wasn't in AWS at all — it was somewhere between my browser and AWS. I tried the same URL from my phone on cellular data, and then just paid closer attention to what was actually in my address bar.

The issue: I'd pasted the bare DNS name into Edge, and the browser had silently prepended https://. My ALB only had an HTTP listener configured — no HTTPS/443 listener existed at all. The browser was trying to reach a port that wasn't listening for that protocol, and quietly timing out instead of giving a clear "connection refused."

The fix: type the URL explicitly as http://your-alb-dns-name....

What this taught me: when every layer of your infrastructure checks out, stop re-checking the same layer and start questioning your assumptions about the client side. I spent close to 20 minutes re-verifying security groups I'd already verified, when the actual bug was in how I was typing a URL.

Incident 2: The Page That Loaded, But Told Me Nothing

Once the ALB was reachable, my page loaded — but it was supposed to display the serving instance's ID and Availability Zone, and both fields were blank.

My EC2 instances pull this data from AWS's internal instance metadata service, via a request like:

curl -s http://169.254.169.254/latest/meta-data/instance-id

This is a well-documented pattern, and I'd used it based on older references. Apache was running fine — the page loaded — so my instinct was that the problem was somewhere in my HTML generation logic. It wasn't. The metadata request itself was silently failing.

The root cause: Amazon Linux 2023 requires IMDSv2 by default — a token-based authentication flow for metadata requests. My script was using the older tokenless (IMDSv1-style) request pattern, which gets rejected without an explicit error visible in the page output — curl just returned empty, and my script happily wrote empty values into the HTML.

The fix required requesting a session token first, then passing it as a header on every subsequent metadata call:

TOKEN=$(curl -s -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 21600")
INSTANCE_ID=$(curl -s -H "X-aws-ec2-metadata-token: $TOKEN" http://169.254.169.254/latest/meta-data/instance-id)

But here's the part that actually tests whether you understand Auto Scaling Groups: fixing the script wasn't enough. My existing instances were already running the old script — updating a Launch Template only affects instances launched after the change, not ones already running. I had to publish the corrected script as a new Launch Template version, set it as default, and then trigger an Instance Refresh — which gradually and safely replaced my running instances with new ones, one at a time, keeping at least one healthy throughout the process. That refresh took about 12–15 minutes for two instances, which is worth knowing going in so you don't assume something's stuck.

What this taught me: "no visible error" doesn't mean "no failure." A script can execute successfully from the OS's perspective while still failing at the thing it was actually meant to do. And infrastructure-as-config (Launch Templates) has a subtlety that infrastructure-as-code makes more obvious later: changing the definition doesn't retroactively change what's already running.

Incident 3: Testing the Alarm I'd Configured

I didn't want to just configure a CloudWatch alarm and assume it worked — I wanted to actually watch it fire. So I deliberately broke connectivity between my load balancer and my instances by deleting the inbound rule on my EC2 security group that allowed traffic from the ALB.

Within about two minutes:

My target group flipped both instances to unhealthy
My CloudWatch alarm transitioned from OK to In alarm
I got an email notification via SNS
My Auto Scaling Group, seeing unhealthy instances via its ELB health check integration, began trying to replace them — and kept trying, since the replacements hit the exact same blocked security group rule

That last part was actually a good thing to witness: it's the ASG correctly attempting to self-heal, even in a scenario where it structurally couldn't succeed until I fixed the actual cause. I screenshotted the failure state, then reverted the security group rule and confirmed everything returned to healthy within a couple of minutes.

What this taught me: monitoring you haven't tested is monitoring you're just hoping works. The five minutes it took to break this on purpose gave me something concrete to talk about, instead of a screenshot of an alarm sitting in a green "OK" state that never proves it does anything.

Trade-Offs I Had to Reason Through

Cost vs. Genuine Fault Tolerance: Multi-AZ RDS

AWS's Free Tier RDS template doesn't offer Multi-AZ as an option at all — it's locked behind the "Production" template, which roughly doubles the database's hourly cost. I had a choice: stay on Free Tier and only be able to describe automatic failover in an interview, or spend a small amount of account credit to actually build and test it.

I chose to spend the credit. The reasoning: the entire point of this project was proving fault tolerance, not just listing services on a resume. Being able to say "I triggered a forced failover and verified the database moved to a different Availability Zone" is a fundamentally different claim than "I know RDS supports Multi-AZ." For a project meant to demonstrate engineering competence, the second claim doesn't hold up well under a follow-up question.

Cost vs. Architecture Purity: NAT Gateways per AZ

The "textbook correct" production pattern is one NAT Gateway per Availability Zone, so that a NAT Gateway failure in one AZ doesn't take down internet access for the private subnet in that same AZ. Doing this at scale, for a real company, is the right call.

For this project, I made a deliberate trade-off in the other direction: I chose to deploy a single NAT Gateway shared across both AZs to manage hourly AWS costs. I recognize that a production-grade environment requires a NAT Gateway per AZ to prevent cross-AZ dependencies and ensure high availability. However, for a portfolio demonstration where network resiliency wasn't the focal point, keeping costs low was the more practical choice.

Target Tracking vs. Manual Threshold Scaling

I initially planned a simple two-threshold scaling setup — scale out at 70% CPU, scale in at 40%. I reconsidered this once I thought through the failure mode: if those two thresholds are tuned too close together, or traffic hovers right around the boundary, you get "flapping" — instances repeatedly launching and terminating in short succession, which wastes money and destabilizes the fleet under real load.

I switched to a target tracking policy at 50% CPU instead, which lets AWS's own algorithm manage the scale-out/scale-in thresholds and cooldown timing dynamically, rather than me hardcoding two static numbers that could misbehave under conditions I hadn't anticipated. I actually heard and read from someone about that two-threshold scaling setup but as I searched further more, I found that there is a better and safer way. This allowed me to not just copy the config from a tutorial or advise but instead understand the failure mode and choose a safer default.

What Actually Mattered Here

If I'm honest, the value of this project wasn't in successfully clicking through the AWS console in the right order. It was in the hours I spent chasing down a browser protocol quirk and a metadata API version mismatch. Production systems don't fail in the ways tutorials describe; they fail in small, unglamorous, easy-to-miss ways, and the skill that matters is having a methodical process for narrowing down the cause instead of guessing.

Next, I'm rebuilding this same architecture in Terraform — taking it from a manual console build to something version-controlled, reviewable, and deployable in minutes.

DEV Community: Knirl Amboy