How I Built a Self-Healing, Multi-AZ Architecture on AWS — And What Broke Along the Way

The Problem I Set Out to Solve

One such example of an AWS project/build can be as simple as setting up one EC2 instance and then installing a web server. The moment that instance crashes or traffic spikes, the whole thing falls over.

I wanted to build something that answers the question when it comes to "high availability": what happens when something fails, and does the system recover without a human stepping in

What did I build?

I set out to build a web architecture with no single point of failure — spanning two Availability Zones, load-balanced, auto-scaling, with a self-healing database — and then to actually break it on purpose to prove it recovers, instead of just trusting the console said everything was configured correctly.

The Architecture

• A VPC spanning 2 Availability Zones, with public subnets for the load balancer and private subnets for everything else
• 1 NAT Gateway for both AZ. Note: This should've been 1 NAT Gateway per AZ, so it should be a total of 2 NAT Gateways The reason for this is explained in this section, where I discussed the trade-offs I did.
• An Application Load Balancer distributing traffic to an Auto Scaling Group of EC2 instances
• RDS running MySQL in Multi-AZ mode, with a live standby ready to take over
• CloudWatch monitoring the whole thing, with an alarm and an SNS to notify me.

The Challenges I encountered

Incident 1: The ALB That Wouldn't Respond

After wiring up the VPC, security groups, target group, and load balancer, I hit the ALB's DNS name in my browser and got:

My first assumption was a security group misconfiguration. So I worked through it methodically:

1. Checked the ALB's security group — inbound HTTP 80 from 0.0.0.0/0 was correctly set
2. Checked subnet placement — the ALB was correctly deployed across both public subnets
3. Checked the ALB's scheme — confirmed "Internet-facing," not "Internal"
4. Checked the listener — HTTP:80 was correctly forwarding to my target group
5. Checked Network ACLs at the subnet level — default allow-all rules were untouched

Every single AWS-side configuration was correct. That's the moment it clicked that the problem probably wasn't in AWS at all — it was somewhere between my browser and AWS. I tried the same URL from my phone on cellular data, and then just paid closer attention to what was actually in my address bar.

The issue: I'd pasted the bare DNS name into Edge, and the browser had silently prepended https://. My ALB only had an HTTP listener configured — no HTTPS/443 listener existed at all. The browser was trying to reach a port that wasn't listening for that protocol, and quietly timing out instead of giving a clear "connection refused."

The fix: type the URL explicitly as http://....

What this taught me: when every layer of your infrastructure checks out, stop re-checking the same layer and start questioning your assumptions about the client side. I spent close to 20 minutes re-verifying security groups I'd already verified, when the actual bug was in how I was typing a URL.

Incident 2: The Page That Loaded, But Told Me Nothing

Once the ALB was reachable, my page loaded — but it was supposed to display the serving instance's ID and Availability Zone, and both fields were blank.

My EC2 instances pull this data from AWS's internal instance metadata service, via a request like:

curl -s http://169.254.169.254/latest/meta-data/instance-id

This is a well-documented pattern, and I'd used it based on older references. Apache was running fine — the page loaded — so my instinct was that the problem was somewhere in my HTML generation logic. It wasn't. The metadata request itself was silently failing.

The root cause: Amazon Linux 2023 requires IMDSv2 by default — a token-based authentication flow for metadata requests. My script was using the older tokenless (IMDSv1-style) request pattern, which gets rejected without an explicit error visible in the page output — curl just returned empty, and my script happily wrote empty values into the HTML.

The fix required requesting a session token first, then passing it as a header on every subsequent metadata call:

TOKEN=$(curl -s -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 21600")
INSTANCE_ID=$(curl -s -H "X-aws-ec2-metadata-token: $TOKEN" http://169.254.169.254/latest/meta-data/instance-id)

But here's the part that actually tests whether you understand Auto Scaling Groups: fixing the script wasn't enough. My existing instances were already running the old script — updating a Launch Template only affects instances launched after the change, not ones already running. I had to publish the corrected script as a new Launch Template version, set it as default, and then trigger an Instance Refresh — which gradually and safely replaced my running instances with new ones, one at a time, keeping at least one healthy throughout the process. That refresh took about 12–15 minutes for two instances, which is worth knowing going in so you don't assume something's stuck.

Now, load balancer distributes traffic and the instances are spread across two AZs

What this taught me: "no visible error" doesn't mean "no failure." A script can execute successfully from the OS's perspective while still failing at the thing it was actually meant to do. And infrastructure-as-config (Launch Templates) has a subtlety that infrastructure-as-code makes more obvious later: changing the definition doesn't retroactively change what's already running.

Incident 3: Auto Scaling Group "flapping" risk

Initially considered a manual step-scaling setup (scale out at 70% CPU / scale in at 40%). Switched to target tracking at a 50% target instead — reduces configuration surface and lets AWS manage thresholds dynamically rather than risking two closely-tuned static alarms causing rapid scale in/out cycles.

---

Verification & Testing

Configuration alone isn't proof — each piece of this architecture was actually tested:

Load balancing: confirmed by refreshing the ALB's URL repeatedly and observing the displayed Instance ID alternate between the two running instances.

Auto healing / monitoring: intentionally removed the ec2-sg inbound rule to cut ALB→EC2 connectivity, then confirmed:

• Target Group flipped both instances to unhealthy
  

• The CloudWatch alarm transitioned to In alarm

• An SNS email notification was recieved

• The ASG attempted to replace the unhealthy instances (visible in its Activity log)

Then reverted the rule and confirmed full recovery (targets back to healthy, alarm back to OK).

Database failover: using RDS's "Reboot with failover" option, forced the Multi-AZ database to promote its standby in the second Availability Zone to primary. Verified via the console that the instance's Availability Zone changed before vs. after — confirming automatic failover actually works, not just that the checkbox was enabled.

Trade-Offs I Had to Reason Through

Cost vs. Genuine Fault Tolerance: Multi-AZ RDS

AWS's Free Tier RDS template doesn't offer Multi-AZ as an option at all — it's locked behind the "Production" template, which roughly doubles the database's hourly cost. I had a choice: stay on Free Tier and only be able to describe automatic failover in an interview, or spend a small amount of account credit to actually build and test it.

I chose to spend the credit. The reasoning: the entire point of this project was proving fault tolerance, not just listing services on a resume. Being able to say "I triggered a forced failover and verified the database moved to a different Availability Zone" is a fundamentally different claim than "I know RDS supports Multi-AZ." For a project meant to demonstrate engineering competence, the second claim doesn't hold up well under a follow-up question.

Cost vs. Architecture Purity: NAT Gateways per AZ

The "textbook correct" production pattern is one NAT Gateway per Availability Zone, so that a NAT Gateway failure in one AZ doesn't take down internet access for the private subnet in that same AZ. Doing this at scale, for a real company, is the right call.

For this project, I made a deliberate trade-off in the other direction: I chose to deploy a single NAT Gateway shared across both AZs to manage hourly AWS costs. I recognize that a production-grade environment requires a NAT Gateway per AZ to prevent cross-AZ dependencies and ensure high availability. However, I got concerned on the adding another NAT Gateway as it can quite stack up costs plus, I wasn't fully aware yet if my free credits on the AWS account would cover it. So, I decided to keep costs low for now.

What Actually Mattered Here

If I'm honest, the value of this project wasn't in successfully clicking through the AWS console in the right order. It was in the hours I spent chasing down a browser protocol quirk and a metadata API version mismatch. Production systems don't fail in the ways tutorials describe; they fail in small, unglamorous, easy-to-miss ways, and the skill that matters is having a methodical process for narrowing down the cause instead of guessing.

Next, I'm rebuilding this same architecture in Terraform — taking it from a manual console build to something version-controlled, reviewable, and deployable in minutes, then adding on some improvements.