A reproducible multi-AZ resilience walkthrough: spread a service across simulated zones, kill one under load, and measure the dropped requests — plus the parts that only show up in real production.
Originally published on the AWS Builder Center: https://builder.aws.com/content/3GnvKe3PBz3fP3m3pFFDFghjIxX/making-a-service-survive-an-availability-zone-outage-and-proving-it-for-dollar0
Repo: https://github.com/PradeepKandepaneni/golden-path-resilience
Most articles about availability zones tell you to "spread across three AZs" and stop there, as if the words alone did the work. They don't. Spreading pods across zones is easy to write and easy to get subtly wrong, and the only way to know whether your setup actually survives a zone going dark is to make one go dark and watch what happens.
So that's what this does. It takes a small service, spreads it across three simulated zones, then kills one zone while a stream of requests is hitting the service — and counts how many of those requests fail. If the architecture holds, that number is zero. And because the whole thing runs on a local cluster and free CI, you can clone it and watch your own zone die without spending a cent or waiting for a real outage.
This is a sequel to a earlier piece where I built a "golden path" — a free, end-to-end pipeline from a git push to a deployed Kubernetes workload. Same tiny service here; the interesting part this time isn't the pipeline, it's the topology. Repo: github.com/PradeepKandepaneni/golden-path-resilience.
I'll be honest up front about one boundary, because it matters: the demo simulates zones on a local cluster. That reproduces the Kubernetes-level behaviour — scheduling, spread, eviction, rescheduling, traffic routing — which is most of what you actually tune. It does not reproduce real cross-zone latency or partial network partitions. I'll cover those separately, from the parts of this that only show up in production. Keeping that line clear is the difference between a demo that teaches and one that quietly lies.
The setup: three zones on one laptop
Real AWS worker nodes get a topology.kubernetes.io/zone label automatically. Locally, I fake it: a kind cluster with three worker nodes, and a script that labels them az-a, az-b, az-c. Now Kubernetes believes it has three zones to reason about, and every zone-aware feature behaves as it would in a real account.
That single label is the hinge the whole thing turns on. Everything below — the spread, the eviction behaviour, the rescheduling — keys off it.
The three pieces that actually do the work
Zone resilience for a stateless service isn't one setting. It's three, and each covers a gap the others don't.
Topology spread constraints distribute the pods evenly across the zones. Six replicas, three zones, two per zone. Without this, the scheduler is free to pile most of your pods into one zone, and "we run in three AZs" becomes a lie the first time the wrong one fails.
The important detail is one field, and it's the field people get wrong:
yaml
topologySpreadConstraints:
- maxSkew: 1 topologyKey: topology.kubernetes.io/zone whenUnsatisfiable: ScheduleAnyway labelSelector: matchLabels: app: golden-path
whenUnsatisfiable: ScheduleAnyway is deliberate, and it's the opposite of what a lot of people reach for. The stricter DoNotSchedule sounds safer — enforce the even spread, always. But think about what happens during an actual outage: a zone dies, its pods are evicted, and they need somewhere to go. With DoNotSchedule, rescheduling those pods onto the two surviving zones would break the even-spread rule, so the scheduler refuses, and the pods sit Pending. You've traded away availability to preserve a tidy distribution during the exact moment availability matters most. ScheduleAnyway says: spread evenly when you can, but during degradation, keep the service up. That's the resilient choice, and it's a one-word decision that most tutorials never explain.
A PodDisruptionBudget is the second piece. Draining a node is a voluntary disruption, and Kubernetes will happily evict everything on it unless you tell it not to. minAvailable: 4 (of six) means the drain of a single zone can proceed — it only needs to evict two — while four pods are guaranteed to stay serving the entire time. The PDB is what turns "drain the node" from a gamble into a guarantee.
Readiness probes are the quiet third piece. When the evicted pods reschedule onto surviving zones, they're not instantly ready. The readiness probe gates traffic so a pod only receives requests once /healthz is green. Without it, you'd route requests at pods that are still starting and serve errors during the recovery — snatching a small outage from the jaws of a successful failover.
None of these three is exotic. The point is that resilience is the combination, and dropping any one of them leaves a gap that only shows up when a zone actually fails.
Killing a zone, and counting the damage
Description is cheap, so the repo doesn't describe — it measures. scripts/az-outage-demo.sh starts a load generator firing a few requests a second at the service, waits for steady state, then simulates the outage:
bash
kubectl cordon "$NODE"
kubectl drain "$NODE" --ignore-daemonsets --delete-emptydir-data --force
Cordon stops new pods landing on the doomed node; drain evicts the ones already there. From Kubernetes' point of view, that zone is gone. The load generator keeps firing the whole time, logging each request as ok or FAIL, and at the end the script tallies them:
==================================================
Requests sent during the zone outage : 175
Requests that FAILED : 0
PASS: the service survived the loss of zone az-a.
Zero failures across the outage. The four pods in the surviving zones never stopped serving; the two evicted pods rescheduled onto those zones and rejoined once ready; the Service only ever routed traffic to pods that were actually up. The client saw nothing.
The CI job asserts this on every push — it fails the build if more than a couple of requests drop. That's the part that makes this more than a story: a green check means the failover genuinely worked on real (if simulated) infrastructure, and you can read the run yourself instead of trusting my screenshot. Because it's a public repo, the whole thing — multi-node cluster, zone outage, load test — runs on free GitHub runners at no cost.
Why "ScheduleAnyway" is the whole lesson in one field
If you take one thing from the reproducible half of this, make it that field. It's a perfect small example of how resilience decisions are counterintuitive: the setting that sounds stricter and safer (DoNotSchedule) is the one that hurts you when it counts, and the "looser" one (ScheduleAnyway) is what keeps you up. You cannot reason your way to that from the words "spread across three AZs." You get there by asking what happens after a zone is already gone — which is the question most resilience writing forgets to ask.
Where the simulation ends and production begins
Everything above is reproducible on your laptop, and that's exactly why it's limited to the orchestration layer. Real multi-AZ adds things a local cluster can't fake, and pretending otherwise would undercut the honest parts. Three in particular are worth naming.
Cross-zone latency is real and it's not free. In a real account, every hop between zones adds a millisecond or two, and traffic that ping-pongs across zones under load adds up — in latency and, on AWS, in cross-AZ data transfer cost. Local kind has effectively zero inter-node latency, so it can't show you this. In production it shapes real decisions, like topology-aware routing to keep traffic in-zone when possible.
Partial failures are messier than a clean drain. My demo removes a zone cleanly. Real zones fail in uglier ways — a network partition where the zone is reachable but slow, or intermittently up. Those are the scenarios that expose whether your health checks and timeouts are tuned correctly, and a clean kubectl drain never triggers them.
The data layer is the hard half, and it's deliberately not in the reproducible demo. Stateless failover — what this repo shows — is the easy part precisely because any pod can serve any request. The moment state is involved, resilience gets genuinely hard: a multi-AZ database failover has a measurable failover window, in-flight transactions to reason about, and connection pools that need to survive the primary moving.
Those three are why the honest framing of this whole exercise is "I reproduced the orchestration behaviour for free, and here's where production goes further" — not "I built a production multi-AZ system on my laptop." The reproducible part is genuinely useful and genuinely verifiable. The rest is where real accounts and real incidents teach you things a simulation can't.
Takeaways
Resilience is a combination, not a setting. Spread, disruption budget, and readiness each cover a gap the others leave. Ship one without the others and you've built a service that looks resilient right up until a zone fails.
Ask what happens after the failure, not before. The ScheduleAnyway decision only makes sense if you reason about the moment a zone is already gone. Most resilience mistakes come from designing for the healthy state and never simulating the broken one.
Prove it, don't describe it. "Spread across three AZs" is a claim. "Here's the demo that kills a zone under load and reports zero dropped requests, and here's the CI run" is evidence — and evidence is worth more than any architecture diagram.
The repo is public and the demo runs green for $0: github.com/PradeepKandepaneni/golden-path-resilience. Clone it, run make up && make demo, and kill a zone yourself.
Top comments (0)