Surviving zone capacity decides whether a zone failover works, and Google's europe-west2 incident on October 19 is a documented case of a failover that had somewhere to go and not enough room when it got there. According to Google's incident report, a power failure took one zone offline, and the load redirected to the other zones saturated their compute slots and processing capacity. Customers saw elevated admission queues, delayed database failover completions and intermittent HTTP 500 and 503 errors.
The zone loss was the trigger. Surviving zone capacity was the second failure, and it is the one architecture reviews rarely measure. This Field Note covers what the report says, what a zone-loss capacity test has to record, and what the test deliberately leaves out. The incident details below are attributed to Google's incident page as it currently stands. Its expanded report includes root-cause and remediation details, while an older preliminary-report notice remains on the page.
What Google's Incident Report Says Happened
Google's incident page records the incident from October 19 at 23:00 to October 20 at 05:08 US/Pacific, a duration of 6 hours 8 minutes, which is 06:00 to 12:08 UTC on October 20. The page's older preliminary section gives 05:10 instead, so the timestamps are not fully consistent across the report. It attributes the zone loss to an underground utility cable fault that cut primary feed A while feed B was locked out for repairs. The facility PLCs sensed voltage upstream of the isolated feed B breaker, concluded a healthy utility source was available and never commanded a generator start, which Google’s root cause section calls a misconfiguration of the PLC logic. When the UPS batteries ran down, servers, network equipment and local storage in europe-west2-b lost power.
The report then describes the part that matters here. Redirected query executions, high-availability database failovers and compute workloads saturated the available compute slots and processing capacity in the other zones. Google reports the symptoms as elevated admission queues, delayed database failover completions and intermittent HTTP 500 and 503 errors.
For mitigation, the report lists traffic isolation, scheduler and capacity adjustments to prioritize active customer reservations, restoration and testing of the generator automation, progressive infrastructure reactivation and controlled traffic readmission. Those are accounts of what Google did during this incident. I treat them as history, not as a property to design against and not as guidance on what a customer should configure.
That is a cloud strategy assumption hiding inside a resilience diagram: that the capacity you fail over to exists, unclaimed, at the moment you need it.
Surviving Zone Capacity Is a Separate Failure From the Zone Loss
A three-zone design draws an arrow from the failed zone to the two that remain. The arrow is a routing statement. It says traffic can go there. It says nothing about whether the compute, the admission path and the database replicas in those zones can take it. Surviving zone capacity answers that, and in europe-west2 the report describes the gap where the answer should have been.
Your failover is also not the only one running. In a regional event, every affected tenant's load is redirected into the same surviving zones at the same moment, and the retries from clients that lost connections arrive with it. This is reasoning from how shared capacity works, not a claim from the report, but the report's description of saturation across the other zones is consistent with it.
Two earlier posts frame the neighboring questions. Two Nodes Can Preserve Availability. They Can't Preserve N+1 Capacity. covers capacity after a failure when the operator owns the nodes. In a public cloud region you cannot add a node to a surviving zone. You hold a claim on capacity the provider holds, and the question is what that claim is worth during the event. The HA strategy playbook states the pattern in one row of a table: retries storm the surviving zones and capacity limits reject failover. The europe-west2 report is documented evidence for that row, and it adds what the row lacks, which is a way to measure it.
A Second Mechanism, Kept Separate: Cross-Zone Control-Plane Validation
Google’s preliminary report describes synchronous control-plane validation dependencies across zonal boundaries; the expanded report shortens this to control-plane dependencies across zonal boundaries. Calls to metadata and catalog components in the offline zone timed out, causing initial job-submission stalls and API latency spikes. This is a separate mechanism from compute-capacity exhaustion in the surviving zones, and I am keeping it out of the capacity argument on purpose.
Capacity saturation shows up as queues and slow failovers because the surviving zones have no room. A validation dependency is about operations that depend on a check spanning zones, and it can delay or block them whether or not room exists. A post-incident review that files both under "the other zones were overloaded" will fix one mechanism and miss the other. The test below covers the first. The second needs its own test.
The Zone-Loss Capacity Test
The artifact is a table that measures surviving zone capacity for each workload tier, from a calculation first and a constrained rehearsal second. A real region cannot be told to lose a zone, so the first pass is arithmetic against your own peak numbers. Its job is to put a number where the diagram has an arrow.
| Record | The question it answers | Where the number comes from |
|---|---|---|
| Displaced demand | How much compute, memory and database failover load does the lost zone carry at peak, including client retries? | Your metrics at peak, not the daily average |
| Surviving capacity | What can the remaining zones serve right now, split into capacity you have reserved and capacity you would have to obtain on demand? | Quotas, reservations and what autoscaling can actually acquire under contention |
| Admission queue | How long between a scale-up request and a running instance when every tenant is asking at once? | Observed provisioning latency under constraint, not the documented happy path |
| Database failover completion | How long until the promoted replica accepts writes while the displaced load is arriving? | A failover run under load, not at idle |
| Per-workload outcome | For each service: served, degraded, queued, rejected or failed over late? | One row per workload tier, signed by the service owner |
The last row is the point. The test does not output a headroom percentage. It outputs a decision for each service about what happens when the load does not fit. A workload that tolerates a queue and a workload that returns a 503 to a customer are different business decisions, and the table forces someone to make them before the event.
A reservation is a zone-scoped claim, as Compute Engine's About reservations page describes it: capacity set aside in a specific zone. That makes the split in the second row a real input. It is a question about which of your claims on the surviving zones are held and which are hoped for.
The headroom this test justifies will appear on a cost dashboard as idle capacity. The test does not prescribe how much to hold. It makes the decision visible, and that is all it is for.
What the Test Leaves Out
The zone-loss capacity test assumes failover runs and asks only whether it has room. Whether failover is configured and exercised at all is the subject of Multi-Cloud Failover Is Mostly Theater, and this test does not repeat that argument.
It does not test whether state is recoverable after a zone loss, which is the boundary drawn in Multi-AZ Is Not a Recoverability Boundary. It does not test shared network capacity inside a region, the failure domain mapped in GCP's us-west1 Outage Mapped a Failure Domain Your Service List Couldn't. Those are different failure domains with their own tests. It does not cover the cross-zone validation dependency described above, and it says nothing about storage recovery after power returned.
Within those limits, it measures surviving zone capacity and answers one question a three-zone diagram cannot: when one zone is gone, which of your workloads fit in the other two.
Architect's Verdict
Zone redundancy is a claim about where load can go. Surviving zone capacity is a claim about whether it fits. The europe-west2 report documents the distance between the two, and most architectures are drawn as if that distance were zero.
The real problem is that nobody owns the number. The capacity lives in the provider's pool, your claim on it lives in a reservation or an assumption, and the gap between them is visible for the first time during the outage. Until someone writes the per-workload outcome down, the failover plan is a routing diagram with a hope attached.
The arrow was drawn. The number on it was never measured.
Originally published at rack2cloud.com
Top comments (1)
This reframes failover from a routing exercise into a capacity exercise. Testing the traffic shift is not enough if the surviving zone cannot sustain the load and its background jobs. The scenario needs a measured headroom target and a rollback threshold before the drill begins.