When Did You Last Actually Test Your Failover?

#discuss #devops #sre #cloud

In October 2026, one of Google Cloud's London zones went down for about six hours. One utility feed was already down for maintenance when a fault cut the other. That's exactly what generators are for. They didn't start. A misconfigured generator controller read voltage on the isolated feed, decided everything was fine, and stayed off while the UPS batteries drained.

I keep coming back to that one.

I work at Control Plane, and for the past few months I've been maintaining a tracker of major public cloud outages: AWS, Azure, Google Cloud, and Cloudflare, back to 2020, 56 of them sourced to the provider's own status page or post-mortem, the rest to press coverage. It's at 62 incidents now. Going through them one by one changed how I think about the word "redundant".

The backups were there. They just didn't work.

In 7 of the 62 outages, a backup or redundant system was part of the failure:

Generators that didn't start (London, October 2026; a Hillsboro, Oregon data center hosting Cloudflare's control plane, November 2023)
A control-system fault that blocked the switch to generator power (Azure West US, February 2026)
UPS batteries that failed when utility power dropped (Google Cloud Columbus, March 2025)
A backup power unit that failed after both feeds tripped (Google Cloud Netherlands, July 2026)
Redundant cooling systems that failed together during record UK heat (Google Cloud London, July 2022)
A regional failover that failed alongside the zone it was protecting (Azure Southeast Asia, February 2023)

These are some of the best-run facilities on the planet. The redundancy existed on paper, it passed audits, and on the day it was needed, it didn't do its job.

If that happens to hyperscaler power systems, I'm pretty sure it happens to our failover runbooks.

The other numbers that stuck with me

Median outage: 6 hours 8 minutes.
Of the 57 that have ended, 37 lasted 4 hours or more. 16 lasted 12 hours or more. 6 lasted more than a day.
55% were caused by a change: a config push, a deployment, or a software bug in automation. Power and cooling were 11. Physical damage was 3: two drone strikes and a flood that led to a fire.

So the realistic question isn't "will my region go down for 30 seconds". It's "what happens if my primary region is gone for six hours, starting with a config change I can't see".

Waiting it out is a strategy. For a lot of apps it's even the right one. Just make it a choice you made on purpose.

What "tested" actually means

The bar I'd use, and I'm curious how many teams actually clear it:

You've failed the primary on purpose, in production or a production-shaped environment, in the last 90 days. Not a tabletop exercise. Traffic actually moved.
You timed it end to end: detection, DNS or routing change, the last client that was still holding a cached answer, and the surviving region absorbing the load.
The surviving region had to scale up and you watched it do it. A standby that's never taken full traffic is the generator that's never been started under load.
You failed back. A lot of teams test the first half and discover the second half the hard way.
Somebody other than the person who built it ran it. If it only works when one engineer is awake, you don't have a system yet.

My guess is most teams have done 1 at some point and very few have done all five recently.

Why nobody tests it

I think it's mostly three things:

It's scary. Failing over production on purpose feels like causing the outage you're insuring against.
It's expensive to stage. Real failover needs real capacity in the second region, and that capacity costs money every month whether you use it or not.
Nothing forces it. Audits ask whether you have a DR plan. They rarely ask when traffic last moved.

The fix for the first two is designing so failover is boring: active-active, where every region already takes traffic, so "failover" is one region going away and the others getting a bit busier. That's the approach we build around at Control Plane, but you can get a long way there with plain multi-region deploys and health-checked routing.