DEV Community

Cover image for Every Big Outage This Week Started With a Change
Noah Grinstein
Noah Grinstein

Posted on AI-assisted

Every Big Outage This Week Started With a Change

Email went down for a day. Every departing flight in the US was held for over four hours. A big bank's transactions stalled for most of a Monday. Social media decided it was a coordinated attack, a solar storm, or AI taking over.

I read the official explanations, and they're far more boring than that. Almost every incident with a published cause traces back to a change: a code fix, a router swap, a maintenance posture. (I used AI tools to help research and draft this, and checked the causes against the companies' and agencies' own statements.)

The causes, as stated so far (plus one from last month)

Date What broke Stated cause Attack?
Nov 12 to 13 Microsoft 365 (Exchange, Outlook, Teams), about a day A defect introduced in a July fix, triggered under heavy load while capacity was already reduced Ruled out by Microsoft
Nov 13 FAA nationwide ground stop, about 4.5 hours A router installed during a copper-to-fiber upgrade flooded the FAA telecom network with connection attempts No sign found (DHS); FBI checking the router
Nov 16 Chase transactions, most of a day A software bug in an in-house hardware system Ruled out by Chase
Oct 20 Google Cloud London region, about 6 hours A utility fault during planned maintenance; a controller misconfiguration kept generators from starting Power event

AWS also had about three hours of sign-in trouble in us-east-1 on November 15 and hasn't published a cause. United Airlines' pilot apps went down for about 90 minutes on November 16, with no cause given beyond "no indication" of a cyberattack.

Both explanations are preliminary. Microsoft's full report is due November 20, and the FAA hasn't said whether the router was misconfigured or faulty.

Three things these have in common

The change and the failure were far apart. Microsoft's defect shipped in July and failed in November. It only showed up under a specific mix: heavy load, newer high-core servers, and about 14% of gateway capacity already out of service for an unrelated reason. Microsoft says its load tests ran on weaker, lower-core hardware than production, so testing never reproduced those conditions.

Headroom was already thinner. Microsoft says its gateways still had enough headroom with nearly 14% of capacity offline. Then Teams traffic was moved onto them about 15 hours earlier, and the combination tipped them into the defect. Google's London zone lost power while one of its two power feeds was isolated for maintenance. Neither "one thing down" was an incident on its own. Each one quietly removed the margin the system needed for the next problem.

Recovery made it worse. As Microsoft's gateways failed, traffic was rerouted to the healthy ones, which then failed too, until more than 80% stopped responding. Restarted servers were hit by a thundering herd of reconnecting clients. At the FAA, routers at two sites flooding the network with connection attempts overwhelmed older equipment, and diagnosis was slowed because some maintenance records were on paper.

This isn't about blaming anyone. Every team running a large system has a version of these stories. That's the point: the scary week was made of ordinary failure modes.

The timing question

One reply on Hacker News suggested a seasonal explanation: teams rush changes out before the Thanksgiving code freeze, so early-to-mid November gets a cluster of change-related failures. I couldn't find data that proves it, and the Microsoft defect is months older than any pre-holiday rush. But it matches what a lot of people say about their own deploy calendars.

So I'm curious

  • Does your team freeze changes before Thanksgiving or the end of the year? Has a freeze ever actually saved you, or does it just move the risky deploys to the week before?
  • How do you track "degraded but not down" capacity, like 14% of a fleet offline, so it counts as an incident before something else lands on top of it?
  • Have you ever had a defect sit quietly for months before the right load found it?

I'd especially like to hear from people who run change management at scale. What actually works?

I wrote up the full five-day timeline and the viral claims, including the November 17 blackout rumor, here: There Is No 90-Day Blackout on November 17.

Disclosure: I work at Control Plane, a cloud infrastructure company.

Top comments (0)