On November 12, Microsoft 365 email and collaboration services failed for about a day across dozens of countries. On November 13, the FAA held every departing flight in the United States for more than four hours. Officials said there was no sign they were related, and neither has been tied to a cyberattack.
They do share a shape. In both, a small, local fault became a system-wide outage because of how the system reacted to it. This post walks through each failure stage by stage, using the preliminary public explanations, then maps each stage to a defense you can apply to your own systems.
Two caveats up front. Both explanations are preliminary: Microsoft's final post-incident report is due November 20, and the FAA hasn't finished its root-cause review. And I wasn't inside either incident. Everything here comes from public statements and reporting, cited inline. (I used AI tools to help research and draft this, and checked every factual claim against those sources.)
Part 1: Microsoft 365, November 12 to 13
What happened
Exchange Online and Outlook were hit hardest, with Teams, Copilot, SharePoint, and other services affected. Morningstar DBRS estimated insured losses of $300 million to $500 million. Microsoft's preliminary post-incident report, covered by GeekWire, The Register, and BleepingComputer, describes a chain of events.
Stage 1: a latent defect
A fix rolled out to Microsoft 365's networking gateway servers in July contained a defect. Under heavy load, it caused lock contention: threads waiting on each other for a shared lock, burning CPU and memory without doing useful work. Microsoft says the contention was inside an internal Windows process and got worse on newer machines with more cores. My reading is that more cores means more threads competing for the same lock.
The defect sat in production for more than three months without causing an outage, because normal load never pushed the gateways into the state where contention ran away.
Stage 2: headroom erodes
Two unrelated things removed the safety margin before the failure:
- About 14% of gateway capacity was already out of service because of an earlier, separate issue.
- Roughly 15 hours before the outage, according to GeekWire, Teams traffic had been moved onto the Exchange gateway pool.
Neither was an incident on its own. Microsoft says the reduced capacity alone left enough headroom. The combination is what tipped the gateways into the defect. Microsoft also says its load testing tended to focus on weaker hardware with fewer cores than production.
Stage 3: overload spreads through rerouting
Once some gateways crossed into runaway contention and stopped responding, automatic rerouting did what it was designed to do: it moved their traffic to the healthy gateways. That extra load pushed those gateways into the same state. The failure spread until more than 80% of gateway servers were unresponsive.
This is the classic shape of a cascading failure. The mechanism meant to contain a failure, moving work away from a broken node, spreads it when the cause is load itself.
Stage 4: recovery fights itself
When engineers restarted servers, clients that had been failing all reconnected at once. The restarted servers were overwhelmed before they could stabilize, a pattern usually called a thundering herd. GeekWire's summary of the report names it explicitly.
Stage 5: containment boundaries leak
Microsoft's design is meant to keep a problem in one region from spreading to others. The preliminary report says safeguards meant to keep a problem in one region didn't hold, and it doesn't yet explain why. That's one of the details the final report should clarify.
Microsoft resolved the outage by rolling out an update that reverted the defective change.
Part 2: the FAA ground stop, November 13
What happened
Around 3:30 to 3:36 p.m. Eastern, air traffic controllers began seeing radar and radio inconsistencies. At 3:54 p.m., the FAA issued a nationwide ground stop. Instability reached 17 of the 22 Air Route Traffic Control Centers that manage high-altitude traffic, along with several approach control facilities. The stop was lifted at about 8:15 p.m. FedScoop reported it was only the second nationwide ground stop since 2001. The FAA's own Q&A covers the basics.
Stage 1: a change at the edge
L3Harris, which runs the FAA's telecommunications network, was replacing copper circuits with fiber at an FAA site in El Dorado, Arkansas. A new router was installed as part of that work. AeroTime reports it was a Juniper unit replacing an older Cisco device. HPE, which owns Juniper, said the device functioned as it was configured to by the installer.
Stage 2: one device floods a shared network
The new router began generating connection attempts across the FAA's telecommunications network. The FAA describes about 1,000 connection attempts every four to five seconds. Whether the cause was a configuration error or a faulty unit is still undetermined, according to Duffy.
Stage 3: older equipment can't absorb it
Legacy equipment elsewhere on the network couldn't handle the flood. Radar and communications feeds to most en-route centers degraded. A fault at one small site became a national event because the network was shared and the legacy equipment apparently couldn't protect itself from a misbehaving peer.
Stage 4: diagnosis is slow
FAA Administrator Bryan Bedford said the agency spent about 45 minutes trying to find the cause without success before issuing the ground stop, partly because this kind of maintenance is logged by hand on paper (FedScoop). When you don't have a fast, searchable record of what changed and where, the first hour of an incident goes to finding the change.
Stage 5: isolation, then a fleet-wide fix
Isolating the El Dorado router, and a second one at a site in Amarillo, Texas, stabilized the network. The FAA then disabled the suspect function on about 60 similar routers across its network. The FBI is examining the router at an L3Harris lab, which officials describe as a precaution rather than evidence of an attack.
The shared shape
| Stage | Microsoft 365 | FAA | Defense |
|---|---|---|---|
| Trigger | July fix with a load-dependent defect | Router installed during a fiber upgrade | Staged rollouts, plus load testing at degraded capacity |
| Hidden precondition | 14% of capacity already out; extra traffic moved in | Shared network; legacy equipment with no protection from a noisy peer | Treat degraded capacity as an incident; per-peer rate limits |
| Amplifier | Rerouting moved load onto healthy servers, which then failed | Connection storm propagated across the network | Load shedding and admission control instead of unlimited failover |
| Recovery problem | Thundering herd on restarted servers | Slow diagnosis; paper logs | Retry backoff with jitter; searchable change records |
| Containment | Regional isolation didn't hold | Two sites affected 17 of 22 centers | Cell-based isolation with tested boundaries |
Defenses you can apply
None of these would have been easy for either organization. They're the standard toolkit for this failure shape, and most are cheaper to add before an incident than during one.
Load-test at degraded capacity, not full capacity. A system tested with every server healthy can pass every test and still fail when 14% of the fleet is out. Run load tests with a meaningful slice of capacity removed, because that's the state your system will be in when the next problem arrives.
Count lost headroom as an incident. "Some servers are down but we're fine" is the most dangerous status in operations. Alert on remaining headroom, not just on errors, so a reduced fleet gets attention before something else lands on top of it.
Shed load instead of moving all of it. Failover assumes the survivors have spare capacity. When the cause is load, moving all of a failed node's work onto its neighbors spreads the failure. Admission control (rejecting some requests quickly when a server is near its limit) keeps the rest of the fleet alive. Google's SRE book covers this in its chapter on addressing cascading failures.
Make clients back off with jitter. If every client retries immediately when a service comes back, the service goes down again. Exponential backoff with random jitter spreads the reconnections out. Server-side, ramping traffic back gradually to restarted nodes does the same thing from the other direction.
Rate-limit peers, not just users. A shared network should assume any device can misbehave. Per-port or per-peer limits on connection rates turn one noisy device into a local problem.
Keep a searchable record of every change. The first question in nearly every incident is "what changed?" If the answer lives in a ticket system, a spreadsheet, and someone's paper notes, diagnosis time goes up. A single timeline of deploys, config changes, and hardware work is one of the highest-value tools an operations team can have.
Test your isolation boundaries. Regions and cells only contain failures if nothing shared crosses them. Shared control planes, config systems, and dependency services are where boundaries leak. The only way to know a boundary holds is to break something on one side on purpose and watch the other side.
Where I landed
The scariest week of outages this year came down to a load-dependent defect, a router installed during an upgrade, and recovery mechanisms that amplified the damage. None of that is exotic. It's the same set of failure modes every large system has, which is why the defenses are well known, and also why they're worth revisiting before your own version of this week.
I'll update this post when Microsoft publishes its final report and the FAA completes its review.
Disclosure: I work at Control Plane, a cloud infrastructure company.
Originally published on Hashnode.
Top comments (0)