✓ Human-authored analysis; AI used for formatting and proofreading.
December 7, 2021. AWS US-EAST-1 goes down. Duration: 7-12 hours. Netflix, Disney+, Robinhood, Roku, major services offline. Root cause: an automated network health system ramped up traffic on internal paths that the control plane itself depended on. The control plane couldn't issue corrective commands because the automated system was consuming its bandwidth. Self-inflicted congestion snowballed.
AWS conducted a thorough post-mortem. Implemented specific fixes. Rate limits. Monitoring. Alerts. The specific incident was addressed. The architecture moved on.
October 19-20, 2025. AWS global outage. Duration: 14+ hours. Vast swaths of the internet affected. Root cause: NLB scaling action triggered a DNS issue. The feedback loop between NLB scaling and DNS amplified both problems. The control plane needed BOTH working to fix EITHER. Both failures fed each other. Recovery required external intervention to break the loop.
Different trigger, subsystem and feedback mechanism. Same structural pattern. Same result.
The 2021 fixes didn't prevent the 2025 incident because the fixes addressed the instance. The CLASS persisted and the class produced a different instance four years later.
Four incidents, three vendors, one pattern
| Incident | Date | Trigger | Pattern |
|---|---|---|---|
| AWS US-EAST-1 | Dec 2021 | Automated system congesting control plane | Internal subsystem + control plane interaction + implicit contract |
| CrowdStrike global | Jul 2024 | Malformed channel file crashing kernel sensor | Fast propagation + insufficient protection + implicit contract |
| AWS global | Oct 2025 | NLB scaling + DNS feedback loop | Internal subsystem + control plane interaction + implicit contract |
| Cloudflare global | Nov 2025 | Security improvement doubling feature file | Fast propagation + insufficient protection + implicit contract |
Four catastrophic incidents. Three vendors. Less than four years. Same structural pattern: an implicit contract between internal subsystems was violated, and no invariant existed to catch the violation before it cascaded.
But there's a fifth incident that makes the case even more viscerally. Go back further.
The typo that broke the internet
February 28, 2017. An AWS engineer performing routine S3 maintenance typed a command to remove a small number of servers from a subsystem. The command had a typo. It removed FAR MORE servers than intended.
The cascade: S3's index subsystem lost too many servers to function. S3 went down. Hundreds of major websites and services that depended on S3 went down with it. The internet partially broke because of a TYPO.
AWS's post-mortem confirmed: a human entered an incorrect command. The system faithfully executed it. No invariant checked whether the command would remove a dangerous number of servers. No gate asked "are you sure you want to remove 80% of the index subsystem's capacity?" The system trusted the human. The human made a mistake.
Humans will make mistakes and it will reach production. How do we design for Mr Murphy?
Era 1 (Perimeter): "Trust the inside" → attacker gets inside, game over
Era 2 (Zero Trust): "Don't trust the network" → verify every request
Era 3 (Invariants): "Don't trust the configuration" → verify every state change
INCLUDING changes made by authorized humans typing correct-looking commands
The AWS engineer was authorized. The command was syntactically valid. The credentials were legitimate. Every access control check PASSED. The system was STILL UNSAFE after the command executed. Because no invariant declared "the index subsystem must never drop below N servers."
The invariant that would have prevented it:
- id: CTL.CAPACITY.MINIMUM
predicate: "obs.remaining_servers >= obs.minimum_safe_capacity"
message: "Operation would reduce capacity below safe minimum"
One YAML control. The command hits the invariant gate. The gate evaluates: "this operation would leave 20% of capacity. Minimum safe capacity is 60%. BLOCKED." The engineer sees the verdict. Corrects the command. Types the right number. The internet stays up.
Time to author this control: ~10 minutes. Cost of not having it: hundreds of millions in damage, the internet partially down for hours, and a permanent case study in every cloud architecture textbook.
Human mistakes are inevitable. The engineer was performing routine maintenance and made a typo. The kind of error every human makes multiple times per day. Training doesn't prevent typos. Checklists don't prevent typos. Only a system that evaluates the resulting state against declared invariants catches typos before they cascade.
This is why the architectural thesis isn't "make humans more careful." It's: assume the human already made the mistake. Assume it's a typo, a wrong parameter, a copy-paste error, an AI hallucination, a misunderstood runbook. The mistake WILL reach the system. The system must be designed to catch it.
The cumulative cost: $10B+. The cumulative lesson the industry hasn't learned: per-incident fixes don't prevent the class.
The AWS recurrence in detail
2021: Automated system starves the control plane
1. Customer activity triggers automated network health response
2. Response increases traffic on INTERNAL network paths
3. Control plane uses THE SAME internal paths for coordination
4. Response traffic congests control plane traffic
5. Control plane can't issue corrective commands
6. Congestion snowballs — the system can't fix itself
The implicit invariant violated: "Automated network health activity must never congest the control plane's communication paths."
Nobody wrote that down. The automated system knew nothing about the control plane's bandwidth needs. The interaction was unanticipated. Both subsystems were locally correct. The combination was catastrophic.
2025: Feedback loop between NLB and DNS
1. NLB scaling action triggered
2. DNS resolution for internal metadata encounters issue
3. NLB scaling generates MORE DNS queries (trying to scale)
4. DNS degradation generates MORE NLB scaling attempts (trying to recover)
5. Feedback loop amplifies both problems exponentially
6. Control plane needs both working to fix either
7. Neither can be fixed because the other is broken
8. Recovery requires external intervention to break the loop
The implicit invariant violated: "DNS resolution and NLB scaling actions must never impair the control plane that performs them."
Same structural property as 2021: an internal subsystem's automated behavior interacted with a control-plane dependency in a way nobody explicitly modeled. Same pattern. Different subsystems. Same result.
Why the 2021 fix didn't prevent 2025
The 2021 post-mortem probably produced:
- Rate limits on the SPECIFIC automated system that caused the 2021 congestion
- Better monitoring for control-plane bandwidth
- Alerts for similar congestion patterns
These fixes prevent the 2021 incident from recurring. They don't prevent, a different:
→ Automated system causing similar congestion
→ Control-plane dependency being overwhelmed
→ Feedback loop emerging from different subsystems
→ Implicit contract being violated
The class of failure is "internal subsystem behavior interacts with control plane in unanticipated ways." The 2021 fix addressed ONE manifestation. The class has THOUSANDS of potential manifestations. One per pair of internal subsystems that could interact with the control plane. Fixing one leaves the rest unaddressed.
The 2025 incident is empirical proof that per-incident fixes don't prevent the class.
Be more careful doesn't work
The natural response: more review, more testing, more careful rollouts. This reduces the rate but doesn't eliminate the class.
Cloud systems are too complex for human review. AWS has thousands of internal subsystems. The interaction space between them is O(N²). No human team can review all possible interactions. The interactions that cause incidents are the ones nobody anticipated by definition, the ones review would miss.
Automation is necessary. Humans can't operate cloud-scale systems manually. The automated systems that cause the cascades are required for the cloud to function. Removing them isn't an option.
Implicit contracts emerge during evolution. Each new feature, each new subsystem, each scaling improvement adds new implicit contracts. The contracts aren't foreseeable at design time. They emerge as the system evolves. The growth rate of implicit contracts EXCEEDS the rate at which any team can identify and formalize them. Unless the formalization is itself systematic.
New cloud features add new contracts continuously. AWS ships hundreds of features per year. Each potentially creates new implicit contracts with existing subsystems. The contract surface grows faster than any per-incident fix can address.
The class is intrinsic to large-scale distributed systems. Caution reduces frequency. Elimination requires architectural shift.
The three reasons the pattern persists
1. Invariants are still implicit
The contracts that matter are buried in code comments, tribal knowledge, or common sense that individual engineers carry. When the engineer leaves, the knowledge evaporates. When the subsystem evolves, the contract drifts. When a new subsystem is added, nobody checks whether it interacts with existing contracts.
Where implicit invariants live: Where they should live:
Code comments nobody reads Declared catalog entries
Team wikis that lag reality Machine-evaluated predicates
Individual engineer's heads Version-controlled YAML
"Common sense" one team has Continuously enforced constraints
2. No universal reconciliation loop
Each team applies fixes to their subsystem. Rate limits here. Monitoring there. Alerts somewhere else. No one ensures cross-subsystem invariants. The control plane has dozens of teams; each is locally correct; the system-level property fails.
Team A: "We added rate limits to our subsystem" ← locally correct
Team B: "We added monitoring to our subsystem" ← locally correct
Team C: "We added alerts to our subsystem" ← locally correct
Cross-subsystem: "Does A's behavior still respect B's envelope
after C's recent change?" ← nobody checks
3. Human-scale processes in machine-scale systems
Dashboards require humans to look. Playbooks require humans to execute. Lessons-learned require humans to remember and apply. The machine-scale system operates at machine speed. Humans can't keep up.
The 2021 incident produced lessons. The lessons were documented. The lessons were specific to the 2021 mechanism. The 2025 mechanism was different with same class, different specifics. The lessons didn't transfer because they were incident-specific, not class-level.
Companies that have eliminated the class
Multiple organizations operate WITHOUT this class of failure:
Google: Explicit invariants built into Borg from early days
Netflix: Spinnaker-managed deployments with invariant-based evaluation
Jane Street: Trading systems with continuous constraint enforcement
Aviation: Flight control systems with explicit safety envelopes
Mars rovers: Autonomous safe-mode with continuous invariant checking
Their record: NO AWS-class outages. The class is structurally absent because the architectural prerequisites (implicit contracts between subsystems) don't exist. Every critical property is an explicit invariant. A universal reconciliation loop forces state to match invariants continuously. Drift triggers immediate healing or safe-mode, not dashboards and pager alerts.
The difference between AWS and Google is historical: Google built explicit invariants into Borg early (1990s-2000s). The discipline is embedded in the culture. AWS scaled rapidly without that discipline. Implicit contracts accumulated faster than they could be made explicit.
The retrofit is hard. AWS's existing systems have thousands of implicit contracts. Making them all explicit is a multi-year project across hundreds of teams. The cost is substantial. The benefit: eliminating a class of multi-billion-dollar incidents.
The economics
| Option | Cost | Outcome |
|---|---|---|
| Continue per-incident fixes | $1B-5B per incident, recurring every 2-4 years | Reduces specific instance; class persists |
| Architectural shift | Multi-year effort, ~$100M-500M total | Eliminates the class |
Over five years at current incident rate:
Per-incident path: 2-3 incidents × $1B-5B = $3B-15B in damage
Architectural path: $100M-500M investment = class eliminated
ROI of architectural shift: 5x-30x over five years
The economics overwhelmingly favor the shift. The politics and culture delay it. Three pressures are building:
Insurance. Cyber insurance pricing is increasing for vendors with implicit-contract architectures. Insurers can identify the pattern. They price accordingly.
Regulation. FedRAMP, EU NIS2, and similar frameworks are moving toward demonstrable safety-architecture requirements. Vendors that can't demonstrate explicit-invariant enforcement face certification challenges.
Customers. Enterprise customers increasingly demand architectural demonstration. Weak architectures fail audits. Failed audits lose contracts.
Invariant catalog for AWS
For AWS-class incidents, the catalog needs three categories:
Category 1: Subsystem resource bounds
- id: CTL.CONTROL.PLANE.BANDWIDTH
predicate: "obs.control_plane_bandwidth_guaranteed_percent >= 30"
message: "Control plane must have guaranteed minimum bandwidth"
- id: CTL.AUTOMATION.RATE.BOUND
predicate: "obs.automated_response_rate <= obs.declared_rate_limit"
message: "Automated system exceeds declared rate limit"
Category 2: Control-plane protection
- id: CTL.METADATA.SERVICE.PROTECTED
predicate: "obs.metadata_service_reachable == true"
message: "Metadata service must remain reachable regardless of NLB state"
- id: CTL.DNS.INDEPENDENCE
predicate: "obs.dns_depends_on_nlb == false"
message: "DNS resolution must not depend on NLB scaling state"
Category 3: Feedback-loop detection
- id: CTL.FEEDBACK.LOOP.DETECTION
predicate: "obs.subsystem_interaction_escalation_rate <= 1.0"
message: "Subsystem interaction producing escalating feedback loop"
Each invariant is SIMPLE to author (domain expertise required, not novel engineering). Each would have prevented a specific cascade stage. Together they ELIMINATE THE CLASS not just the instance.
For every cloud operator
1. List your fast-propagation and automated systems. Each is a potential cascade source. Each has implicit contracts with the control plane.
2. Ask: "what implicit invariant does this subsystem assume about the control plane?" If the answer is in someone's head, it's an implicit contract waiting to be violated.
3. Write the invariant down. As a machine-evaluable predicate. In a version-controlled catalog. Not in a wiki or a code comment. As DATA the system continuously evaluates.
4. Accept that per-incident fixes won't prevent the class. The 2021→2025 recurrence at AWS proves it. Different trigger, different subsystem, same pattern, same result. The class persists until the architectural shift happens.
5. The economics are on your side. $100M-500M to eliminate a class that costs $1B-5B per incident. The ROI is 5x-30x over five years. The hardest part is the organizational discipline to make every implicit contract explicit across hundreds of teams.
AWS had the same class of outage twice in four years. They fixed the first one. The second one happened anyway. Because the fix addressed the instance and the class persisted. Four incidents across three vendors in four years. Same structural pattern. Same implicit contracts and cascading failures. The fix is making the implicit explicit and enforcing it continuously, mechanically, at machine speed.
Making implicit contracts explicit with 3,000+ CEL-predicate invariants replacing tribal knowledge, code comments, and "common sense" with machine-evaluated constraints. Continuous enforcement. Binary verdicts. The class-level fix for cascade failures. Stave, an open-source risk reasoning engine. Don't fix instances. Eliminate classes: bash examples/demo-ai-security/run.sh
Top comments (0)