For two hours on February 28, 2017, Amazon could not mark S3 as down on its own health dashboard, because the dashboard's admin console depended on S3. Every postmortem contains two stories, and that detail belongs to the second one. The first story explains what broke, and it gets the headlines. The second explains how the team got back, and it keeps revealing that the tools needed to end an outage were standing on top of the thing that failed. A recent essay on dependency blindness and the quiet ways invisible systems break flags the habit behind it as a warning sign: recovery plans that "depend more on memory and intuition than on clear system design." The biggest outages of the past decade keep proving that warning right.
The Status Page That Lived on the Patient
The S3 story is worth retelling in full. At 9:37 a.m. Pacific, an engineer on the S3 team, following an established playbook, ran a command meant to remove a small number of servers from a subsystem used by the billing process. One input was entered incorrectly. A far larger set of servers disappeared, including the ones behind two subsystems that the rest of S3 in us-east-1 relied on: the index, which manages the metadata and location of every object in the region, and placement, which allocates storage for new ones. Both needed a full restart.
That is where the first invisible system surfaced. As Amazon's own summary of the S3 disruption admits, the company had "not completely restarted the index subsystem or the placement subsystem in our larger regions for many years." S3 had grown enormously in the meantime, so restarting those systems and running the safety checks that validate their metadata took longer than expected. The index was fully back at 1:18 p.m. and placement at 1:54 p.m. Meanwhile, the service statuses on that dashboard couldn't be updated until 11:37 a.m., so the largest cloud provider in the world reported on its own outage through Twitter and banner text.
The typo started that morning, but it didn't decide its shape. A cold-start path nobody had walked at current scale decided how long it lasted, and a status page living on the patient decided how little customers could see while it did.
The Safety Feature That Hid the Building
Facebook's outage on October 4, 2021 followed the same script at a higher volume. During routine maintenance, a command meant to assess the availability of global backbone capacity took down every connection in the backbone instead. An audit tool existed to block exactly that kind of command, and a bug in it let the command through.
Then a safety mechanism finished the job. Facebook's DNS servers are built to stop advertising their BGP routes when they can't reach the data centers, which is normally a sensible way for an unhealthy location to step aside. With the backbone gone, all of them stepped aside at once, and Facebook, Instagram and WhatsApp vanished from the internet for roughly six hours.
Meta's engineering write-up of the outage is unusually candid about the way back. Engineers could not reach the data centers "through our normal means because their networks were down," and "the total loss of DNS broke many of the internal tools we'd normally use to investigate and resolve outages like this." So people were sent on site, into facilities designed to be hard to get into, full of hardware and routers designed to be hard to modify even with physical access. Every lock that keeps an attacker out is also a lock your own engineers will have to open on the worst day of the year.
The same post holds the hopeful half of the story. Once the network was back, Meta couldn't simply switch everything on, because its data centers had dropped tens of megawatts of power draw, and reversing that suddenly could put "everything from electrical systems to caches at risk." It restored traffic in stages and credited the "storm" drills it had been running for a long time, exercises that take a service, a data center or an entire region offline on purpose. The way back had been rehearsed, and it showed.
Fixing the Bug Is the Short Part
Newer incidents let you measure the gap between repairing a cause and recovering a system. On October 19 and 20, 2025, a latent race condition in DynamoDB's automated DNS management left the service's regional endpoint in us-east-1 with an empty DNS record. The trouble started at 11:48 p.m. Pacific, and all DNS information was restored by 2:25 a.m. The event itself didn't end until 2:20 that afternoon.
Those remaining twelve hours were recovery, as everything built on DynamoDB tried to come back, and one of the first systems in line was a component few customers had heard of. EC2's DropletWorkflow Manager holds a lease on every physical server, or droplet, that hosts instances, and its checks on those leases depend on DynamoDB. While DynamoDB was unreachable, the leases expired one by one, and a droplet without a lease can't receive new instances. When DynamoDB returned at 2:25 a.m., the manager began re-establishing leases across the entire EC2 fleet, the work took so long that it timed out and was queued again, and the system fell into what AWS calls "congestive collapse." The summary also contains a sentence worth reading twice: the situation "had no established operational recovery procedure."
Google Cloud hit the same wall in June 2025. A policy change with unintended blank fields sent its Service Control binaries into a global crash loop, and an emergency switch that disabled the failing code path was fully rolled out within 40 minutes. In us-central1, though, full recovery took up to about two hours and forty minutes, because the restarting tasks stampeded the Spanner table they depend on. Google's report says plainly that Service Control "did not have the appropriate randomized exponential backoff." GitHub met a version of the same problem on August 17, 2026, when errors in some Copilot services set off a client-side retry loop that added traffic during recovery, and the team had to tame it before it could safely restore traffic.
The purest example arrived in May 2026. Coinbase ran its exchange matching engine as a five-node Raft cluster inside an AWS cluster placement group, a feature that packs instances close together within one availability zone for low latency. After several chillers failed in one AWS data hall, a thermal shutdown followed, and when AWS terminated instances in that group, three of the five nodes went down and the cluster lost quorum. According to Coinbase's postmortem, getting it back took a new node group outside the impaired placement group and "an emergency code change shipped during the incident to remove a startup assumption that all five cluster nodes were resolvable." The startup path assumed a healthy cluster. It was code that only matters after a disaster, built on a premise that only holds when there isn't one.
Why the Way Back Rots
None of these teams was careless. They ran into a property of recovery code that has nothing to do with skill: it is the least-executed code you own. The happy path runs millions of times a day, so a broken assumption in it gets caught within minutes. The cold start, the failover script, the break-glass account and the status-page login may run once every few years, and the system keeps changing underneath them the whole time.
That change comes from three directions, and each one looks like progress. Growth changes the physics, so a restart that took minutes at launch takes hours at today's scale, as S3 discovered. Consolidation changes the topology, because putting chat, docs, dashboards, paging and deploys behind one identity provider in one cloud is efficient right up until that provider or that cloud is what's down. Hardening changes the locks, so the controls that keep attackers away from production also slow down your own responders. Each decision is reasonable on its own, and none of them shows up on a dashboard, because dashboards measure a running system, not the path for bringing a dead one back. Recovery paths rarely break in one dramatic moment. They drift, one sensible decision at a time, until they route straight through the failure they were built to escape.
Write a Bill of Materials for the Way Back
We already know how to make hidden dependencies visible. Teams generate software bills of materials so nobody has to guess what a build contains. The way back deserves the same treatment: a short document listing everything a recovery needs, built by asking one question of every item, "what does this need in order to work, and could that be down at the same time?" Start with these:
- The status page. If it shares a cloud account, a DNS provider or a login with your product, assume it goes dark with your product, the way AWS's dashboard did in 2017.
- Access. Can someone with authority reach production when SSO, the VPN or the office network is down? Break-glass credentials belong outside the identity provider they exist to bypass, and they need testing on a schedule, because an untested break-glass account is only a rumor.
- Runbooks. If they live in a wiki behind the same login and network as production, keep an exported copy somewhere that shares neither.
- Communication. Pick an incident channel that depends on neither your own infrastructure nor your main identity provider, and make sure everyone on call knows where it is before they need it.
- Cold starts. Record when each critical system last started from zero, not one instance at a time, and how long it took. If the honest answer is "never," every recovery estimate you have is a guess.
- Startup assumptions. Read your boot and init code for anything that expects a full cluster, a reachable config service or a fixed peer count. Those are Coinbase's five resolvable nodes waiting to happen.
- Retries. Every client and worker that retries should back off exponentially with jitter, or your recovery will arrive as a stampede.
Expect at least one loop on the first pass, such as a secrets vault hosted on the cluster it holds the keys for, or a deploy pipeline that needs the service it is supposed to redeploy. Finding it on paper is cheap. Finding it during an incident is how a short outage turns into a long one.
Rehearse Without Your Tools
Chaos engineering usually asks whether a system survives a failure. The postmortems above raise a narrower and more uncomfortable question: can your team recover when its own tools are part of the failure? Meta's storm drills point toward the answer. For your next game day, take something away from the responders instead of from production. Declare the identity provider down, move the team off its usual chat workspace, or require one service to come up cold in a fresh environment with nothing cached. Then listen for the moment someone says, "wait, how do we..." Each of those moments is an invisible dependency surfacing on a quiet afternoon instead of in the middle of the night with customers watching. Add it to the bill of materials, fix the worst one, and run the drill again next quarter.
The Fire Code Principle
Buildings learned this long before software did, mostly from fires that taught it the hard way. Fire codes insist that exits keep working after everything else in the building has failed. Exit signs and emergency lighting carry their own backup power, stairwells are enclosed and separated from the rest of the structure, and everyone knows not to take the elevator. Underneath all of it sits one rule: the way out must not depend on the systems that are on fire.
Software has no fire marshal, so the inspection falls to us. Walk your exit route while the building is quiet, and make sure it doesn't run through the fire.
Top comments (1)
Official Platform Update
Security protocols have been updated for all developer accounts.