Software disasters are usually told as legends. These five are documented — root causes from official accident investigations, costs from the record. Each was, at bottom, a single decision a checklist could have caught.
1. A misplaced break silenced a nation's phone network (AT&T, 1990)
One 4ESS switching center crashed. Normal. But its recovery message hit a bug: a break in the wrong place of a switch statement let the recovery code itself fail — and every switch that received the recovered signal crashed in turn. Nine hours later, 60,000 people had no long-distance, and 65 million calls had failed.
The one decision: test the recovery path itself, not just the code. Everyone tests the code; almost nobody tests what runs when the code fails.
2. A 64-bit float grounded a $370M rocket 37 seconds after launch (Ariane 5, 1996)
Guidance code reused from Ariane 4 converted a 64-bit horizontal-velocity float into a 16-bit signed integer. Ariane 5's faster trajectory overflowed it; both inertial reference systems halted; the rocket interpreted garbage as violent deviation and tore itself apart.
The bitter detail: the converting code ran in a phase where its output wasn't even needed — the overflow was reachable only during the 37 seconds after liftoff, on a path executed for nothing.
The one decision: re-validate reused code against the new envelope. The code was correct — for a different rocket.
3. Pounds-force met newtons; a $327M Mars orbiter disintegrated (1999)
Ground software reported thruster impulse in pound-force seconds. The spacecraft expected newton-seconds. The error accumulated for months. A separate navigation team computed the correct trajectory, disagreed with the drifting one, raised it — and the discrepancy was investigated and dismissed.
The one decision: units are types. Or at minimum: an independent reviewer whose disagreement cannot be closed as checked-fine.
4. Copy-paste deployment missed one server; $440M evaporated in 45 minutes (Knight Capital, 2012)
New code reached 7 of 8 servers. The eighth ran a dormant, obsolete flag-handling function that reactivated — and started buying at the ask, selling at the bid, millions of times a minute. The kill switch existed. Nobody authorized could press it in time.
The one decision: deployment verification per node, dead-code removal, and a kill switch with a named owner.
5. One missing bounds check turned a keep-alive ping into a server-memory faucet (Heartbleed, 2014)
OpenSSL's heartbeat trusted a client-supplied length and echoed back that many bytes — without checking the buffer. Private keys, session tokens, passwords leaked from millions of servers for two years.
And the detail that should haunt the industry: the OpenSSL team maintaining this code ran on roughly $2,000/year in donations.
The one decision: memory-safe languages for parsers — and funding critical infrastructure like it's critical.
The pattern
Four shapes hide behind all of these: the boundary nobody re-checked (Ariane, Mars), the recovery path that was itself broken (AT&T, Knight), the convenience feature nobody knew was armed (Log4Shell, from the same family), and the check that moved from hardware to software (Therac-25). Every good code-review checklist is hunting these four shapes wearing different clothes.
I collected twelve of these — each with root cause, real cost, and the single preventing decision, plus the four-shape pattern — in When Code Broke. If you run code reviews, the shapes are worth more than the stories.
Top comments (0)