A few years ago I was the on-call engineer when we shipped what should have been a boring Thursday afternoon deploy. It turned into the incident that changed how I think about rollbacks, and I've never trusted the big red button since.
Here's the story, then the lessons.
The deploy
The change was ordinary: a new field on the orders table, an API endpoint that used it, and a migration that added the column as NOT NULL with a default. We'd done fifty deploys like it. CI green, review approved, staging fine.
The deploy went out at 6:40 PM. By 6:47, error rates on the checkout path were climbing. Not a spike, a slope - the kind of graph where you watch it for ninety seconds hoping it's a blip, and it isn't.
The on-call playbook said what every playbook says: when in doubt, roll back. So at 6:55 we rolled back.
The rollback
The application servers went back to the previous release in about four minutes. Error rates did not go back with them. They got worse.
The reason took us twenty painful minutes to see, and it's obvious in hindsight: the migration had already run. The new column existed. Worse, the new code had been writing to it for fifteen minutes, and a backfill job we'd helpfully included had rewritten a few hundred thousand rows into a shape the old code didn't expect.
Rolling back restored the code. It did nothing to the state. The old binary was now running against a database that had moved on, and every request that touched the affected rows failed in a new and exciting way.
We'd pressed the button marked "make it like it was." The button was lying.
The recovery
What actually saved us was rolling forward. We redeployed the broken release (the thing causing the original slope), then shipped a narrow fix for the actual bug, which turned out to be a missed null check in the new endpoint. Total customer impact: about 70 minutes. Time spent making it worse with the rollback: about 25 of those.
In the review the next morning, someone asked the question that reframed the whole incident: "When did we last practice a rollback against a database that had already migrated?" The answer was never. We drilled application rollbacks constantly. We had never once rehearsed what happens to the data.
What we changed
1. We stopped treating rollback as the safe default. Rollback is a tool with a blast radius, not an undo key. It restores code, not state. Any deploy that touches state - schema, queues, caches, config formats - has to answer "what does the old version do against the new state?" before it ships, not at 6:55 PM.
2. Migrations became expand-contract, no exceptions. Never change meaning and structure in the same deploy. First deploy expands: add the new column as nullable, or add the new table, and ship code that writes both. A later deploy migrates and backfills. A third contracts: drop the old shape once nothing reads it. Every intermediate state is safe for both old and new code, which means rollback stays boring at every step. That's the entire trick: keep every deploy reversible by making sure the state never outruns the code in a way the code can't survive.
3. The NOT NULL with a default got banned on hot tables. On a big table, that migration rewrites rows or holds locks, depending on your engine and version. New columns start nullable or get a separate backfill. This one rule would have prevented our exact failure.
4. We rehearsed the bad path. Once a quarter, in staging, we now run the drill: deploy a change with a migration, then roll back and watch what breaks. The first time we did it, three things broke. Better there than in production. A rollback you've never tested is a hope, not a plan.
5. Deploy and release got separated. Where it made sense, risky behavior moved behind configuration we could flip without a deploy, so "turn it off" stopped meaning "ship old code and pray the state is compatible." Even a crude version of this beats a heroic rollback.
The part that stuck with me
The failure that night wasn't the bug. Bugs ship; that's what on-call is for. The failure was that our safety mechanism had a hidden precondition - state compatibility - that nobody had written down, tested, or even said out loud. It worked in every drill because the drills never included a database.
I still roll back deploys. But now the question comes first: what state has this release already touched, and can the previous version live with it? If the answer is "I don't know," the rollback isn't the safe option. It's just the familiar one.
What's your worst rollback story? I collect these - every team has one, and they're never in the postmortem template.
Top comments (0)