The rollback that wasn't: how one bad deploy became two outages

The rollback that wasn't: how one bad deploy became two outages

We had a rollback plan. We were proud of it. It lived in the runbook, one tidy paragraph, and it said — in essence — "re-run the previous deploy."

Then the day came. A migration shipped at 14:00, error rates climbed by 14:20, and the on-call engineer typed the sacred words: "just roll it back."

The rollback ran. The errors climbed faster. Because the rollback was never a rollback — it was an intention. It said "revert the deployment." It did not say that the migration had already rewritten 40,000 rows into a shape the old code couldn't read. We rolled the application back into a database it no longer understood, and one outage became two. The first outage lost the afternoon. The second one lost the data.

The postmortem took one afternoon and produced one sentence that has run our deploys ever since: a rollback that has never been run is not a rollback — it's a hope.

The card that replaced the hope

Every deploy now ships with a one-page card, and the card has five fields. Not a committee. A card.

1. Rollback as exact commands. Not "undo the change" — kubectl rollout undo deploy/billing --to-revision=41, or git revert 4f2c1a9 && ./deploy.sh, or "restore snapshot pre-migration from panel". The 2am version of you executes text, not judgment. If you can't write the commands, you don't have a rollback; you have a second incident on layaway.

2. Measured rollback time. We run the rollback on staging and write the real number: 6m40s. Then we set the decision point at roughly half of it. If the new deploy looks sick at +3 minutes and recovery takes 7, you're already behind when you start. Slow rollbacks executed late are how outages double.

3. Migration blast radius, stated out loud. Code rollbacks are usually safe. Schema rollbacks can eat data written since the deploy. Our rule is forward-only migrations — add a nullable column this release, drop the old one next release — and the card must state explicitly whether data written during the bad window survives.

4. Tripwires, decided before shipping. "Roll back if error rate > 2% for 5 minutes, checkout fails twice, or p95 passes 800ms." Deciding the criteria in advance removes the worst part of incident response: arguing with yourself mid-outage and losing.

5. Rollback is a deploy. It gets the same watcher, the same smoke checks (login, one test order, error dashboard), and the same one-line announcement. "It's just a rollback" is the sentence that starts the second outage.

The upgrade path

The fastest rollback isn't a rollback at all — it's a feature flag. The toggle flips in seconds while a redeploy takes minutes, and most of our scary deploys are behavior changes, not code defects. Flags for behavior, deploys for defects, forward-only for schemas. That's the whole policy, and it fits on a sticky note.

We also stopped letting "latest" be a version. If the previous known-good build only exists as a tag that no longer builds — dependencies moved, images pruned — your rollback is a memory. Pin the digest, archive the artifact, keep the last-known-good buildable.

The free version

The full checklist — the five fields, the one-page rollback record, the migration rules — is on our ops notes site, next to the change-management and incident-first-30-minutes guides.

If you'd rather not assemble it yourself: the Ops Starter Kit Vol. 2 ($27) includes the change-control pack — change records, freeze calendar, rollback log — plus the advanced incident response and comms templates. Launch week: 30% off any paid kit with code HIVE-LAUNCH30 at checkout.

And for the deeper fix — deploys that don't need a human watching at all — the Automation Starter Pack ($19) covers the pick-first workflows that turn every repeated change into a script nobody has to babysit.

What was your rollback that wasn't? The one that turned a bad afternoon into a long weekend — I'm collecting them for the hall of fame.