Two rollbacks, same quarter. One took 4 minutes and nobody outside the team noticed. The other took 6 hours and made the local news. The difference wasn't luck — it was three decisions we made months apart.
Rollback #1: the 4-minute one
Standard Tuesday deploy. Error rate ticked from 0.1% to 2.3% within 90 seconds. We ran one command — rollout undo — and the previous version was serving traffic before the first user complaint hit support. Total customer impact: zero.
What made this possible: the previous version was always one command away. Immutable artifacts, versioned config, and a deploy pipeline that treats rollback as a first-class operation, not an emergency procedure invented during the outage.
Rollback #2: the 6-hour one
A schema migration shipped inside the release. rollout undo brought the old code back, but the old code couldn't read the new schema. The rollback command was a lie — it undid the app, not the release.
We spent 6 hours writing a manual migration back, arguing about data loss, and testing queries against production backups. The postmortem took one line to explain: we treated a migration as reversible because the deploy tool said so.
The three decisions
Migrations get their own rollback scripts, written at the same time as the forward migration, and tested against a copy of prod data before the deploy window. If writing the down-migration is too painful, that's a signal the migration is too big.
Rollback is rehearsed, not improvised. Once a month we roll back a deploy to staging for no reason except to prove the path works. The muscle memory matters — at 2 AM you don't want to be reading docs for the first time.
Error rate triggers rollback automatically. We set a threshold (5xx rate > 1% for 2 minutes → auto-rollback). Every debate about "wait, maybe it'll settle" is a debate you're having while users suffer. Automation wins that argument every time.
The template we use
Our rollback runbook (decision tree, auto-rollback config, migration down-script template, rehearsal checklist) is part of the Automation Starter Pack ($19) and ships in the Ops Mega Bundle with the monitoring layer that catches the error-rate spike in the first place. One-time purchase, plain scripts, no subscription.
Question for the comments: what's the longest a rollback has taken you, and what was the root cause? Collecting war stories.
Running a stack like this? Our Ops Starter Kit ships the runbooks, checklists, and monitoring patterns behind these posts — grab it and cut your next incident short.
Top comments (0)