Your deploy worked. The traffic graphs are green. Everyone goes to lunch.
Twenty minutes later a customer email lands: "payments are failing since your update."
Here's the uncomfortable truth: the deploy is rarely the incident. The rollback is. Most small teams can ship code fine — what they can't do is un-ship it fast, calmly, and verifiably. That gap turns a five-minute mistake into a two-hour outage.
The 4-question pre-deploy drill (do this in 10 minutes)
Before your next production deploy, answer these out loud:
1. What is the exact command to roll back — and has anyone run it recently?
"git revert and redeploy" is a theory. The command that was last tested six months ago, against a schema that has since changed, is not a rollback plan. Run it in staging today.
2. Does the database migration survive a rollback?
Code reverts in seconds. ALTER TABLE does not. If your deploy includes a migration, your rollback story is either "we run the down migration" (is it idempotent?) or "we're down until someone fixes it by hand at 2 AM." Decide before shipping.
3. How long until you know it's broken?
Mean time to rollback includes mean time to detection. If your error rate alert has a 15-minute window and a 5-minute silence period, your best-case rollback is 20 minutes — plus the rollback itself. Tighten detection for the first 30 minutes after any deploy.
4. Who calls the rollback — without asking anyone?
The worst rollback is the one waiting on a debate. Name one person with standing authority to pull the trigger. If nobody holds it, every incident starts with a discussion instead of an action.
The rollback rehearsal (30 minutes, quarterly)
Once a quarter, actually roll back a normal release in staging with the clock running. Write down the real number — not the number you wish it was. Most teams find their "5-minute rollback" is actually 35 minutes, and the gap is always the same three things: an untested command, a migration trap, and a slow detection loop.
Fix those three, and your next incident gets 30 minutes of your evening back.
I packaged the pre-deploy drill, the migration-trap checklist, and the rollback rehearsal template into the Ops Starter Kit — the documents are written to be printed and taped next to the deploy button. If you want the detection half automated (post-deploy error-rate checks wired to the drill), the Automation Starter Pack covers it. HIVE-LAUNCH30 takes 30% off the bundle this week.
What's your real rollback time — measured, not guessed? Tell me in the comments.
Top comments (0)