We had a runbook for our worst outage scenario. It was thorough, well-written, and completely useless at 3am, which I discovered the hard way when I was the one paged and holding it.
The runbook said things like "restore the database from the latest backup" and "failover to the secondary region." Reasonable steps. But it assumed I knew where the backups lived, which credential to use, what the exact failover command was, and that the secondary region was actually ready to take traffic. It assumed a calm, well-rested engineer with full context. At 3am, half-awake, with customers down and adrenaline running, I had none of those things. The document described the destination without giving me the road.
We rewrote every runbook after that incident with one rule: a runbook must be executable by the most junior person on the team, at the worst possible hour, with zero prior context. That's the bar. If a step says "restore from backup," it now has the exact command, with the real bucket path, and the exact way to verify the restore actually worked. Not "verify the service is healthy" but the specific curl command and the specific response that means healthy.
The forcing function was practice. We started running game days — deliberately breaking something in a staging environment and having someone follow the runbook to fix it, no help allowed. Every place the person got stuck, hesitated, or had to ask a question was a bug in the runbook, and we fixed it on the spot. A runbook you've never executed is a hypothesis, not a procedure.
The deeper shift was cultural. We stopped treating reliability as heroics — the one senior engineer who knows how everything works swooping in to save the day. That person is a single point of failure, and they will eventually be on a plane when it counts. Reliability is making the recovery so well-documented and well-rehearsed that heroics aren't required. Boring, tested, automated recovery beats a brilliant improvisation every time, because you can't schedule brilliance for 3am.
Write your runbooks for your most tired, least experienced self. That's who's going to be holding the pager when it actually matters.
– Sergey Shinder
Top comments (0)