Runbooks die the same way every time: they're written by the person who knows the answer, formatted like a manual, and read by a person having the worst day of their month. After four handovers that failed (and one bridge call that made it worse), here are the six rules that made ours stick.
1. One screen or it's two documents
If a runbook scrolls more than one screen for the critical path, people will skim it during the incident. Keep the 2am path on one screen: symptom → check → action → verify. Everything else (background, history, architecture diagrams) goes BELOW a divider labeled "context — don't need this at 2am."
2. Start from the symptom, not the system
Nobody at 2am thinks "what could cause elevated latency in the ingestion service?" They think "checkout is down." Organize runbooks by what the on-call actually sees, not by how the codebase is structured. System-view belongs in the context section.
3. Every step has a verify
"Restart the worker" is half a step. "Restart the worker; queue depth drops below 100 within 3 minutes" is a step. Without the verify, people can't tell if the fix worked — so they escalate, and now two people are awake for one incident.
4. Name the failure modes you can't fix
Every runbook should have a short "if this doesn't work" section: the two or three dead ends that eat an hour each. "If the queue doesn't drain, do NOT restart the broker — call the vendor, that's their bug." Negative instructions save more time than positive ones.
5. Test it with a person, not a linter
The 2am test: once a month, someone who didn't write the runbook runs it against a controlled fault, alone. Every question they ask is a doc bug. Every time they have to improvise is a gap. This one ritual does more for on-call quality than any template.
6. Keep a "why" line per alert that pages
An alert that pages someone must justify itself in the runbook: what it means, why it pages, what breaks if ignored. If nobody can articulate the "why," the alert is a candidate for downgrade to ticket. Alert noise is a runbook problem before it's a monitoring problem.
Copy, don't write
We packaged the working versions of these — handover format, escalation ladder, the 2am test script, routing templates — as The Automation Starter Pack (A$19; coupon HIVE-LAUNCH30 takes 30% off the whole store).
Start with the free checklist instead if you're new: The First 30 Minutes — the incident-opening sequence small teams actually run.
Related: The on-call handover that fits on one screen · The $0 monitoring stack
Top comments (0)