DEV Community

Hive80-lab
Hive80-lab

Posted on

Stop Writing Runbooks Nobody Reads — Build These 5 Instead

#devops #sre #operations #productivity

I've inherited a lot of runbooks. Most of them are 40-page Word documents that nobody has opened since the day they were written. Here's the uncomfortable truth: if a runbook takes more than 90 seconds to use during an incident, it doesn't exist.

After ~100 solo incidents, here are the five that survive contact with reality — steal the structure for free.

1. The "First 10 Minutes" Card (one page, printed)

Not a document. A card. Five steps, in order, with the exact commands:

  1. Confirm from external vantage (phone on LTE, status page) — is it really down?
  2. Snapshot changes: find /etc /opt /srv -mmin -60 -type f + git log --since="1 hour ago" --oneline
  3. Stop the bleeding: rollback beats debugging when users are locked out
  4. Communicate: 90-second status update, set a 15-minute timer
  5. Stabilize: restart → rollback → df -h → free -m → clear queues

That's it. No diagrams. If step 2 doesn't tell you what changed, jump to step 3 anyway.

2. The "One Command" Runbook per service

For every critical service, a file that answers exactly three things:

# /srv/runbooks/payments.md
symptom: checkout returns 502
first command: sudo systemctl restart payments && sleep 3 && curl -fsS localhost:8080/health
if that fails: journalctl -u payments --since "15 min ago" | tail -40
escalate: [who] via [channel] after 10 minutes
Enter fullscreen mode Exit fullscreen mode

Three commands, one escalation rule. Written after the second incident with that service, not before the first.

3. The Rollback Card

The most valuable runbook is the one you use most: how to undo. Per deploy target:

  • Which artifact/commit was live an hour ago?
  • The single command to restore it
  • How to verify it's actually back (health endpoint, not vibes)

If your rollback takes more than 5 minutes, your deploy process is the real incident.

4. The Postmortem One-Pager

Six sections, one page, same day as the incident: Impact (3 lines), Timeline (state changes only), Root Cause (the change, not the person), Detection (minutes to detect), one automated check that would have caught it, max 3 secondary items.

The discipline: one automated check per incident. After a year you have 20+ checks and your incident rate falls off a cliff. That's not theory — that's compounding.

5. The "Who to Call" Card

The 3 AM failure mode isn't technical — it's not knowing who owns the payment gateway account, the domain registrar, the DNS provider. One page: service → owner → account contact → escalation path. Update it when people change teams, or it lies to you.

The anti-patterns that killed every bad runbook

  • Written for auditors, not for the person with the pager
  • Prose instead of copy-pasteable commands
  • Never updated after the incident it was written for
  • Lives in a wiki that needs 4 clicks to reach (put it in the repo, next to the service)

Want these pre-built?

I turned this into ready-made kits — fill-in-the-blank runbook templates, the checklists above, and alert scripts you can adapt in minutes: Custom Incident Runbook (done-for-you, 48h) · AI Ops Workflow Kit · Ops Mega Bundle.

Or just steal the structures above — they work either way. And my free checklist repo: https://github.com/Hive80-lab/incident-response-checklists

Top comments (0)