Most outages are not exotic. They are the same five failures wearing different clothes: disk full, cert expired, the one script nobody owns, the dependency that moved, the human who was on holiday. A runbook does not prevent these — it makes them boring.
Here is the uncomfortable part: most small teams do not have runbooks. They have a Slack thread from eight months ago and one person who "just knows". That person is a single point of failure with a commute.
The fix is smaller than you think. You do not need a wiki. You need one page per recurring failure, written while calm, so future-you can execute it while panicked.
The 15-minute runbook format
For each failure mode, write exactly five things:
- Symptom — what the on-call human actually sees (the alert text, not the theory).
- Diagnose — the two commands that confirm it. Copy-pasteable, not "check the logs".
- Fix — the steps, in order, including the rollback if the fix makes it worse.
- Escalate — who to page if step 3 fails, and after how many minutes.
- Verify — how you prove it is actually fixed, not just quiet.
That is it. Fifteen minutes per failure mode. Do your top five and you have covered the majority of real incidents.
Why verification is the step everyone skips
A runbook that was never executed is a rumor, same as a backup that was never restored. Once a quarter, run one runbook start-to-finish against a staging copy. Time it. The number you get is your real recovery time — put that in your status page promises, not a vendor brochure number.
If the runbook says "restart the service" but the restart needs a password that lives in one person's head, you just found your next single point of failure for free. That is the drill paying rent.
Make it findable at 2 AM
A runbook stored in a doc nobody can search at 2 AM does not exist. Put the index where the alerts point: alert text should link straight to the runbook page. If your alert says "high error rate" and the runbook is called "misc-notes-final-v2", you have a documentation problem, not an incident problem.
The kit
We packaged this as a full ops kit — runbook templates, an incident timeline sheet, the offline fallback checklist, and a monitoring baseline for teams with no on-call rotation:
- Agent Ops 24/7 — monitoring & runbook kit for small teams ($19)
- Ops Mega Bundle — every playbook, template and checklist in one purchase ($249)
More free playbooks at Hive80Lab.
Top comments (0)