It's 3:07 AM. PagerDuty fires. The on-call engineer opens the wiki. The runbook was last edited 14 months ago by someone who left the company.
This is not a documentation problem. It's a placement problem: your runbook lives where people browse, not where people burn.
The rule: a runbook must live where the page fires
Link runbooks directly to the alert. Every monitor that can page a human carries a runbook_url annotation. When it fires, the notification includes the link. Zero navigation. If an alert has no runbook URL, it cannot page — it becomes a ticket until someone writes one.
Three tiers of documentation (and only one matters at 3 AM)
- Tier 1 — the card. 5 lines. Symptom, first check, quick fix, escalation path. This is what lives in the alert payload. Written under duress, tested under duress.
- Tier 2 — the wiki page. Full background, architecture, history. Good for postmortems and onboarding. Nobody reads it mid-incident.
- Tier 3 — the deep doc. Design decisions, trade-offs. Read quarterly, never at night.
Most teams only have tier 2 and wonder why 3 AM goes badly.
The 90-second drill that keeps runbooks alive
Once per sprint, one engineer picks a random alert, reads only its runbook card, and simulates the first action. If the card is wrong or stale, they fix it in the same PR (10 minutes). Cards decay; the drill resets them. Teams that run this keep >90% of cards accurate; teams that don't, decay to the 14-months-stale wiki.
What to do Monday
- List your top 10 paging alerts by page count.
- Write a tier-1 card for each: symptom / first check / quick fix / escalation.
- Wire the card into the alert payload. Remove the wiki link — nobody was reading it at night anyway.
If you want the card templates, alert-annotation wiring, and the drill schedule pre-built, I packaged them in the Agent Ops 24/7 kit. More field notes at dev.to/hive80lab.
Top comments (0)