Every team has one person who "just knows" how to fix the weird stuff. When they're on holiday, incidents last four times longer. The usual fix — "let's write documentation" — fails because nobody writes docs when the building is on fire.
The trick that worked for us: capture the runbook during the incident, not after. One person drives the fix; a second person (or the same person, voice-memo style) logs three fields as they go:
- Symptom in the customer's words — what made someone notice.
- Exact commands that changed state — copy-paste quality, not descriptions of commands.
- The verification step — how we knew it worked.
That's it. No blame, no architecture diagrams, no "lessons learned" essay. Fifteen lines per incident. After a quarter you have a runbook written entirely from real events, in the order things actually break, and it reads like a checklist because it is one.
Why post-hoc documentation always loses
Documents written a week later sanitize the mess, and the mess is the value. The fix that worked was step 3 before step 2 because of a stale cache — that ordering is exactly what a polished retrospective smooths away. Capture raw, polish never, and let the doc grow ugly but true.
Small teams: put the capture file in the repo next to the deploy config. Incidents are code, so the runbook should live with the code.
We package this stuff as done-for-you runbooks + 24/7 monitoring kits: 24/7 Agent Ops Kit — self-serve, instant download.
Want to try before you buy? A free starter kit lives on the same page.
Top comments (0)