Most incident reviews change nothing. Not because the team doesn't care, but because the review is built like a courtroom drama: ninety minutes, a timeline reconstructed to the second, twelve "action items" nobody owns, and a document nobody opens again until the next incident forces a search for cover.
The fix isn't more process. It's less. Fifteen minutes, a fixed template, two changes that actually ship. Here's the template, the rules that keep it at fifteen minutes, and the failure modes that quietly kill most review cultures.
Why fifteen minutes works
Three reasons, all practical:
- Attention is the scarce resource. Nobody sustains quality analysis for ninety minutes in a conference room. Fifteen minutes forces the room to spend attention on the moments that mattered instead of narrating logs at each other.
- Freshness beats forensics. Reviewed the same day, people still remember what they were thinking at 2am when they made the call. A week later you get a sanitized reconstruction that explains everything and teaches nothing.
- Cadence compounds. A review culture that handles small incidents in fifteen minutes runs reviews weekly by default, and weekly reviews make big incidents rarer. Ceremony-heavy cultures review rarely, so every review carries the weight of a season of problems — and drowns.
The template
Seven sections, filled in live, in the meeting. Not after. Not "I'll draft it and send it around." In the meeting or it doesn't happen.
1. What happened, in one sentence. Plain language a new hire could understand. If the sentence needs acronyms, rewrite it. "Checkout failed for 20 minutes because a disk on the job runner filled up" — that's the bar.
2. Blast radius. Who saw it, for how long, and what it cost. Users, revenue, jobs delayed, support tickets opened. One line each. If you can't quantify it, say so — "unknown, roughly" is acceptable; a confident fake number is not.
3. The three to five decision points. This is the heart of the review, and it is not a log replay. You're looking for the moments where a different decision would have changed the outcome. "We tuned the 80% disk alert to fire only at 95% last month" is a decision point. "The disk grew at 3:41" is not. List only decision points. Everything else is decoration.
4. What the system got right. Every review should name what worked: the queue that retried, the health check that caught it, the on-call engineer who had the right runbook. This isn't feel-good filler. It tells you which strengths you can lean on during the next incident, and it keeps the room out of pure-fault framing, which produces worse analysis, not better.
5. Two fixes. Exactly two. One automated guard (a check, a limit, a script that prevents or catches the condition) and one visibility or process change (a dashboard panel, an alert route, a runbook section). Each fix gets an owner, a date, and — critically — a location where it will visibly land: a PR, a config change, a runbook diff. "We should be more careful" is not a fix. A fix is a thing a stranger could verify.
6. The missed signal. What could you have known earlier, and why didn't you see it? Usually the answer is that the signal existed but was muted, misrouted, or drowned in noise. This section is where alert-quality debt gets named out loud instead of absorbed silently.
7. Suppression check. Will this exact alert fire again tomorrow under the same conditions? If yes, you either fix the alert now or deliberately downgrade it with a comment explaining why. Reviews that skip this step manufacture alert fatigue one incident at a time.
The rules that keep it at fifteen minutes
- A visible timer. At minute fifteen you stop. Unfinished discussion becomes a ticket with the relevant people on it. The timer is the whole trick — it converts "we should wrap up" from a social problem into a mechanical one.
- A blame-vocabulary ban. Replace "why did you do that" with "what made that reasonable at the time." This isn't politeness; it's precision. People hide decision context when they expect punishment, and hidden context produces wrong fixes.
- Two action items, maximum. More than two means you're collecting debt instead of paying it. If the room lists ten candidate fixes, pick the two with the best prevention-per-effort and log the rest as candidates, not commitments.
- Every fix gets a verification step. "How will we know it worked?" A dashboard that changes color, an alert that fires in a drill, a test in CI. A fix without a verification step is a wish.
- Written live in the repo. The template lives next to the code; the filled review gets committed before the meeting ends. Reviews that live in a doc-tool draft die in a doc-tool draft.
A worked example
Incident: the job runner's disk filled up; the queue stalled for 22 minutes; nothing was lost but two reporting jobs were late.
- One-liner: disk on the job runner filled, queue stalled 22 minutes, no data loss.
- Blast radius: two daily reports ~20 min late; no user-facing errors; no revenue impact.
- Decision points: (a) the 80% disk alert was tuned to 95% during last month's noise purge — the 30-minute warning window we thought we had didn't exist; (b) log rotation on that box was never enabled after the runner was migrated.
- What worked: the queue retried cleanly; the runner's health check marked it unhealthy and stopped new jobs from landing on a full disk.
- Two fixes: an automated guard — a disk-pressure webhook that drains the runner and alerts at 80% with a documented 30-minute runway; a visibility change — a runbook section with the exact drain command and the "confirm queue resumed" check.
- Missed signal: the 80% alert was silenced in a bulk-mute during a migration and never re-enabled.
- Suppression check: re-enabled the alert route, verified the webhook fires in a staged drill.
Fifteen minutes, two concrete changes, both verifiable. That's a good review.
How the changes actually stick
- Attach each fix to an existing ceremony. A change that ships in this week's release train is real. A change that "gets scheduled" evaporates. If it can't ride the next deploy, it escalates to the next one loudly.
- Track one metric: repeat-incident rate. If the same contributing factor shows up twice in ninety days, the review failed, not the engineer. This reframes accountability toward the process, which is where it belongs.
- Rotate the facilitator and the scribe. Reviews become training for juniors instead of a tax on seniors. It also exposes — gently and early — the people who run reviews like interrogations.
The failure modes to watch for
- Review theater: beautiful diagrams, no owner, no date. If nothing lands in the repo, it didn't happen.
- Fix inflation: twelve action items, zero shipped. Two, shipped, beats twelve, forgotten.
- Timeline forensics: reconstructing every second to satisfy the document instead of isolating the decision points.
- Skipping small incidents: small incidents carry the same patterns at a fraction of the price. Skipping them is choosing to learn the lesson during the expensive one.
Cadence
Fifteen minutes for every Sev2+ incident, same day. Weekly, a thirty-minute pattern scan across recent reviews: same contributing factor twice in ninety days means a systemic fix is overdue. That's it — no quarterly postmortem festival, no review board. The cadence is the system.
Operating around the clock?
Incident reviews are one leg of running something 24/7. The other legs are runbooks, monitors, and escalation paths that already exist before the pager fires. We packaged the templates and monitor configs we run into one kit: Agent Ops 24/7. The full set of ops kits and playbooks lives on the HIVE80Lab storefront.
Top comments (0)