Every engineering team has outages. The teams that improve fastest are not the ones that have the fewest incidents — they are the ones that extract the most learning from each one. A disciplined, blameless postmortem process is the mechanism that turns a painful production failure into a lasting systemic improvement.
What Is an Incident Postmortem?
An incident postmortem — also called a post-incident review (PIR) or after-action review — is a structured analysis conducted after a production incident has been resolved. The goal is not to assign blame. The goal is to understand what happened, why it happened, and what systemic changes will prevent it from happening again.
The blameless postmortem philosophy was pioneered by John Allspaw and Paul Hammond at Etsy and later formalized in Google's Site Reliability Engineering book. The core insight: engineers always act with the best intentions given the information and tools available at the time. Blaming individuals destroys psychological safety — and when safety goes, future incidents get hidden rather than learned from.
"The goal is to understand the contributing causes of the incident in order to prevent similar incidents from occurring in the future, not to punish the people involved." — Google SRE Book
When to Run a Postmortem
Always run a postmortem when:
- The incident lasted more than 30 minutes
- Multiple systems or teams were affected
- Customer data was at risk or affected
- A manual rollback was required
- P1 or P2 severity per your framework
Consider running one when:
- A bug reached production that tests should have caught
- Detection was delayed for an already-active issue
- The same class of issue has occurred before
💡 Near-miss postmortems are some of the most valuable. Same learning opportunity, zero customer impact.
The Right Timing
Hold the postmortem 24–72 hours after resolution.
| Window | Verdict | Why |
|---|---|---|
| < 4 hours | ❌ Too soon | Team still decompressing, surface-level analysis |
| 24–72 hours | ✅ Sweet spot | Memories fresh, team recovered, logs reviewed |
| 3–5 days | ⚠️ Acceptable | For major multi-day incidents |
| > 1 week | ❌ Too late | Details forgotten, actions won't happen |
Who Runs It
Facilitator: Someone NOT directly involved in the response. An EM, senior engineer from another team, or dedicated facilitator. Their job: ask "what" and "why," never "who."
Required attendees: Everyone in the incident response + on-call engineer + EM + PM if customers were notified. Keep it under 10.
The Full Template
Fill this in before the meeting, not during it. The meeting is for discussing and challenging the draft.
## INCIDENT POSTMORTEM
Incident ID: INC-2026-XXXX
Severity: P1 / P2 / P3
Date: YYYY-MM-DD
Duration: X hours Y minutes
Postmortem Date: YYYY-MM-DD
Facilitator: @name
Attendees: @name1, @name2, @name3
### 1. EXECUTIVE SUMMARY
2–4 sentences: what happened, customer impact, how resolved.
### 2. CUSTOMER IMPACT
- Users affected: ~X,XXX
- Revenue impact: ~₹X failed transactions
- SLA breached: Yes / No
- External comms: Status page updated HH:MM
### 3. TIMELINE (UTC)
HH:MM First error in logs (not yet detected)
HH:MM Alert fires — on-call paged
HH:MM On-call begins investigation
HH:MM Incident declared, team assembled
HH:MM Root cause identified
HH:MM Fix deployed / rollback initiated
HH:MM Service fully restored
### 4. ROOT CAUSE
Be precise. Not "database issue" — but exactly what failed,
why, and under what conditions.
### 5. CONTRIBUTING FACTORS
Every factor that made this possible or worse. Ask "why" 5x.
- No load test for the PR introducing the slow query
- Connection pool exhaustion had no dedicated alert
- DB incident runbook was out of date
- Deploy had no post-deploy monitoring window
### 6. DETECTION
- How detected: Alert / Customer report / Internal discovery
- Detection lag: XX minutes from first error to page
- Could be faster? Yes — specific alert would have fired 12 min earlier
### 7. RESOLUTION
What specifically resolved it? Was it temporary or permanent?
### 8. ACTION ITEMS
| Action | Owner | Due | Ticket |
|--------------------------------------------|---------|------------|---------|
| Add DB connection pool saturation alert | @alice | 2026-07-23 | INC-101 |
| Add load test step for DB-heavy PRs | @bob | 2026-07-30 | INC-102 |
| Update DB incident runbook | @carol | 2026-07-23 | INC-103 |
| Post-deploy monitoring: 15-min SOP | @dave | 2026-07-28 | INC-104 |
### 9. LESSONS LEARNED
What went well? What surprised you? What would you do differently?
Running the 60-Minute Meeting
- Opening (5 min) — Set blameless ground rules. "We are here to improve the system, not evaluate the people."
- Timeline walkthrough (15 min) — Walk through chronologically. Correct errors, fill in gaps. Get everyone on the same page.
- Contributing factors (20 min) — This is the most important section. Use the 5 Whys. Keep asking until you reach something systemic — a process gap, tooling gap, knowledge gap. Not a person.
- Action items (15 min) — Owner + due date + ticket number for every item. If you can't assign an owner in the room, the item won't get done.
- Lessons learned (5 min) — What went well? Good incident response behaviors deserve positive reinforcement too.
Red flags to watch for
- "The engineer shouldn't have done X" → redirect to "what made that possible?"
- "We just need to be more careful" → not an action item, that's wishful thinking
- All action items assigned to the same person → SPOF, not a fix
Writing Good Action Items
This is where most postmortems fail. Vague, unowned, untracked items are the #1 reason postmortems don't prevent the next incident.
| ❌ Bad | ✅ Good |
|---|---|
Improve monitoring |
Add P1 alert on checkout_service 5xx rate > 2% for 3 min — @alice — due 2026-07-23 |
Be more careful with DB migrations |
Add mandatory staging dry-run to migration PR checklist — @bob — due 2026-07-30 |
Write runbook for DB issues |
Update /runbooks/db-connection-pool.md with exhaustion recovery steps — @carol — due 2026-07-25 |
Track action items in your project management tool — not in the postmortem doc. They need to live alongside regular work where they'll actually get prioritized and tracked.
Common Mistakes
- Single root cause thinking — Most incidents have multiple contributing factors. Stopping at the trigger misses the systemic gaps.
- Postmortems only for big incidents — Smaller, frequent incidents have some of the most actionable learning.
- Untracked action items — Every item needs a ticket, an owner, a due date. In the same system as regular work.
- Not sharing broadly — Other teams can learn from your incident. Don't silo the knowledge.
- Skipping when busy — High-incident periods are exactly when you need to learn fastest. Postmortems are not optional when things are hard.
What's your team's postmortem process? Drop a comment — always curious how different teams handle this. 👇
Originally published at projiq.app
Top comments (0)