DEV Community: Kushagra

Postmortems that don't suck: the template we actually use (steal it)

Kushagra — Mon, 17 Aug 2026 07:52:28 +0000

I've read a lot of postmortems. Most of them are written for an audience that doesn't exist.

They're five pages long. They have an "Executive Summary" for executives who will never open the doc. They have a root-cause section that stops at "human error" (never the root, always an excuse to stop digging). And they end with action items like "improve monitoring" — assigned to nobody, due never, closed by nobody.

Then the same incident happens eight months later, and someone says "wait, didn't we write this up last time?" and nobody can find the doc.

A postmortem has exactly one job: make the next incident shorter or prevent it entirely. Everything in the document either serves that job or is theater. Here's the format we landed on after writing plenty of theatrical ones — it fits on one page, takes under an hour, and people actually read it. Template is at the bottom; steal it.

The three failure modes that kill postmortems

1. They're written to assign blame while claiming not to. Everyone says "blameless," then writes a timeline where one person's name appears eleven times. Here's the practical test of blamelessness: replace every name in the doc with a role ("the on-call engineer," "the reviewer"). If the story stops making sense, your postmortem was about a person, not a system. Systems can be fixed; shaming a person just teaches everyone to hide the next mistake — and hidden mistakes are how small incidents become big ones.

2. They're written too late. Two weeks after the incident, memories have quietly rewritten themselves into a cleaner story than what happened. The messy truth — the ten minutes spent debugging the wrong service, the alert everyone swiped away — is exactly the valuable part, and it evaporates first. Draft the timeline within 48 hours, even if the meeting happens later.

3. Nobody can reconstruct what actually happened. This one's structural. The incident lived across your alerting tool, an error tracker, three chat threads, a DM where the actual fix was discussed, and a deploy log. Assembling a truthful timeline means archaeology across five tools, so people skip it and write vibes instead. (This scattered-evidence problem is half of why I'm building triage that catches on-call alerts in one queue — but you don't need tooling to start: just make "paste timestamps into the doc as things happen" an on-call habit. Future-you, doing archaeology at 5 PM on a Friday, will be grateful.)

What actually belongs in the doc

A timeline with timestamps, including the embarrassing parts

Not a narrative — a table. When did the issue start (not when was it detected — the gap between those two numbers is one of your most important findings). When did the first human notice. What did they try. What didn't work.

The wrong turns are the whole point. "We spent 25 minutes restarting the API before realizing it was the database" is not embarrassing filler — it's the single most reusable finding in the doc, because the next person will make the same wrong turn unless you write it down.

"Why did our safeguards not catch this?" — the only root-cause question that matters

Skip the Five Whys ritual if it doesn't suit you, but do not skip this question. Every incident that reaches production walked past every defense you have: tests, review, staging, alerts, rate limits. The bug is rarely the interesting part. The interesting part is the hole in the net.

Why didn't tests catch it? (No coverage there? Coverage exists but the case is untestable?)
Why didn't review catch it? (Diff too big? Reviewer had no context? Rubber-stamp culture?)
Why didn't alerts catch it before users did? (This one hurts the most and teaches the most.)

Note the shape of these questions: they're all about the system. That's what blameless actually means — not "no accountability," but "the system failed, and the system is what we can change."

Three action items, maximum

This is the most opinionated rule in our format, and the one I'll defend hardest.

A postmortem with twelve action items produces zero completed action items. Twelve items means nobody prioritized, so the backlog will — which means none of them ship. Force the ranking: what are the three changes with the best ratio of "prevents recurrence" to "effort"? Each one gets a single named owner (a person, not a team — a team owner is no owner) and a date. Everything else you thought of goes in a "considered but not doing" list, with the reason. That list is quietly the most honest section of the doc, and it saves the next incident's responders from re-litigating ideas you already rejected.

And track them. An action item that isn't checked four weeks later is a decision to accept the risk — fine, but make it a decision, not a drift.

A "how will we detect this next time?" line

One sentence. If the answer is "same way — a customer will tell us," say so explicitly. Writing that sentence down is uncomfortable in exactly the way that gets a monitoring gap actually fixed.

The template

Copy this into your wiki. Ruthlessly delete any section you're tempted to add back — length is the enemy.

# [YYYY-MM-DD] Short, searchable title (name the system + symptom)

**Impact:** Who/what was affected, for how long, how badly.
One sentence. Numbers if you have them.

**Detection gap:** Issue started HH:MM · detected HH:MM · by [alert / customer / luck]

## Timeline (UTC)
| Time  | What happened |
|-------|---------------|
| 09:12 | Deploy #482 ships |
| 09:14 | Error rate on auth-svc rises (no alert fires) |
| 09:31 | First customer report |
| 09:40 | On-call restarts API (wrong tree — 25 min lost) |
| 10:05 | DB connection pool identified as cause |
| 10:11 | Mitigated (rollback) |

## Why our safeguards missed it
- **Tests:** …
- **Review:** …
- **Alerts/monitoring:** …

## Root cause
2–4 sentences. If it ends at "human error," keep digging —
what made the error easy to make and hard to catch?

## Action items (max 3)
| # | Change | Owner (person) | Due | Done? |
|---|--------|----------------|-----|-------|
| 1 |        |                |     |       |

## Considered but not doing
- [Idea] — [reason we're accepting this risk]

## Next time, we'll detect this via
One sentence. Be honest if the answer is "we won't."

Anti-patterns to delete on sight

"Human error" as root cause. It's where the analysis stops, never where the truth is.
"Improve monitoring" as an action item. Which monitor? On what threshold? Owned by whom? Vague items are unfalsifiable, and unfalsifiable items are never done.
The Word-doc graveyard. If postmortems aren't searchable from where engineers already work, they might as well not exist. The doc's title should contain the system name and the symptom, because that's what the next responder will frantically search for at 2 AM. ("We wrote this up last time" is only useful if last time is findable — the same reason duplicate tickets are a search problem before they're an AI problem.)
Postmortems only for big incidents. The 20-minute near-miss teaches the same lesson as the 4-hour outage, at 5% of the cost. Write the small ones; they're cheaper tuition.

The uncomfortable summary

If your postmortems are long, late, blameful, and unfindable, people will treat them as punishment paperwork — and you'll get exactly the quality of analysis that punishment paperwork deserves. Make them one page, fast, systemic, and searchable, and they become the cheapest reliability investment you can make.

What's the best (or most cursed) postmortem practice you've seen at your company? Genuinely collecting these — the "considered but not doing" section above was stolen from a comment on a previous post.

Your Engineering Team's Biggest Bottleneck is Slack

Kushagra — Sun, 02 Aug 2026 18:22:59 +0000

For most engineering teams, this is a painfully familiar story:

A PagerDuty alert fires at 2 AM.
The on-call engineer sees it, rolls over, and posts the Sentry link in the #dev-urgent Slack channel.
They @mention the original author of the code, the team lead, and maybe one other senior engineer for good measure.
They go back to sleep, assuming the problem is now "handled."

The next morning, the team wakes up to a sea of notifications, but zero ownership. The error is still there, nobody has touched it, and three different people have now created three different Jira tickets for the same underlying issue.

This isn't a process. It's a cry for help disguised as a workflow.

The Hidden Cost of Chat-Driven Triage

Using Slack or Teams as your de-facto incident management tool feels free, but it's one of the most expensive decisions an engineering-led company can make. The cost isn't in dollars; it's in lost focus, duplicated work, and engineer frustration.

No Clear Ownership : An @mention is not an assignment. When you notify everyone, you notify no one. The responsibility is diffused across the entire group, leading to bystander apathy.
No Durable Record: Slack is a river. Critical context, logs, and screenshots get washed downstream within hours. When the same issue reappears next month, you're starting the investigation from scratch.
No Deduplication : Chat has no concept of semantics. The same database error can trigger 10 different alerts from 5 different tools, and each one will get its own thread, creating a storm of noise that hides the real signal.
Constant Interruption : Every notification pulls engineers out of deep work to assess the same problem. You're paying for your entire team's context-switching, not just the one person who will eventually fix the issue.

Moving from Chaos to Clarity

The solution isn't to ban Slack. It's to treat incident response with the same seriousness as you treat your production code. It needs a clear, structured, and automated workflow.

At Mecverse, we live this problem, and it's why we built FlowTux. Our internal process, powered by our own tool, looks like this:

A Single Point of Ingestion : All signals—from chat, issue trackers, observability tools, and customer portals—feed into one place.
AI-Powered Triage : Before a human is ever notified, an AI layer categorizes the issue, sets a priority, and, most importantly, blocks any semantic duplicates. That storm of 50 alerts for one database issue becomes a single, actionable ticket.
Code-Grounded Context : The ticket arrives with the stack trace already mapped to the exact files in our codebase that are responsible. The investigation is already 90% done.
Automated Resolution (with Guardrails) : For a known set of problems, a sandboxed agent can run a pre-approved command (like clearing a cache or restarting a pod) and close the ticket automatically.

This isn't about replacing engineers. It's about protecting them. It's about automating the tedious, soul-crushing parts of triage so they can focus on what they do best: building and solving complex problems.

If your incident response plan starts with an @mention, it's time for an upgrade.

How do you dedupe support tickets that don't share any words? Here's our messy attempt.

Kushagra — Tue, 07 Jul 2026 09:58:09 +0000

We build an internal helpdesk, and I want to talk through a problem we only partly solved — because I suspect a lot of you have hit it too, and I'd genuinely like to hear how you handled it.

The most requested thing from our users was never "better ticket forms." It was "please make the duplicates stop."

Here's the shape of it. A deploy goes slightly wrong at a 40-person company. Within ten minutes you have:

a handful of chat messages: "login is broken", "can't get into dashboard???", "deploy looks weird"
several error-tracker events (whatever you run — Sentry, Rollbar, an APM): TokenExpiredError ×2, a 401 spike on /api/auth, a 5xx spike on auth-svc
a couple of emails to IT: "access token expired", "need login reset"

Nine items across three channels. One root cause: token rotation broke in that deploy. Whoever's on rotation spends the morning proving that, instead of fixing anything.

We wanted to automate the recognition step — "these are the same thing" — not the fixing step. This is the honest version: what we tried, the small thing we actually shipped, and the parts we haven't cracked. If you've built something similar, I'd love to be told what we got wrong.

Attempt 1: rules and keywords (broke immediately)

The obvious first cut: normalize ticket text, match on keywords and categories, merge on high overlap.

It fails on the example above, and it fails structurally: "login is broken" and TokenExpiredError share zero tokens. The human on rotation isn't string-matching — they know a deploy just happened, they know what auth-svc does, they've seen this failure shape before. Rules encode none of that.

Rule systems also rot. Every incident teaches you a new synonym for "it's down," and six months in you own a regex museum nobody wants to touch. Maybe you've kept one of these healthy long-term — if so I'd honestly like to know how.

Attempt 2: embed everything, cluster by similarity (the one we didn't ship)

The tempting next move: embed ticket text, cluster on cosine similarity, call anything close enough a duplicate.

We already use embeddings elsewhere in the product — for retrieving relevant code context from a customer's repo, not for tickets. When we thought about pointing the same machinery at ticket dedup, two things gave us pause:

It can't cross the complaint/cause gap. "Can't get into dashboard???" and TokenExpiredError at auth.rotate_credentials aren't semantically similar — they're related only through a fact that lives outside both tickets (the deploy). Embedding text you don't have doesn't help.
Pure text similarity produces confident false merges. "Dashboard is slow" and "dashboard is broken" embed close together and often have unrelated causes. And a false merge is the expensive failure: a missed duplicate wastes a few minutes; a wrong merge buries a real incident inside an unrelated one.

So we didn't ship the vector-clustering pipeline for tickets. It felt like a lot of moving parts to end up with worse precision on exactly the merges we most wanted to get right. Possible we were wrong to skip it — if you've made embedding-based dedup work well, I'd love to hear what made the difference.

Attempt 3: what we actually shipped — one strict Tux AI call

The version in production is deliberately small. When a new ticket arrives, we make a single Tux AI call (our own triage model, tuned low-temperature for consistency) that gets:

the new message text
a small candidate list of existing open tickets (title / description / status)
optionally, code context to disambiguate which part of the app is meant

The prompt is short and strict — the load-bearing instruction is roughly "only mark this a duplicate when it's genuinely the same underlying issue; be strict." The model returns a duplicate flag, the matched ticket, and a confidence score. That's the whole dedup engine. No embeddings, no clustering, no vector store for tickets.

Two choices are doing most of the work:

The candidate set is tiny on purpose. We don't ask "is this a duplicate of anything, ever?" We ask "is this a duplicate of these few open tickets?" — the reporter's own open tickets on the chat intake path, or a short list of recent open tickets from the same source on the monitoring path. A small, same-source candidate list is what keeps a fast model accurate. It's also our biggest limitation — more below.

We tuned the threshold toward precision. Duplicate handling only fires on high confidence. Below that, the ticket just gets created normally. We'd rather miss a duplicate than merge two things that shouldn't be.

What "duplicate" does depends on the channel, and here's a rough edge I'm not proud of:

Chat intake: if the new message looks like a duplicate, we route the reporter to update the existing ticket instead of opening a new one. Non-destructive, reversible.
Monitoring intake (error trackers, alerting): if a new event matches an existing open ticket, we drop the new event silently — no comment, no link, no "merged because…" record. It just doesn't get created.

That silent drop is the part that keeps me up a little, and it leads into the honest limitations.

What we haven't cracked (and would take advice on)

I'll be blunt, because dev.to comments are unforgiving and I'd rather you trust the parts that are true:

No stored merge reason. The confidence and matched-ticket come back, route the request, and then get discarded. We don't persist a "merged because all reference auth failures after deploy #482" string yet. We want to — an auditable, reversible merge record is the next thing on the list — but it doesn't exist today.
No cross-channel dedup. A chat complaint, a monitoring event, and an email about the same outage are compared only within their own channel's candidate list. The nine-item example that opens this post? We'd catch some same-channel duplicates and miss the cross-channel links. Unifying those channels into one queue is the direction we're building toward, but honest dedup across them is still open.
The silent monitoring drop. No audit trail on a suppressed event means when the model is wrong, nobody can see why. First thing we're fixing: link the dropped event to the ticket it matched and store the reasoning, so a bad drop is debuggable.
Slow-burn duplicates. Two tickets a week apart about the same flaky config rarely connect — not in each other's small, recent candidate window.
The first ticket of an incident. By definition there's nothing to match it against, so confidence is worst exactly when the incident is quietest. The system gets more useful as the incident gets louder, which is backwards from what you'd want.

The thing we think unlocks the rest — and are building now

The gap under all of these is that the tickets don't carry their own cause. The human on rotation solves it by knowing the state of the world: auth-svc deployed at 10:02, TokenExpiredError started 10:04, the dashboard depends on auth, therefore these are one incident.

So the piece we're actively working on is service-topology / blast-radius reasoning — a dependency map plus deploy timing, so "auth-svc is failing" can explain "dashboard logins are failing" even when the tickets share no words. If it works, cross-channel dedup and the "first ticket of an incident" problem both get easier: you're correlating on shared cause, not shared text.

I'm genuinely unsure of the right shape here, and this is the part I'd most like a reality check on:

How do you get a trustworthy service dependency graph without asking every team to hand-maintain one that goes stale in a month?
Deploy-time correlation feels powerful but noisy — how do you keep "something shipped recently" from swallowing every unrelated ticket in the window?
For those of you running incident tooling: does topology actually pull its weight, or is it the feature that demos well and rots in prod?

Takeaways so far

Deduplication is a "shared cause" problem wearing a "text similarity" costume. The cause usually isn't written in the tickets.
A small, same-source candidate set beats a clever global search for keeping a fast model accurate — at least until you have real cause signals to correlate on.
Tune for precision; false merges are the expensive failure. The costs aren't symmetric, so the threshold shouldn't be.
If your dedup path is destructive, make it auditable and reversible before you make it smart. A silent correct drop and a silent wrong drop look identical to the humans who have to trust the system.

If you've built ticket dedup, incident correlation, or topology-aware triage — especially the topology part — I'd really like to hear what worked and what you'd never do again. That's mostly why I wrote this.

I work on FlowTux, where this dedup path runs across chat, email, and monitoring intake. We're building toward one deduplicated inbox across your alerting, error tracking, and code — the topology piece above is the current frontier. Happy to go deeper in the comments, including the parts that still embarrass us.