The most important lesson we learned is this: you need to build strong foundations of system context before you can trust any automated reasoning or analysis. If your system lacks basic visibility—like topology, deploy tracking, or incident history—it simply cannot point engineers in the right direction, no matter how smart it seems.
In our first attempt at building an incident detection system, we made some early missteps. We focused on reasoning before establishing a foundation of system context. There was no topology, deploy tracking, or incident history. As a result, the system often misidentified the cause of incidents, frequently blaming the most recently deployed service regardless of the real issue. It produced plausible-sounding explanations, but they were frequently inaccurate.
Engineers agreed with the system’s top hypothesis for only about 35% of incidents. The issue wasn’t just accuracy—it was the confidence in wrong answers, which led to costly mistakes. Within two weeks, three engineers stopped using it entirely. One person said he trusted it less than reviewing the raw trace data himself, which was disappointing since the tool was supposed to help avoid that manual work.
This type of failure isn’t unique. Many incident analysis tools don’t fail outright—they produce explanations that seem credible enough to act on, even when they’re mistaken.
These experiences led us to rethink our approach. The main issues we encountered taught us more than any early success would have.
The frame that matters
Incident response isn’t about language. It’s about finding and connecting information quickly. Engineers don’t need extra text. They need to know what changed, what’s related, and if they have seen this issue before.
The purpose of a summarization layer is to clearly explain what the retrieval system uncovers. It doesn’t generate answers on its own. Overlooking this distinction led to problems in our first version.
What the system needs to see
Everyone collects metrics, logs, and traces. That’s the basic requirement. But most teams miss a few important things.
Deploy events and config changes matter. In our incidents, something shipped in the previous two hours was involved about 60% of the time. To actually track deploys, we set up a simple integration between our CI/CD pipeline and a central event feed: every deploy job would trigger a webhook that recorded the service name, deploy time, commit hash, and engineer. We then linked this timeline directly to our incident context, making it possible to correlate metrics and errors within the right deploy window. If your system can’t see the deploy window, it will blame the wrong thing confidently. That’s how our first version ended up at 35%. Adding deploy awareness made the biggest difference: agreement with the top hypothesis jumped from about 35% to 70% with just that change.
Service ownership and the dependency graph are key. Without knowing the system’s structure, you can’t figure out the blast radius. Just saying, “This service has high latency,” isn’t helpful. But if you say, “This service has high latency, it’s upstream of your payment flow, team X owns it, and it was last deployed 40 minutes ago,” an on-call engineer can act even before fully understanding the problem.
Historical incidents and postmortems are often overlooked. Structured postmortems, which record actual causes, failed hypotheses, and documented fixes, provide real patterns to match instead of just scanning log data. Without this collected history, the tool can only summarize what’s currently visible. With six months of structured postmortems, it can act more like institutional memory.
Architecture
Six layers, clear contracts between them. Here’s the flow on a real incident:
① Normalization means using one schema, synced timestamps, and consistent service identity across every source. It’s basic infrastructure work but determines whether everything downstream is reliable. We learned this the hard way: checkout-svc in our deploy pipeline and checkout-service in our trace data were the same service. Because the system didn’t connect them, it once cleared the service actually having problems. The fix was a simple service-identity mapping table, but we didn’t find the bug until it caused trouble. Having two names for the same thing messes up every correlation above it, and you won’t notice until it’s too late. We spent two weeks on this, and it was worth it.
② Context Graph connects services to owners, dependencies, recent deploys, SLIs, alert rules, and past incidents. Without it, the reasoning layer just guesses the system’s structure. With it, a “checkout-service latency spike” becomes "checkout-service latency spike, one deploy 40 minutes ago, three downstream services affected, matches an incident from six weeks ago." The hard part isn’t building the graph but keeping it up to date. Ours changes all the time: services get renamed, ownership changes, and dependencies are added in code before the registry updates. We use a mix of automation and lightweight scheduled audits to keep the context graph current. Service registries and deployment pipelines feed updates automatically, while a regular review cycle flags anything stale or inconsistent. We treat a stale connection as a bug with a severity level, not just a chore, because a confident answer based on outdated information is worse than no answer.
③ Evidence Retrieval means when an incident happens, you only pull the relevant data window. This includes blast-radius services, unusual metrics, outlier traces, recent deploys, and the closest historical matches. Don’t send your entire log collection to the model; focused retrieval is faster and leads to better reasoning. Tuning this is important. Our first window was too narrow and missed a slow-burning root cause just outside the blast radius we defined. The scope for retrieval is something you adjust over months, not set once and forget.
④ The reasoning system should only produce structured output: an incident brief, ranked hypotheses, confidence levels, missing evidence, and next steps. If the system can’t show uncertainty, it will give false confidence, which is worse than no answer, especially in high-pressure situations.
⑤ Human Approval means the AI recommends actions, but an engineer carries them out. Auto-remediation should be the last feature you add, not the first, and only after replay testing on past incidents proves it works. It fits into the same process as human approval once you’ve built enough trust. The architecture stays the same, but the trust changes.
⑥ Learning Loop means every resolved incident feeds back information: the timeline, actual cause, failed hypotheses, and the fix. This helps the system improve over time, and I learned this lesson the hard way. Our first version of the loop made retrieval worse. We fed back raw, unstructured postmortems, and the noise made it harder to find good matches. Inconsistent write-ups led to wrong assumptions for new incidents. It only helped once we enforced a strict format for what gets written back. Bad data in the loop is worse than no loop. That change is the difference between a system that improves and one that falls apart. Most platforms skip this step, which is why they stop getting better.
Here is the format we settled on for structured postmortems:
Incident summary: brief description of what happened
Timeline: key events with timestamps
Actual cause: root cause identified
Failed hypotheses: alternatives considered and ruled out
Fix: what resolved the issue (rollback, config change, etc)
Affected systems: services, teams, or components involved
Evidence: relevant metrics, logs, traces, or screenshots
Lessons learned: what should be changed or watched in the future
Enforcing this structure meant each postmortem was immediately useful as a template for new incidents, making it easier for both humans and the system to learn from history.
An opinion I’ll defend
Structured postmortem retrieval offers more value than any advanced reasoning feature in the first six months. Many teams get this backwards, spending months on the analysis layer and treating postmortem ingestion as a secondary concern. But it’s often the tenth incident—similar to one from months ago—that’s resolved quickly because the system can reference that history. Prioritize ingestion first, then build out advanced features.
What good output looks like
Not this:
The system is experiencing elevated error rates potentially attributable to cascading infrastructure instability across multiple service layers.
This:
p95 latency up 4x since the 02:11 deploy on checkout-service. Errors concentrated on checkout → inventory calls. Two prior incidents had this signature — both connection pool exhaustion. Confidence: medium. Missing: DB wait time metrics pre/post deploy. Next: check connection pool stats on both sides of the deploy window.
It’s specific. It admits what’s missing. It tells you exactly what to do next. This earns trust not because it’s always right—it says confidence is medium and points out what’s missing—but because it guides the engineer to the next step. That is the whole point.
Rollout order
Read-only incident summarizer — watches the alert feed and incident channel, keeps a live brief. Zero operational risk. Engineers use it immediately because it eliminates the worst part of an active incident: reconstructing context from 300 Slack messages.
Evidence retrieval assistant — answers “what deployed recently,” “which services correlate,” “have we seen this error before.” This is where you start saving real minutes.
Hypothesis ranking — structured list, each entry with evidence, counter-evidence, and confidence. Still advisory. Now it shapes how engineers reason through the problem.
Action recommendations — suggested steps with blast-radius estimates. Human approves each one.
Narrow auto-remediation — reversible actions only, policy-bounded, after extensive historical validation. Last, not first.
There are two rules I’d insist on: every claim should link to a supporting artifact, and engineers must be able to flag any output as wrong or potentially harmful. To enable flagging, we added a simple feedback button directly in the incident UI. When an engineer clicks it, they can categorize the issue and send instant feedback to the system. For automated integrations, we also exposed an API endpoint so teams can programmatically flag misleading or harmful outputs. Linking claims to evidence keeps the system accountable, and the flagging feature helps spot failures before they have real consequences.
If you remember just one thing, it’s this: the order matters. Do retrieval before reasoning, context before cleverness, and ingestion before auto-remediation. We learned all of these lessons the hard way, starting with a version that blamed every problem on the latest deploy.
Top comments (0)