Incident Deep Dip

Anand Kumar — Mon, 28 Sep 2026 19:58:45 +0000

I asked Hindsight what keeps breaking — it named the deploy

By Anand

Most incident tools are reactive. Something breaks, you investigate, you fix it, you write a postmortem nobody reads, and then the same class of thing breaks again three weeks later. I got tired of that loop, so when I built IncidentDeepDip I gave it one job beyond responding to incidents: notice what keeps happening. The moment the agent had durable memory of past incidents, it could answer a question no single incident ever can — what is our recurring failure, and what triggers it? The answer, in our case, was blunt: the deploy.

This is a write-up about turning incident memory into pattern discovery, using Hindsight as the memory layer and a small amount of aggregation on top.

From recall to patterns

The core agent already used Hindsight to recall incidents similar to a new one — that's the reactive path. But once incidents live in durable memory as structured records, each carrying a service, a root-cause category, a severity, and whether a deployment preceded it, you can stop asking "what's like this one?" and start asking "what's the shape of everything we've seen?"

The pattern discovery is deliberately simple — it groups incidents by root-cause category and measures how many followed a deployment:

for category, incs in by_category.items():
    count = len(incs)
    deploy_related = sum(1 for i in incs if i.get("deployment"))
    deploy_pct = round(100 * deploy_related / count)

    if deploy_pct >= 60:
        insight = (f"{deploy_pct}% of '{category}' incidents occurred shortly "
                   f"after a deployment — strongly deployment-correlated.")

No machine learning, no clustering model. Just honest counting over memory that Hindsight makes durable and queryable. The intelligence isn't in the algorithm; it's in the fact that the incidents are remembered at all, in a structured form, instead of evaporating into closed tickets. That's the argument Vectorize makes about agent memory — memory is the substrate that makes higher-order reasoning possible — and pattern discovery is a concrete example of it.

What it found

When I ran this over our incident history, the output wasn't subtle:

63% of all incidents followed a deployment. Not a hunch — a computed share across every stored incident.
One category dominated: Kafka consumer rebalance, nine occurrences, 100% deployment-correlated. Every single time that class of outage happened, a deploy had just gone out.
The rest of the pattern list ranked the next-most-common failure families and their own deploy correlation, so I could see the whole landscape at once.

I want to be precise here: I did not type "63%" anywhere. It's derived from the stored incidents at request time. If we resolve more incidents and retain them, the number updates itself. The tool's opinion about what keeps breaking is always a reflection of what actually broke — which is exactly the property you want and almost never get from a static runbook.

The before/after that made it click

The reactive agent already showed a strong before/after: with memory off, a new payment incident gets a generic "could be a few things" and a confidence around 35%; with Hindsight memory on, it recalls three near-identical past incidents, names the consumer-rebalance root cause, and confidence rises to about 85%.

Pattern discovery extends that from a single incident to the whole system. Before, "we seem to have a lot of Kafka issues" was a feeling someone voiced in a retro. After, it's a ranked, quantified pattern with a deploy-correlation percentage attached — and it points at a specific, testable hypothesis: our rolling deployments are triggering consumer-group rebalances. That's not a vibe you argue about. That's a checklist item you add before the next release.

The honest limitation

I'll be direct about what this is and isn't. Correlation is not causation, and my deploy-correlation metric is exactly that — a correlation. A category being "100% deployment-correlated" means every stored incident of that type had a recent deploy, not that the deploy provably caused it. The tool is good at pointing a human at the right hypothesis fast; it is not a root-cause oracle.

I also had to resist the urge to over-engineer this. My first instinct was to reach for clustering and embeddings to "discover" categories automatically. I stopped, because the honest version — group by the category we already record, count deploy correlation — was more trustworthy and infinitely more explainable. When the tool says "63% followed a deploy," I can point at the exact incidents behind that number. An engineer will believe a number they can audit; they won't act on a black-box "risk score." Keeping it countable was the right call.

Where this goes

The natural next step writes itself: if the memory knows that a failure class is deployment-correlated, the agent can move from response to prevention — flag a risky deploy before it ships, not after it pages someone. The foundation is already there, because every resolved incident gets retained back into memory:

await hindsight.aretain(bank_id=BANK_ID, content=content,
                        context="Incident resolution feedback")

Every confirmed resolution makes the next pattern sharper. The system gets more useful the longer it runs, without anyone retraining anything.

Lessons I'd reuse

Durable, structured memory turns retros into queries. The reason I could compute recurring patterns at all is that Hindsight kept incidents around as real records instead of letting them die in a ticket tracker.
Count before you cluster. The explainable, auditable metric beat the fancy one. Engineers act on numbers they can trace.
Compute insights from memory at request time. Don't cache a "risk score." Derive it from what's actually stored, so it's always current.
Be honest about correlation. Point humans at hypotheses; don't claim causation you can't back.
Response is step one; prevention is the payoff. Once memory knows the pattern, warning before a deploy is a small step, not a new system.

The single most valuable thing the tool ever told me wasn't how to fix an incident. It was which incident we were going to have again — and that our own deploys were the trigger. Memory made that visible. Nothing else did.

DEV Community: Anand Kumar