When My Incident Bot Found the Same Failure Twice

Gayathri Reddy Lenkala — Tue, 29 Sep 2026 18:04:58 +0000

An incident rarely arrives with a message saying, "You've seen me before."

It arrives as a symptom.

A 502.

A growing queue.

A sudden latency spike.

A service that worked yesterday and doesn't work today.

I built RETRACE around a simple idea: before the incident assistant starts inventing explanations, let it look at what happened the last time.

Incidents leave clues behind

I started with structured incident records.

For example:
{
"id": "INC-1042",
"service": "checkout-api",
"severity": "SEV-1",
"symptoms": "POST /checkout returns 502",
"root_cause": "Database connection pool exhausted",
"fix": "Reduced retry fan-out and increased the pool",
"lesson": "Check DB pool saturation during traffic spikes"
}
That record already contains something valuable.

It tells us:

what broke
where it broke
why it broke
how it was fixed
what we learned

The problem is making that information available when another incident arrives weeks later.

That's where Hindsight comes in.

## RETRACE has two kinds of context

The assistant has the current incident.

Hindsight provides access to historical experience.

So instead of treating every request independently, the workflow becomes:
Current incident
│
▼
RETRACE agent
│
▼
Search memory
│
┌──────────┴──────────┐
│ │
Similar incident No useful match
│ │
▼ ▼
Historical evidence Normal investigation
│
└──────────┬──────────┘
▼
Response
Hindsight's memory model is designed around storing information and retrieving relevant memories later.

That made it a natural fit for incident history.

The failure pattern I cared about

**
One of the most interesting examples in the data involves checkout-api.

There are two separate incidents:
INC-1042
SEV-1
checkout-api
502 responses
traffic spike
DB connection pool exhaustion
retry storm

and :

INC-1098
SEV-2
checkout-api
intermittent 502s
retry loop
DB connection pool exhaustion

They aren't duplicates.

But they're related.

Both tell us that retries can amplify a checkout failure until the database connection pool becomes the limiting resource.

That relationship is much more useful than simply searching for the exact phrase "502".

Why I didn't want a giant incident prompt

One tempting approach would have been to put every incident into the system prompt.

Something like:

Here are all 500 incidents we have ever experienced...

INC-1...
INC-2...
INC-3...
...

Then ask the model to figure out which ones matter.

That doesn't scale well conceptually.

It also makes the current question compete with a huge amount of historical information.

Instead, I wanted the memory layer to answer a narrower question:
What previous incidents are relevant to THIS problem?

That is the job I gave Hindsight.

The investigation becomes more specific

Consider this query:

"Checkout is returning 502s again. Traffic just increased."

A generic assistant might say:
Check:

application logs
load balancer
database
network
dependencies
CPU
memory

RETRACE has another option.

It can find previous checkout incidents.

Then the response can focus on the evidence:
A similar checkout incident occurred previously.

The previous failure involved:

a traffic spike
retry amplification
database connection exhaustion

Check the DB connection pool and retry behavior first.
That doesn't prove the current incident has the same root cause.

And that's important.

Historical memory should generate a useful hypothesis, not pretend to be proof.

This changed how I thought about agent memory

Before building RETRACE, I mostly thought about memory as something that helps an agent remember users or conversations.

Incident response made the use case feel different.

An engineering agent can remember:

What happened?
Why did it happen?
What fixed it?
What should we check next time?

That's operational memory.

Hindsight provides the infrastructure for retaining and recalling those memories rather than forcing the entire history into every model request.

The structured incident records are the source material.

The assistant turns the current symptoms into a question about previous experience.

The memory layer brings back the relevant incident.
The assistant can then combine the current symptoms with the historical evidence.

The architecture is deliberately small

The application doesn't need a complicated collection of services to demonstrate the core idea.

The lesson hidden in every incident report

The most useful field in the incident data may actually be the lesson.

For example:

When checkout traffic spikes,
check DB pool saturation before changing application logic.

That's different from the root cause.

The root cause describes what happened.
The lesson describes what someone should remember next time.

That's exactly the distinction that makes an incident history valuable to an agent.

What I would keep if I rebuilt it

There are a few things I'd preserve.

Keep incident history structured

A useful incident should contain more than a paragraph.

The combination of:
service
severity
symptoms
root cause
fix
lesson

makes the memory much more actionable.

Keep memory separate from reasoning

The LLM should reason about the incident.

The memory layer should help it find relevant experience.

Keeping those responsibilities separate makes the architecture easier to understand and debug.

*Treat historical matches as evidence
*
A previous incident is not a diagnosis.

If two incidents look similar, that's a reason to investigate the same failure mode—not a reason to declare that the current incident has the same root cause.
That distinction matters in production.

The part I found most useful

The interesting result wasn't that the bot could answer questions about incidents.

A normal LLM can do that.

The interesting part was being able to ask:

"Have we dealt with this before?"

and have the answer come from the system's accumulated operational history.

That's what changed RETRACE from a troubleshooting chatbot into something closer to an incident assistant with institutional memory.

The goal is straightforward:

When production breaks again, I don't want the investigation to start from zero.

DEV Community: Gayathri Reddy Lenkala