I Built an Incident-Response Agent That Remembers What Failed

Shiva Kumar Kanneboina — Tue, 29 Sep 2026 16:07:26 +0000

I Built an Incident-Response Agent That Remembers What Failed

Production incidents have a frustrating habit of repeating themselves.

A database gets overloaded. An API starts timing out. A cache runs out of memory. Engineers investigate, try a few fixes, eventually resolve the problem, and write everything down.

Then, weeks or months later, something similar happens again.

The problem is not always a lack of documentation. The problem is that the useful knowledge from the previous incident is usually scattered across tickets, postmortems, dashboards, Slack conversations, and the memories of the engineers who handled it.

So when the next incident arrives, the investigation often starts from scratch.

I wanted to build something different: an incident-response agent that could remember what happened before and use that experience when investigating the next incident.

That is how I built OpsMind.

The idea behind OpsMind

OpsMind is an AI incident-intelligence agent designed to help engineers investigate production incidents using two sources of information:

What is happening right now
What happened during similar incidents in the past

The second part is what makes the system interesting.

A typical AI assistant can look at the current symptoms and suggest possible causes. OpsMind can also ask a different question:

Have we seen something like this before?

If the system remembers a previous incident involving connection-pool exhaustion, database pressure, failed mitigation attempts, and a successful resolution, that experience becomes part of the investigation.

The agent is no longer starting with an empty context.

When I first thought about incident memory, the obvious approach was to store summaries of previous incidents.

But a summary like this:

"Database was overloaded. Increased the connection pool size. Incident resolved."

doesn't contain enough information to be useful during the next incident.

What caused the overload?

What symptoms appeared first?

Which service was involved?

What did the engineer try?

Which attempts failed?

What actually fixed the problem?

And perhaps most importantly, what should the next engineer avoid doing?

That last question turned out to be particularly important.

During an incident, a failed mitigation is still valuable information.

If restarting a service did not solve the problem, that is useful.

If adding more replicas made the underlying database pressure worse, that is useful.

If clearing a cache increased database load, that is useful too.

These are not just historical details. They are operational lessons.

That led me to a simple principle for OpsMind:

Remember not only what worked, but also what failed and why.

Using Hindsight as the memory layer

I integrated Hindsight as the persistent memory layer for OpsMind.

The goal was to move beyond storing previous conversations or plain incident summaries. The agent needs to retain useful information from an incident and later recall relevant experiences when another incident occurs.

The memory can capture information about facts, entities, relationships, temporal context, experiences, and lessons.

For OpsMind, this means an incident can become more than a document sitting in a database. It can become part of a connected body of operational knowledge.

The basic learning loop is:

Incident → Recall → Investigate → Recommend → Retain the outcome

During an incident, OpsMind can recall relevant historical experiences.

After the incident is resolved, the outcome can be retained so that the experience becomes useful the next time a similar problem appears.

That creates a feedback loop rather than a stateless assistant.

A real example from OpsMind

One of the incidents in my dataset involved a Payments API experiencing connection-pool saturation.

The underlying problem was a leaked client caused by a missing client.release().

What made this incident useful as training data was that it contained more than the final solution.

A restart was attempted, but it did not address the underlying problem.

Replica scaling was also attempted, but it was not the right solution. In this situation, increasing capacity could actually make database exhaustion worse.

The eventual resolution was a rollback.

So instead of remembering only:

"Payments API had connection-pool saturation and was fixed by a rollback."

OpsMind can retain the broader experience:

Symptoms
  ↓
Connection pool saturation
  ↓
Investigation
  ↓
Client leak identified
  ↓
Restart attempted
  ↓
Unsuccessful
  ↓
Replica scaling attempted
  ↓
Unsuccessful / potentially counterproductive
  ↓
Rollback
  ↓
Lesson learned

That difference matters.

If a similar incident happens again, the system has more context than just the final answer.

It has the path that led to the answer.

Similar symptoms do not always mean the same problem

Another important part of the system is that historical memory should not be treated as a guaranteed answer.

For example, connection-pool pressure can happen for very different reasons.

In the OpsMind dataset, one incident involved a client leak. Another involved a database connection-pool configuration problem. Another involved a slow, unindexed database query that caused long-running operations and eventually contributed to pool starvation.

The symptoms can look similar while the root causes are different.

That is why I don't want OpsMind to simply retrieve an old incident and copy its solution.

Instead, historical incidents are additional evidence.

The agent can compare the current symptoms with previous experiences, look for similarities and differences, and then use that context when forming a recommendation.

The goal is:

Use the past to improve the investigation, not to replace the evidence from the present.

That distinction is important for any system that is going to be used around production infrastructure.

The value of remembering failed fixes

While building the incident dataset, I noticed a pattern that became one of my favorite parts of the project.

A lot of operational knowledge is hidden inside failed attempts.

For example, the dataset contains incidents where:

Restarting a service did not fix an underlying resource leak.
Increasing capacity could delay or worsen database exhaustion.
Killing idle database sessions did not address the underlying query behavior.
Adding Kafka consumers could make a rebalance problem worse.
Clearing a cache could increase load on the database.
Increasing memory could delay an out-of-memory problem without fixing the underlying issue.

These lessons are easy to lose.

A traditional incident summary tends to focus on the final resolution. But during a real incident, knowing what has already been tried can prevent engineers from repeating the same mistakes.

That gives the memory system another useful question to answer:

What did we try last time that did not work?

For incident response, that can be just as important as asking what worked.

What I learned while building OpsMind

The first thing I learned is that memory quality matters more than memory quantity.

It is easy to store a large amount of incident data.

It is much harder to make that information useful.

The memory needs to preserve the parts of an incident that actually matter during an investigation: symptoms, root causes, actions, outcomes, failed attempts, and lessons.

The second lesson was that retrieval needs to stay grounded in the current incident.

Historical experience is useful context, but it should not become unquestioned truth.

If current logs, metrics, or traces contradict something that happened in an older incident, the current evidence should take priority.

The third lesson was that visualizing memory is surprisingly useful during development.

Looking at the Hindsight memory graph made it easier to understand how incident information was being connected and whether the retained data contained meaningful relationships.

It also exposed data-quality issues.

For example, duplicate or test memories can introduce unnecessary noise. Cleaning those up became part of preparing the system for a more realistic demonstration.

How the learning loop works

I think about OpsMind's memory system in four stages.

1. Retain

After an incident, the system retains the useful operational knowledge.

That includes what happened, what was discovered, what actions were attempted, what worked, what failed, and what was learned.

2. Recall

When a new incident arrives, OpsMind looks for relevant historical experiences.

The goal isn't to retrieve everything the system knows.

The goal is to find experiences that can actually help with the current investigation.

3. Reflect

The recalled experiences become context for reasoning.

The agent can compare the current symptoms with previous incidents and consider both the similarities and the differences.

This is where memory becomes more than search.

4. Retain again

Once the incident is resolved, the new outcome becomes another experience.

Over time, the system can build a growing collection of operational knowledge.

That is the part that turns the architecture from a chatbot into a learning incident assistant.

What I would improve next

OpsMind is still a prototype, and there are several areas I would improve before treating it as a production incident-response system.

The first is evaluation.

It is not enough to demonstrate that the agent can retrieve a related incident. I want to measure whether the retrieved memory actually improves investigation quality, helps identify likely root causes faster, and reduces repeated failed mitigations.

The second is better evidence handling.

A recommendation should clearly distinguish between:

Current incident evidence
Historical experience
Agent inference
Recommended action

That makes it easier for an engineer to understand why the agent reached a particular conclusion.

The third is deeper integration with real operational systems.

A production version could connect directly to logs, metrics, traces, deployment history, incident-management systems, and runbooks.

I would also keep human approval in the loop for actions that can change production infrastructure.

The goal isn't to build an agent that blindly executes changes.

The goal is to build an agent that gives engineers better context when they need to make decisions quickly.

Why persistent memory matters

The more I worked on OpsMind, the more I realized that incident response is fundamentally a knowledge problem.

Organizations accumulate years of operational experience.

Engineers learn which failures happen repeatedly.

They learn which fixes are safe.

They learn which seemingly obvious fixes can make a situation worse.

They learn patterns that never quite make it into a formal runbook.

But a stateless AI agent cannot reliably carry that experience from one incident to the next.

Persistent memory changes that.

With Hindsight, OpsMind can retain operational experiences and bring relevant ones back into a future investigation.

The result is not simply an AI that answers:

"What should I do?"

It can also ask:

"What happened the last time we saw something like this?"

And, perhaps more importantly:

"What did we try last time, what failed, and what did we learn?"

That is the behavior I wanted to build.

Because when the next production incident happens, the investigation should not have to start from a blank context window.

It should start with everything the system has already learned.

Project: OpsMind
Memory layer: Hindsight
Core loop: Retain → Recall → Reflect → Retain
Focus: Persistent incident intelligence and learning from operational experience

DEV Community: Shiva Kumar Kanneboina

I Built an Incident-Response Agent That Remembers What Failed

I Built an Incident-Response Agent That Remembers What Failed

The idea behind OpsMind

Using Hindsight as the memory layer

A real example from OpsMind

Similar symptoms do not always mean the same problem

The value of remembering failed fixes

What I learned while building OpsMind

How the learning loop works

1. Retain

2. Recall

3. Reflect

4. Retain again

What I would improve next

Why persistent memory matters