I Built an Incident-Response Agent That Remembers What Worked and What Didn’t

Anjali devi Muppidi — Tue, 29 Sep 2026 18:14:20 +0000

When something goes wrong in a software system, engineers usually start by checking the logs and trying to understand what caused the problem.

But there’s another question that can be just as important:

Has this happened before, and what did we try last time?

A team might have already dealt with a similar incident a few days or weeks ago. Maybe one fix worked, while another didn't. If that information isn't available when the next incident happens, someone might end up repeating the same failed approach.

That was the main idea behind our project.

We wanted to build an incident-response agent that doesn't just look at the current error and generate a response. Instead, it can also look at previous incidents and use that experience when dealing with a new one.

For this, we used Hindsight as the memory layer, Groq for the language model, and Streamlit for the interface.

The overall idea is pretty simple:

New incident → Check previous incidents → Generate a response → Record the result → Use it in the future

Why we wanted the agent to have memory

Imagine that an application suddenly starts showing:

Database connection timeout under heavy load

An AI model can obviously suggest a few possible solutions. It might recommend checking the database, increasing the connection pool, restarting a service, or looking at network issues.

But what if the team has already dealt with this?

Suppose they previously found that increasing the connection pool solved the issue, while restarting the service didn't make any difference.

A normal AI conversation doesn't automatically know about that previous experience.

That's where we thought memory could make the agent more useful.

Instead of treating every incident as a completely new problem, the agent can search through previous incidents and bring relevant ones into the current conversation.

Our Incident Command Center

We built a Streamlit interface called Incident Command Center.

The user can enter an incident or error log and provide information such as its severity and the service or component involved.

The system then analyzes the incident and looks for relevant information from its memory.

[ADD YOUR INCIDENT COMMAND CENTER SCREENSHOT HERE]

Our Incident Command Center where a new incident can be entered and analyzed.

The interface also shows whether the AI engine and memory engine are ready.

The three main parts of the application are:

Streamlit for the interface
Hindsight for remembering previous incidents
Groq for generating the AI response
What we wanted the agent to remember

We didn't want to save only the error message.

For each incident, the useful information includes the error itself, what we thought caused it, what fix we tried, and what happened after trying that fix.

That last part is particularly useful.

If a fix didn't work, that's still useful information. The next time a similar incident happens, the agent can know that the same approach was already tried.

The code for saving an incident looks like this:

def save_incident(log, root_cause, fix, outcome):
text = (
f"Error: {log}\n"
f"Root cause: {root_cause}\n"
f"Fix: {fix}\n"
f"Outcome: {outcome}"
)

response = requests.post(
    f"{BASE_URL}/spaces/{SPACE_ID}/retain",
    headers={"Authorization": f"Bearer {HINDSIGHT_KEY}"},
    json={"content": text}
)

return response.json()

Finding something similar from the past

When a new incident comes in, we can ask Hindsight to find relevant memories.

The basic idea is that the current error becomes the search query, and Hindsight returns related incidents that the agent can use.

def find_similar(new_log):
response = requests.post(
f"{BASE_URL}/spaces/{SPACE_ID}/recall",
headers={"Authorization": f"Bearer {HINDSIGHT_KEY}"},
json={"query": new_log}
)

return response.json()

This gives the agent some history to work with instead of making it start from scratch every time.

Giving the AI that history

This is where the memory becomes part of the actual response.

After finding similar incidents, we include them along with the new incident when creating the prompt for the language model.

prompt = f"""
You are an on-call engineering assistant.

New incident: {log}

Similar past incidents from memory:

{similar}

Based on past incidents, suggest the likely root cause and fix.

If a past fix failed, do not suggest it again.
"""

So rather than simply asking the model what it thinks the solution should be, we're giving it some context about what happened previously.

That doesn't mean the previous solution will always be correct for the new incident. It simply gives the model more information to consider.

What changes when the agent has memory?

Here's a simple example.

Without memory

An engineer enters:

Database connection timeout under heavy load

The AI looks at the current error and suggests a possible solution.

It has no information about what the team tried during previous incidents.

With memory

The same type of incident happens again.

This time, the agent can retrieve something like:

Error: Database connection timeout after 500 requests
Root cause: Connection pool exhausted
Fix: Increased pool size from 10 to 50
Outcome: Worked

Now the response has some real history behind it.

It might also retrieve an earlier attempt that failed:

Fix: Restarted database service
Outcome: Failed

That gives the agent another useful piece of information: something was already tried and didn't solve the problem.

That's the main difference we wanted to show.

Why failed fixes are useful too

We usually think of remembering successful solutions as the useful part.

But while working on this idea, one thing that stood out was that failed attempts can be just as valuable.

If an engineer already tried restarting a service and it didn't solve the issue, there isn't much value in suggesting exactly the same thing again without a reason.

So instead of only remembering:

"This fix worked."

the agent can also remember:

"This fix was tried, and it didn't work."

That creates a simple feedback loop:

Incident → Suggested fix → Worked/Failed → Memory → Future incident

Over time, the memory becomes a record of previous troubleshooting experiences.

Giving the memory some initial data

Of course, if the memory starts completely empty, there isn't much for the agent to recall.

So for testing, we use example incidents covering different types of problems, such as database issues, API timeouts, memory leaks, deployment failures, and third-party service outages.

This gives the agent some initial history.

Then we can enter a new incident through the Streamlit interface and see whether relevant previous incidents are retrieved.

The basic workflow is:

Enter the incident.
Select the severity and service information.
Analyze the incident.
Retrieve relevant memories.
Generate a suggested response.
Record whether the suggested fix worked or failed.
What I learned from building this
Memory needs context

One thing that became clear is that simply saving an error message isn't enough.

The error becomes much more useful when it's connected to the root cause, the attempted fix, and the outcome.

That gives the agent more information to work with when it encounters something similar later.

A failed solution isn't wasted information

It's easy to think that only successful fixes are worth remembering.

But during incident response, knowing what didn't work can save time too.

It can stop the same approach from being blindly repeated.

Memory works better when it's part of the workflow

We didn't want memory to be something that users had to manually search through whenever an incident happened.

Instead, it is part of the response process:

Recall → Analyze → Respond → Record

That makes the memory useful at the point where the agent is actually making a suggestion.

Similar incidents aren't necessarily identical

There's also an important limitation.

Two incidents can look similar but have completely different causes.

So previous incidents should be treated as context, not as a guaranteed answer.

The suggested fix still needs to be checked against the actual system.

The main idea

For me, the interesting part of this project isn't just getting an AI model to explain an error.

It's giving the agent some experience to work with.

An incident happens.
The agent looks at what happened before.
It suggests a response.
The result gets recorded.
And that experience can be useful the next time something similar happens.

That's what we wanted to explore with Hindsight.

Instead of an incident-response agent that starts from zero every time, we wanted one that can remember:

what happened, what was tried, and what actually worked.

That small change makes the idea of an AI incident-response agent much more interesting

# I Built an AI Incident Agent with Hindsight That Remembers What Failed

DEV Community: Anjali devi Muppidi

I Built an Incident-Response Agent That Remembers What Worked and What Didn’t