I Built a DevOps Agent That Remembers How Incidents Were Fixed

Saipriya41 — Tue, 29 Sep 2026 07:46:11 +0000

Most incident-response tools can tell you that something is broken. The harder problem is remembering that you have already seen something like it before.

I built a DevOps incident-response agent around that idea: instead of treating every production error as a completely new question, the agent retrieves relevant incidents from persistent memory, gives that context to an LLM, and stores the resulting diagnosis and resolution for the next time the same class of failure appears.

The interesting part wasn't simply connecting an LLM to a log stream. It was giving the agent a memory that could survive beyond a single conversation.

The Problem With Starting From Zero

Imagine a production system suddenly starts reporting:
PostgreSQL connection pool exhausted
sqlalchemy.exc.TimeoutError
.
.
QueuePool limit exceeded
An LLM can explain what a connection pool is. It can suggest increasing the pool size. It can recommend checking for connection leaks.

But that isn't necessarily what I want from an incident-response system.

what I really want to know is:

Have we seen this before, and what happened when we fixed it?

A previous incident might reveal that the real problem wasn't the pool size at all. Perhaps an unindexed query caused connections to remain open under load, and the eventual fix was to roll back a deployment and correct the query.

That historical context is much more valuable than a generic explanation.

This is where I use Hindsight.

The Architecture Is Deliberately Simple

The core system has three stages:

                     │Incident
                     ▼
             ┌───────────────┐
             │    RECALL     │
             │   Hindsight   │
             └───────┬───────┘
                     │
              Past incidents
                     │
                     ▼
             ┌───────────────┐
             │    Groq LLM   │
             │   Reasoning   │
             └───────┬───────┘
                     │
              Diagnosis +
              remediation
                     │
                     ▼
             ┌───────────────┐
             │    RETAIN     │
             │   Hindsight   │
             └───────────────┘

The application is written in Python and exposed through Streamlit. Groq provides the LLM interface, while Hindsight provides persistent memory.

The important architectural decision is that memory isn't treated as a static document that gets pasted into every prompt.

The agent explicitly recalls relevant information for the current incident.

Then, after producing a resolution, it retains the new experience.

That gives the system a feedback loop:

Incident
↓
Recall
↓
Reason
↓
Resolve
↓
Retain
↓
Future Incident
↓
Recall the previous experience

The more incidents the system processes, the more useful its historical context can become.

For background on the approach, I used the Hindsight documentation and the broader agent memory explanation from Vectorize.

The Recall Step

The first interesting piece of the implementation is the memory lookup.
My Streamlit application sends the incident to Hindsight:
def recall_memory(query):
headers = {
"Authorization": f"Bearer {HINDSIGHT_API_KEY}"
}

res = requests.post(
    f"{HINDSIGHT_BASE_URL}/banks/{HINDSIGHT_BANK_ID}/recall",
    json={"query": query},
    headers=headers,
    timeout=5
)

if res.status_code == 200:
    memories = res.json().get("memories", [])
    return "\n".join(
        [m.get("text", "") for m in memories]
    )

return ""

There is an important detail here.
I'm not retrieving the entire incident history and dumping it into the prompt.
The current incident becomes the query:

PostgreSQL connection pool exhausted

Hindsight returns relevant memories, which are then passed to the reasoning stage.

This changes the LLM's job.

Instead of:

"What could possibly be causing this?"

the agent can reason more like:

"This looks similar to incidents we've already encountered.
What did we learn from those incidents?"

That distinction is the central idea behind the project.

Memory Becomes Part of the Prompt

Once Hindsight returns relevant memories, I construct the system instruction:

system_instruction = (
"You are an expert DevOps Incident Response Agent "
"powered by Hindsight persistent memory.\n"
"Analyze system errors and recommend fixes based on "
"past incident post-mortems.\n"
f"Relevant Past Memory Context:\n{context}\n"
)
The LLM isn't replacing the memory system.
It's sitting on top of it.
The responsibilities are separated:

Hindsight
↓
"What have we seen before?"

LLM
↓
"Given that history, what should we do now?"
That separation made the architecture easier to reason about.

The memory layer doesn't need to be responsible for generating an answer, and the LLM doesn't need to pretend that it remembers everything itself.

The Agent Then Retains What It Learned

After the LLM generates its response, the incident and solution are stored:

retain_memory(
f"User Incident: {user_input} | "
f"Agent Solution: {reply}"
)
The retention function sends that information back to Hindsight:
def retain_memory(text):
requests.post(
f"{HINDSIGHT_BASE_URL}/banks/"
f"{HINDSIGHT_BANK_ID}/retain",
json={"content": text},
headers=headers,
timeout=5
)
This is the part that turns the application from a conventional chatbot into a system with persistent incident memory.

Suppose the first incident produces:
Incident:
PostgreSQL connection pool exhausted.

Resolution:
Rollback v2.4.1 and investigate the query causing
connections to remain open.
That experience becomes available to the next relevant incident.
The second time something similar happens, the agent isn't starting from an empty context.
It can retrieve the previous resolution and use it as evidence during its reasoning.

A Concrete Incident Flow

Consider a production checkout service.
The live logs show:
[19:35:01 STDOUT] Gateway routing healthcheck OK.
[19:35:05 STDOUT] /api/v1/orders 200 OK - 42ms

[19:35:12 FATAL ERROR 500]
PostgreSQL connection pool exhausted at /checkout endpoint.

sqlalchemy.exc.TimeoutError:
QueuePool limit exceeded
The incident is passed into the agent.

Hindsight retrieves a previous incident involving the database connection pool.

The historical memory contains something like:
Incident #204:
Connection Leak in DB Proxy

Root Cause:
An unindexed JOIN query in release v2.4.1
held open DB connections under spike load.

Previous Resolution:

Scale DB connections dynamically
Roll back deployment to v2.4.0 The LLM now has both pieces of information: CURRENT INCIDENT + HISTORICAL EXPERIENCE ↓ LLM REASONING ↓ RECOMMENDED MITIGATION

That is much closer to how an experienced engineer approaches an incident.

The engineer doesn't just look at the current stack trace.

They ask:

"Didn't we have this problem two weeks ago?"

The agent is trying to make that question automatic.

From Chatbot to Incident-Response Interface
The repository also contains a dashboard concept called OpsMind AI.

The interface is organized around three areas:
┌──────────────────┬──────────────────┬──────────────────┐
│ │ │ │
│ LIVE ERROR LOG │ HINDSIGHT MEMORY │ RUNBOOK │
│ │ │ │
│ Current failure │ Previous cases │ Recommended │
│ Stack traces │ Root cause │ mitigation │
│ Service state │ Resolution │ execution │
│ │ │ │
└──────────────────┴──────────────────┴──────────────────┘
The first panel represents the operational reality: errors arriving from a system.
The second represents the agent's memory.
The third represents action.
That separation matters because incident response isn't finished when an LLM writes a paragraph explaining the error.
The useful output is something an engineer can act on.

For example:

Suggested Mitigation

☑ Identify faulty query parameters
☐ Increase connection pool limit
☐ Roll back to stable release v2.4.0
The eventual production implementation can connect those recommendations to controlled runbooks rather than merely displaying them.

Why Persistent Memory Changes the Design

Without persistent memory, an agent generally has two choices.
It can rely on the current conversation context, or the application can build increasingly large prompts containing historical information.
Neither approach is particularly attractive for an incident system with a growing history.
A dedicated memory layer gives the application a different model:
┌─────────────────┐
│ Current Context │
└────────┬────────┘
│
▼
┌─────────────────┐
│ Hindsight │
│ Persistent │
│ Memory │
└────────┬────────┘
│
▼
┌─────────────────┐
│ LLM Reasoning │
└─────────────────┘
The agent can selectively retrieve what matters instead of treating every historical incident as equally relevant.
That is the reason Hindsight is central to the architecture rather than just another dependency.

What I Learned Building It

1. Memory is more interesting when it changes future behavior

Saving conversations isn't enough.

The useful question is:
Does something the agent learned today improve what it can do tomorrow?
That became the design criterion for the memory layer.

2. Retrieval should happen before reasoning

I found it more useful to make historical context available before asking the LLM to analyze the incident.
The sequence matters:
Recall → Reason → Retain
rather than:
Reason → Hope the model remembers

3. Incident resolution needs an action layer

An explanation is useful, but production engineering ultimately requires action.
That's why the dashboard concept includes a runbook layer.
The long-term goal is not:
AI:
"Here is what you should probably do."
It is:
AI:
"Here is the evidence.
Here is the historical incident.
Here is the proposed runbook.
Here is what will happen if you execute it."
with appropriate human approval and safeguards around execution.

4. Persistent memory introduces its own engineering problems

Once an agent can remember previous incidents, memory quality becomes part of system quality.
Bad historical information can lead to bad future recommendations.
That means productionizing this architecture requires thinking about retention quality, stale resolutions, conflicting incidents, confidence, observability, and human review.
Memory is powerful precisely because it has consequences.

5. The most useful agent isn't necessarily the one with the largest prompt

The project reinforced a simple idea for me:
More context isn't automatically better context.
The useful context is the context that is relevant to the incident being investigated.
That is why the recall layer is such an important part of the system.

Where This Goes Next

The current architecture gives me a foundation for a more complete incident-response system.
The next stage is connecting the incident stream, memory retrieval, reasoning, and runbook execution into one controlled workflow:

Production Logs
│
▼
Incident Detection
│
▼
Hindsight Recall
│
▼
LLM Analysis
│
├───────────────┐
▼ ▼
Root Cause Historical Evidence
│
▼
Runbook Proposal
│
▼
Human Approval
│
▼
Controlled Execution
│
▼
Incident Result
│
▼
Hindsight Retain

The final step is the one I care about most.

Every resolved incident becomes potential knowledge for the next one.

That creates a system where incident response doesn't have to start from zero every time.

The goal isn't to build an AI that magically knows how to operate production systems.

It's to build an agent that can remember what happened before, reason from that history, and turn previous operational
experience into useful context when the next failure arrives.

DEV Community: Saipriya41