I Gave My SRE Agent a Memory With Hindsight

Mandadi Vennela Naga Sai — Tue, 29 Sep 2026 18:07:02 +0000

Most incident-response agents can analyze an outage.

The harder question is: what happened the last time we tried that fix?

While building IncidentIQ, I wanted my incident agent to do more than analyze the current logs. I wanted it to remember previous incidents, understand which fixes worked, which failed, and which were only temporary, and use that experience when recommending what to do next.

That became the reason I added Hindsight as the persistent memory layer.

Instead of starting every incident from zero, IncidentIQ can look at what happened before and turn those experiences into evidence for the next decision.

The Problem With Starting Every Incident From Zero

Imagine a payments-api suddenly starts returning HTTP 503 errors.

The incident might look like this:

Service: payments-api
Severity: CRITICAL

Alert:
HTTP 503 surge

Logs:
Database connection pool exhausted.

An LLM can look at those logs and suggest several possible fixes.

Restart the service.

Increase the connection pool.

Check database latency.

Inspect long-running queries.

But the LLM doesn't automatically know what happened during previous incidents in the same environment.

What if restarting the service worked once but only provided temporary relief?

What if increasing the database connection pool had already solved the same problem twice?

What if connection-timeout monitoring had previously prevented the same issue from recurring?

That history can change the recommendation.

This is the problem I wanted IncidentIQ to solve.

What I Built

IncidentIQ is an SRE incident-response system that combines a React and TypeScript frontend, a FastAPI backend, Hindsight for persistent memory, and Groq for AI reasoning.

I wanted memory to participate in the investigation itself, rather than being something separate from the incident workflow.

The basic architecture is:

React Frontend
      |
      v
FastAPI Backend
      |
      +----------> Hindsight
      |            Persistent Memory
      |
      +----------> Groq
                   AI Reasoning

The frontend never talks directly to Hindsight or Groq.

The backend controls the memory and reasoning pipeline.

The main incident workflow is:

Current Incident
       |
       v
Hindsight Recall
       |
       v
Previous Incidents
       |
       v
Previous Actions + Outcomes
       |
       v
LLM Reasoning
       |
       v
Recommendation
       |
       v
Engineer Action
       |
       v
New Outcome
       |
       v
Hindsight Retain

This creates a feedback loop instead of a one-time AI interaction.

Starting an Incident Investigation

The first step is describing the incident.

For example:

Service: payments-api
Severity: CRITICAL

Alert:
HTTP 503 surge

Logs:
Database connection pool exhausted.

IncidentIQ's investigation workspace allows the engineer to provide the service, severity, alert, and logs.

It also provides quick presets for common operational problems such as:

DB connection pool
Redis failure
Auth failure
Certificate expiry

Once the incident is submitted, the system doesn't immediately ask the LLM for an answer.

It first asks a more useful question:

Have we seen something like this before?

Giving the Agent a Memory

This is where Hindsight comes in.

IncidentIQ has a dedicated Hindsight service responsible for connecting the application to the Hindsight memory bank.

The actual implementation is small:

import os

from dotenv import load_dotenv
from hindsight_client import Hindsight


load_dotenv()

HINDSIGHT_API_KEY = os.getenv("HINDSIGHT_API_KEY")
HINDSIGHT_BASE_URL = os.getenv(
    "HINDSIGHT_BASE_URL",
    "https://api.hindsight.vectorize.io"
)
HINDSIGHT_BANK_ID = os.getenv(
    "HINDSIGHT_BANK_ID",
    "incidentiq-sre"
)

client = Hindsight(
    base_url=HINDSIGHT_BASE_URL,
    api_key=HINDSIGHT_API_KEY
)


def store_memory(content: str):
    return client.retain(
        bank_id=HINDSIGHT_BANK_ID,
        content=content
    )


def search_memory(query: str):
    return client.recall(
        bank_id=HINDSIGHT_BANK_ID,
        query=query
    )

There are two operations that matter here:

recall()
    |
    v
Retrieve previous operational experience

retain()
    |
    v
Store new experience for future incidents

You can explore Hindsight on GitHub and read the Hindsight documentation.

Recalling Relevant Incidents

When memory is enabled, IncidentIQ builds a query from the current incident:

query = f"""
Service: {request.service}
Severity: {request.severity}
Alert: {request.alert}
Logs: {request.logs}
"""

hindsight_response = search_memory(
    query
)

The recalled memories are then filtered according to the affected service before being used for analysis.

The idea is straightforward:

Current Incident
       |
       v
Hindsight Recall
       |
       v
Relevant Memories
       |
       v
Historical Facts
       |
       v
Recommendation

This is where the agent starts behaving differently from a stateless incident assistant.

The Interesting Part: Remembering What Worked

Retrieving an old incident is useful.

Retrieving the outcome of the previous solution is much more useful.

Suppose the historical memory contains:

Incident: INC-001

Action:
restart service

Outcome:
temporary

Another memory contains:

Incident: INC-001

Action:
increase database connection pool size

Outcome:
successful

And another contains:

Incident: INC-001

Action:
add connection timeout monitoring

Outcome:
successful

Now the agent has more than historical context.

It has evidence about what actually happened.

The interface makes this pattern visible by connecting previous actions with their outcomes.

For the payments-api example, the system can identify that restarting the service was not the same as permanently resolving the underlying connection-pool problem.

Turning Memory Into Evidence

This was one of the most important design decisions in IncidentIQ.

I didn't want the LLM to decide historical success rates on its own.

The backend extracts historical facts from the recalled memories and calculates action statistics.

The project explicitly separates successful, failed, and temporary outcomes:

if fact["outcome"] == "successful":

    if incident_id:
        data["successful_incidents"].add(
            incident_id
        )

elif fact["outcome"] == "failed":

    if incident_id:
        data["failed_incidents"].add(
            incident_id
        )

elif fact["outcome"] == "temporary":

    if incident_id:
        data["temporary_incidents"].add(
            incident_id
        )

The system then calculates the historical success rate from those explicit outcomes:

attempts = len(data["attempts"])
successful = len(
    data["successful_incidents"]
)

if attempts > 0:
    success_rate = successful / attempts
else:
    success_rate = None

This means the application owns the historical evidence.

The LLM reasons over that evidence.

That separation matters because a generated explanation should not become the source of truth for historical statistics.

From Historical Evidence to a Recommendation

After Hindsight memories have been recalled and processed, IncidentIQ passes the current incident and historical memories to Groq.

The reasoning prompt explicitly tells the model to stay within the evidence:

prompt = f"""
You are an SRE incident response assistant.

Analyze the current incident using the historical
Hindsight memories provided below.

IMPORTANT:
- Base your reasoning on the evidence.
- Do not invent historical incidents.
- Do not invent evidence IDs.
- Recommend practical actions an on-call engineer can take.
- Confidence must reflect how strongly the evidence
  supports the diagnosis.
- Historical success rates should only be estimated
  from explicit outcome information in the memories.
- If there is insufficient historical evidence, say so.

CURRENT INCIDENT:
{incident}

HISTORICAL HINDSIGHT MEMORIES:
{memories}
"""

The recommendation is also represented using a structured model:

class Recommendation(BaseModel):
    action: str
    rationale: str
    historical_success_rate: float | None = Field(
        default=None,
        ge=0.0,
        le=1.0,
    )
    drift_detected: bool = False
    evidence_ids: list[str] = []

This gives the frontend structured information rather than one large block of generated text.

The Recommendation

For the connection-pool incident, the resulting recommendation can be:

Increase the database connection pool size
(e.g., from 50 to 100) and redeploy the
payments-api service.

The recommendation is displayed together with its historical effectiveness, confidence, and evidence.

For example, the interface shows:

Historical Effectiveness:
100%

2 Successful / 2 Recorded

Confidence:
95%

Evidence:
INC-001

This changes the interaction from:

AI:
"Try increasing the connection pool."

to:

Historical Evidence:
2 successful / 2 recorded

Recommendation:
Increase the database connection pool.

Evidence:
INC-001

The engineer can inspect the evidence and then decide what action makes sense.

What Changes With Memory?

The simplest way to understand the role of Hindsight is to compare the two workflows.

Without Memory

Incident
   |
   v
LLM
   |
   v
Possible Causes
   |
   v
Generic Recommendation

With Hindsight

Incident
   |
   v
Hindsight Recall
   |
   v
Previous Incidents
   |
   v
Previous Actions
   |
   v
Previous Outcomes
   |
   v
LLM Reasoning
   |
   v
Evidence-backed Recommendation

The LLM itself hasn't magically become better at debugging.

The context available to it has changed.

Memory doesn't replace reasoning. It gives reasoning something more useful to work with.

Using the Same Memory Before Deployment

I also wanted the memory layer to be useful before an incident happens.

IncidentIQ therefore includes a pre-deployment risk review.

An engineer can enter a proposed change:

Service:
payments-api

Proposed Change:
Increase connection pool 50 → 100

[INSERT SCREENSHOT — Pre-Deploy Risk Review]

The system evaluates the proposed change using historical operational context.

The result can include a risk classification, risk score, rationale, safeguards, and related historical incidents.

This means the same memory layer can support two different moments in the software lifecycle:

Before Deployment
       |
       v
Historical Experience
       |
       v
Risk Assessment


During Incident
       |
       v
Historical Experience
       |
       v
Incident Recommendation

The memory isn't tied to only one screen or one workflow.

Exploring What the Agent Remembers

I didn't want the memory layer to be invisible.

IncidentIQ includes a Memory Explorer where engineers can inspect the operational memories available to the system.

For example, the memory view can contain information such as:

Incident INC-001 was mitigated by restarting
the payments-api service and permanently resolved
by increasing the database connection pool size
and adding connection timeout monitoring.

The memory explorer is useful for another reason: it makes the agent itself easier to inspect.

If a recommendation seems unexpected, an engineer can look at the memories that were available to the system.

Memory becomes something that can be investigated instead of a hidden part of the prompt.

Learning From What Actually Happened

A recommendation shouldn't automatically become a permanent truth just because an AI generated it.

The engineer still needs to record what happened.

IncidentIQ accepts an outcome and turns it into a new memory.

The backend builds an outcome memory like this:

outcome_memory = f"""
Incident {request.incident_id} affected the
{request.service} service.

Resolution attempt:
{request.action}

Outcome:
{outcome_description}

Engineer notes:
{request.notes}

Interpretation:
The engineer marked this resolution attempt as
{outcome_description}.
"""

It then stores that experience:

result = store_memory(
    outcome_memory
)

So the loop becomes:

Incident
   |
   v
Recall
   |
   v
Historical Evidence
   |
   v
Recommendation
   |
   v
Engineer Action
   |
   v
Real Outcome
   |
   v
Retain
   |
   v
Future Incident

This is the part of IncidentIQ I find most interesting.

The agent isn't simply generating answers.

It has a mechanism for accumulating operational experience.

When There Isn't Enough Evidence

IncidentIQ also includes a Fix Drift view.

The idea is that a fix that worked previously should not automatically be considered reliable forever.

As more outcomes are recorded, the system can compare newer results with historical results.

But there is an important limitation.

The current interface explicitly reports when there aren't enough recorded outcomes to detect fix drift.

I prefer this behavior to manufacturing a conclusion.

If the system doesn't have enough evidence, the answer should be:

Not enough recorded outcomes to detect fix drift.

That is much more useful than pretending the system knows something it doesn't.

What I Learned

Memory Is More Useful When It Includes Outcomes

Remembering that an incident happened is useful.

Remembering what was tried and whether it worked is much more useful.

Retrieval Should Be Visible

Historical context shouldn't disappear inside an LLM prompt.

Making memories and evidence visible gives engineers a way to inspect why a recommendation was produced.

The LLM Shouldn't Own the Historical Facts

The application extracts historical outcomes and calculates explicit statistics.

The LLM reasons over those facts.

This separation makes the system easier to inspect.

Insufficient Evidence Is Still Information

If there aren't enough recorded outcomes to detect a pattern, the system should say so.

The Feedback Loop Is the Real Memory

The interesting architecture isn't:

Incident → AI → Answer

It's:

Incident
   |
   v
Memory
   |
   v
Recommendation
   |
   v
Action
   |
   v
Outcome
   |
   v
Memory

That is what turns previous incidents into experience that can influence future incidents.

Final Thoughts

IncidentIQ started from a simple observation:

SRE teams repeatedly solve problems that have similar histories.

An LLM can reason about the incident in front of it.

But persistent memory gives that reasoning access to what happened before.

That changes the question from:

"What should I do?"

to:

"What happened the last time this occurred,
what actually worked,
and what evidence do we have?"

That's what I wanted to explore with IncidentIQ: an SRE agent that doesn't just respond to incidents, but can build a persistent memory of operational experience.

The complete source code for IncidentIQ is available on GitHub:

Pravallika2789 / IncidentIQ-SRE-Agent

AI-powered SRE incident response agent that uses Hindsight memory to learn from past incidents, successful and failed fixes, and deployment history.

IncidentIQ - Intelligent SRE Incident Response

IncidentIQ is an intelligent SRE incident-response agent that uses Hindsight persistent memory to help engineers investigate production incidents, recall previous operational experience, learn from successful and failed resolution attempts, and make safer deployment decisions.

It combines a React frontend, FastAPI backend, Hindsight for persistent operational memory, and Groq-powered AI reasoning to turn past incidents into reusable engineering knowledge.

Team Code & Chaos

1. Reddy Gayatri Satya Sai Pravallika

2. Mandha Varshitha

3. Chenna Keerthana

4. Mandadi Vennela

Video Presentation

Watch the IncidentIQ Demo

Problem

When a production incident happens, engineers need more than an analysis of the current logs.

They need to know:

Has this happened before?
What caused the previous incident?
Which fix actually worked?
Which approaches failed?
Which runbook was useful?
What did the team learn from previous incidents?
Has a previously successful fix become less effective?

Solution

IncidentIQ turns previous incidents…

View on GitHub

DEV Community: Mandadi Vennela Naga Sai