DEV Community: Alekhya Allatipalli

RecallOps: Using Hindsight to Recall Past Production Incidents

Alekhya Allatipalli — Mon, 28 Sep 2026 09:58:05 +0000

How Operational Memory Changes Incident Investigation with Hindsight

Most of the time in an incident goes to a single question: have we seen this before? The metrics, logs, and alerts describe what is happening now. The answer to that question usually sits in a closed ticket, an old chat thread, or one colleague's memory.

I worked with a team of five on RecallOps, an AI incident-response copilot built around that question. In this article I focus on one idea: what changes in an investigation when the system has operational memory, and what has to be true for that memory to help without misleading anyone.

What Investigation Looks Like Without Memory

An assistant without memory can only reason about the symptoms in front of it. It knows nothing about your system's history, so it cannot tell you that a similar incident was resolved three months ago, or how.

We wanted recommendations that could be traced to specific earlier incidents. That meant retaining the things an engineer would want to know later: the incident, its root cause, its resolution, its outcome, and engineer feedback. It also meant recalling those records when a new incident begins.

The loop RecallOps follows is:

Incident → Recall → AI Investigation → Resolve → Retain → Reflect → Better Future Investigation

Each resolved incident becomes input for the next one.

Where Hindsight Fits

We designed the memory layer around Hindsight's retain-and-recall model. Retain stores what an incident taught us, and recall brings back what is relevant to a new one. The Hindsight documentation and Vectorize's overview of agent memory explain the broader concept.

The architecture also includes a durable database fallback and degraded-mode behavior, so the application keeps working from its own persisted incident data if the memory provider is unavailable.

To be precise about what was verified: in the local demo, the system uses SQLite with a deterministic fallback. The remote Hindsight service, production PostgreSQL, and live Groq/OpenAI inference are configured in the architecture but were not live-verified. Everything I describe below comes from the local demo.

A Worked Example: INC-001 and INC-017

The demo data contains two linked incidents.

INC-001 was a Payment API database timeout. The root cause was connection-pool exhaustion caused by a connection leak, and the resolution was to fix the leak and increase pool capacity. It is resolved and retained.

INC-017 is a later Payment API database timeout with similar symptoms. When it is investigated, RecallOps recalls INC-001 as the top historical match at 91% similarity and lists the reasons:

same service
same service family
similar symptoms
similar database behavior
similar timing

The memory card also shows INC-001's root cause and resolution. It notes that connection utilization and lifecycle checks were prioritized because of that earlier outcome. This is the practical effect of memory: the investigation starts from a better-informed hypothesis instead of a blank page.

[Fig 1 — Incident workspace for INC-017. Shows the current incident, the AI investigation console, and the recalled memory panel in one view.]

[Fig 2 — INC-001 memory card at 91% similarity. Shows the match reasons and the historical root cause.]

Keeping Current and Historical Evidence Apart

Recalled memory is only useful if it is clearly marked as history. RecallOps separates four things:

Current evidence
Historical evidence
AI recommendation
Uncertainty

For INC-017, the current evidence is 98% connection utilization, a 14.2% timeout rate, a deployment 23 minutes earlier, and a connection-timeout error. The historical evidence from INC-001 is the same service and error family and a confirmed connection leak.

A blended paragraph would hide which claims are observed now and which are borrowed from the past. Two labeled panels let an engineer check each side independently.

[Fig 3 — Current vs Historical Evidence. Shows the two evidence panels side by side.]

Surfacing Uncertainty

A 91% match says the two incidents are alike, not that INC-017 has the same root cause. The recent deployment shows why. It may have introduced a new leak, or it may be unrelated.

So the memory card describes the match as evidence for investigation, not confirmation of the current root cause. The workspace shows uncertainty alongside the recommendation and offers structured investigation paths, so the engineer can test the hypothesis instead of accepting it. The Learning page uses the same framing for recurring patterns, describing them as patterns to investigate and not as a confirmed universal cause.

The Engineer Makes the Decision

RecallOps does not change production systems automatically. It provides evidence, historical context, recommendations, investigation paths, and uncertainty, and the engineer makes the final operational decision.

That boundary matters more once a system has memory. Memory can make a wrong answer more convincing when a new incident only resembles an old one. Keeping a person in the decision means a mistaken recall costs a wasted check, not a wrong action.

Recall and Retain in Code

The snippets below are simplified, representative examples of the shape of the code, not the exact implementation.

Recall falls back to the database when the memory provider is unavailable and marks the response as degraded:

@router.get("/incidents/{incident_id}/recall")
def recall_memory(incident_id: str, db: Session = Depends(get_db)):
    incident = get_incident_or_404(db, incident_id)
    try:
        matches = memory.recall(incident)
        degraded = False
    except MemoryUnavailable:
        matches = db_fallback_recall(db, incident)
        degraded = True
    return {"matches": matches, "degraded": degraded}

Retention is allowed only after an incident has been resolved:

@router.post("/incidents/{incident_id}/retain")
def retain_incident(incident_id: str, db: Session = Depends(get_db)):
    incident = get_incident_or_404(db, incident_id)
    if incident.status != "resolved":
        raise HTTPException(400, "Resolve the incident before retaining it")
    memory.retain(incident)
    incident.retained = True
    db.commit()
    return {"retained": True}

The degraded flag means a fallback result is never presented as if it came from the primary memory provider.

Retain and Reflect

After INC-017 is resolved and retained, it becomes memory for later investigations. The Learning area then looks across retained incidents for recurring patterns. On the demo data it identifies a Payment API pattern involving 5 related incidents, with 6 stored incidents and 5 retained memories. It also shows which incidents each lesson came from, so a lesson can be traced back to its evidence.

[Fig 4 — Learning page. Shows the recurring Payment API pattern, the related incident IDs, and the stored and retained counts.]

Architecture Diagram

What I Learned

Memory needs labels. Recalled history helps when it is clearly marked as history. Separating current from historical evidence did more for trust than any change to the recommendation wording.

Similarity is not causation. A high match score tells you where to look first. We had to say that explicitly in the interface, because a percentage on its own reads like an answer.

Design for degraded modes early. The database fallback and the deterministic AI fallback made the app dependable locally. They also forced us to decide what "degraded" should look like to the user.

Be exact about what was verified. The local demo, SQLite persistence, backend tests, and the browser workflow were checked. The remote integrations were not tested live, so the article says so.

Takeaway

Operational memory improves incident investigation when past incidents are retained, recalled by similarity, and shown as clearly labeled history next to current evidence, with uncertainty visible and the decision left to the engineer.

RecallOps: Using Hindsight to Recall Past Production Incidents

Alekhya Allatipalli — Mon, 28 Sep 2026 07:21:33 +0000

Introduction

Every on-call engineer has had the same feeling: an alert fires, the symptoms look familiar, and you can't remember where you saw them before. Somebody fixed this months ago, but the fix lives in a closed ticket, a Slack thread, or one person's head.

I built RecallOps, an AI incident-response copilot, around that problem. It keeps a persistent record of past incidents (root causes, resolutions, outcomes, and engineer feedback) and retrieves the relevant ones when a new incident starts.I designed the operational memory layer around Hindsight's retain-and-recall model.

This article covers what I built, how the recall loop works, and which parts I verified end to end and which are configured integrations I haven't verified live.

The Problem: Incident Response Without Operational Memory

In an incident, engineers often have access to current metrics, logs, and alerts, but historical incident knowledge can be harder to surface at the moment it is needed.

Without that, engineers re-investigate problems from scratch. An AI assistant without memory has the same limitation: it can reason about the current symptoms, but it knows nothing about your systems' history. I wanted an assistant whose recommendations could be traced to specific past incidents.

What I Built: RecallOps

RecallOps follows one loop:

Incident → Recall → AI Investigation → Resolve → Retain → Reflect → Better Future Investigation

The frontend has seven areas: Dashboard, Incidents, Incident Workspace, Copilot, Memory, Learning, and Services. The Incident Workspace is where most of the work happens. From there an engineer can:

inspect the current incident
run AI analysis
recall historical memory
inspect similarity and match reasons
compare current and historical evidence
follow structured investigation paths
resolve the incident
retain the resolution as operational memory

One design principle shaped everything: RecallOps never changes production systems automatically. It provides evidence, history, recommendations, investigation paths, and uncertainty. The engineer makes the final decision.

Why Hindsight Is the Memory Layer

I wanted memory to be a separate layer with a clear contract, not a table of past tickets bolted onto a prompt. Hindsight fits that role. It provides retain and recall operations for agent memory, which map directly onto the two things RecallOps needs: store what we learned from an incident, and bring back what's relevant to a new one. The Hindsight documentation and Vectorize's overview of agent memory explain the concept in more depth.

Because I couldn't assume a remote memory service would always be reachable, RecallOps includes a durable database fallback and degraded-mode behavior. If Hindsight is unavailable, the app keeps working from its own persisted incident data instead of failing.

How the Incident Recall Loop Works
Recall: when an incident is opened, RecallOps looks for similar past incidents.
AI Investigation: the copilot analyzes the current incident in five visible stages, using recalled memory as context.
Resolve: the engineer resolves the incident.
Retain: the resolution is stored as operational memory.
Reflect: the Learning area looks across retained incidents for recurring patterns.
INC-001 → INC-017 Walkthrough

The demo data has two linked incidents.

INC-001 was a Payment API database timeout. Root cause: connection-pool exhaustion caused by a connection leak. Resolution: fix the leak and increase pool capacity. It is resolved and retained.

Later, INC-017 occurs with a similar Payment API database timeout. RecallOps retrieves INC-001 as the top historical match at 91% similarity, with these match reasons:

same service
same service family
similar symptoms
similar database behavior
similar timing

The memory view also shows INC-001's root cause, its resolution, and why it influenced the recommendations.

Fig 1 — RecallOps incident workspace showing INC-017 during investigation, including the recalled historical memory and current evidence.

Fig 2 — INC-001 historical memory with 91% similarity. Shows the match reasons, historical root cause, and resolution in the memory card.

Technical Architecture
Frontend: React and Vite, with a responsive UI
Backend: FastAPI (Python), SQLAlchemy, REST APIs
Persistence: SQLite for the verified local/demo path; the architecture is PostgreSQL-ready
Memory: Hindsight integration with retain/recall, plus a durable database fallback
AI: Groq/OpenAI-compatible structured completion, with a deterministic fallback when live credentials aren't available

The snippets below are simplified, representative examples of the shape of the code, not the exact implementation.

A simplified incident model:

class Incident(Base):
    __tablename__ = "incidents"

    id = Column(String, primary_key=True)
    service = Column(String, nullable=False)
    summary = Column(Text)
    status = Column(String, default="open")
    root_cause = Column(Text, nullable=True)
    resolution = Column(Text, nullable=True)
    retained = Column(Boolean, default=False)

This model stores the incident information needed by RecallOps, including its service, status, root cause, resolution, and whether the resolved incident has been retained as operational memory.

A simplified recall endpoint:

@router.get("/incidents/{incident_id}/recall")
def recall_memory(incident_id: str, db: Session = Depends(get_db)):
    incident = get_incident_or_404(db, incident_id)

    try:
        matches = memory.recall(incident)
        degraded = False
    except MemoryUnavailable:
        matches = db_fallback_recall(db, incident)
        degraded = True

    return {
        "matches": matches,
        "degraded": degraded,
    }

When an incident is investigated, RecallOps attempts to recall relevant historical incidents. If the memory provider is unavailable, the application uses its durable database fallback and marks the response as degraded rather than silently presenting the fallback as the primary memory provider.

A simplified retention step:

@router.post("/incidents/{incident_id}/retain")
def retain_incident(incident_id: str, db: Session = Depends(get_db)):
    incident = get_incident_or_404(db, incident_id)

    if incident.status != "resolved":
        raise HTTPException(
            400,
            "Resolve the incident before retaining it",
        )

    memory.retain(incident)
    incident.retained = True
    db.commit()

    return {"retained": True}

After an engineer resolves an incident, RecallOps can retain the incident as operational memory. This closes the loop: the outcome of today's investigation becomes context that can be recalled during a future investigation.

Current Evidence vs Historical Evidence

The part I care most about is that RecallOps keeps four things separate:

CURRENT EVIDENCE
HISTORICAL EVIDENCE
AI RECOMMENDATION
UNCERTAINTY

For INC-017, the current evidence is 98% connection utilization, a 14.2% timeout rate, a deployment 23 minutes earlier, and a connection timeout. The historical evidence from INC-001 is 98% connection utilization, the same service and error family, and a confirmed connection leak.

The overlap is worth investigating, but a similar past incident is not proof of the same root cause. RecallOps treats INC-001 as relevant evidence, not a conclusion, and the UI shows uncertainty next to the recommendation. The workspace also offers structured investigation paths so the engineer can check the hypothesis (for example, whether the recent deployment introduced a new leak) instead of accepting it.

Fig 3 — Current vs Historical Evidence. Shows the two evidence sets side by side, with the recommendation and uncertainty panels.

Retain and Reflect

After the engineer resolves INC-017 and retains it, the incident becomes part of the memory. The Learning area then looks across retained incidents for recurring patterns. On the demo data it identifies a recurring Payment API pattern involving Five related incidents, and it shows the evidence provenance behind each synthesized lesson so an engineer can see which incidents a lesson came from.

Fig 4 — Learning page showing the recurring Payment API pattern. Shows the Five related incidents and the evidence provenance for the lesson.

What I Learned Building It

Separate the evidence. Mixing current and historical information in one blob makes the AI output harder to trust. Labeling each source made it easier to review and to debug.

Design for degraded modes early. Adding the database fallback and deterministic AI fallback made the app dependable in local development, and it forced me to define what "degraded" should look like in the UI.

Be strict about what is verified. This is the part I want to be clear about:

Verified: the frontend build, backend startup, API health, the browser end-to-end workflow (Launch Demo through INC-001, INC-017, recall, resolve, retain, Learning, and demo reset), SQLite persistence, backend tests, memory recall, retention, learning/reflection, responsive behavior, and accessibility checks. The verified demo runs on SQLite with the deterministic AI fallback.
Not live-verified: a production PostgreSQL deployment, a remote Hindsight service, and live Groq/OpenAI inference. Those are configured integrations in the architecture, but I haven't tested them live, so I don't make claims about them.

I also made no performance claims, because I haven't benchmarked the system.

Conclusion

RecallOps started from a simple observation: incident knowledge is valuable and easy to lose. Using Hindsight as the memory layer, I built a workflow where past incidents are retained, recalled by similarity, and presented next to the current evidence, with the uncertainty visible and the decision left to the engineer.

The next step is validating the remote integrations (PostgreSQL, a live Hindsight service, and live LLM inference) so the configured architecture is tested too. If you're interested in agent memory, the Hindsight repository is a good place to start.