Designing Hindsight Retain and Recall for SRE Incidents

Devalla Harika — Mon, 28 Sep 2026 11:09:51 +0000

An incident-response system can retrieve the right information and still fail to learn anything.

That was one of the design problems behind OpsMind. We wanted an AI SRE agent that could not only recall previous incident experiences while investigating a failure, but also retain the outcome of a resolved incident so that the experience could influence future investigations.

The resulting workflow is built around two operations:

Recall before diagnosis. Retain after resolution.

Hindsight provides the persistent memory layer that connects those two stages.

The Role of Memory in OpsMind

OpsMind receives an incident containing information such as its service, severity, symptoms, and current telemetry.

The agent then gathers current evidence from logs and metrics.

Only after establishing the current situation does it query Hindsight for historical context.

The overall flow is:

Incident → Evidence → Hindsight Recall → AI Diagnosis → Human Approval → Resolution → Hindsight Retain

The important part is the position of memory in this workflow.

Memory is not the first source the agent consults, and it is not the final authority. It sits between current evidence collection and reasoning as a source of additional operational context.

Figure 1 — Hindsight connects historical incident experience to the current incident-response workflow.

Designing the Recall Query

A useful memory system depends heavily on what you ask it to retrieve.

For OpsMind, the recall query includes details from the current incident:

Incident ID
Service
Severity
Symptoms
Current metrics
Performance behavior
Error patterns
Resource saturation
Previous remediation experience

For example, a payment API incident with high latency, HTTP 500 errors, and high database connection utilization should retrieve previous incidents involving similar operational behavior.

The purpose is not to search for an identical incident ID.

The purpose is to find relevant operational experience.

The query therefore asks Hindsight to identify previous SRE incidents with similar symptoms, service behavior, performance problems, errors, resource saturation, remediation experiences, and outcomes.

That gives the AI agent historical context without hardcoding a specific previous incident as the answer.

The Recall Implementation

The memory layer uses Hindsight's asynchronous API:

python
response = await hindsight_client.arecall(
bank_id=HINDSIGHT_BANK_ID,
query=query,
)

The result is then processed before being passed into the diagnosis pipeline.

One important step is excluding the current incident from the historical results.

The current incident should never appear as if it were already historical knowledge.

OpsMind also extracts incident identifiers from returned memory and removes duplicates. This gives the reasoning layer a cleaner historical context.

The result is effectively a list of previous incident experiences that may be relevant to the current investigation.

Why Recall Cannot Be the Source of Truth

One of the easiest mistakes in an AI incident-response system would be to let memory override current telemetry.

Imagine that an old incident had a database connection problem and was resolved by increasing the connection pool.

A new incident might also mention database latency, but its actual root cause could be completely different.

If the agent blindly copies the old remediation, persistent memory becomes a source of incorrect automation.

OpsMind therefore gives current logs and metrics priority.

The system prompt explicitly instructs the AI agent to use current evidence as the primary basis for claims and historical incidents only as supporting context.

This creates an important separation:

Current telemetry answers: “What is happening now?”

Hindsight answers: “What have we experienced before?”

The AI uses both to reason about the incident.

Retain Comes After Resolution

Recall solves only half of the problem.

If OpsMind can retrieve historical experiences but never creates new ones, its knowledge becomes static.

The second half of the workflow is therefore retention.

After an incident is successfully resolved, OpsMind creates an incident learning record.

The record contains:

Incident ID
Service
AI diagnosis
Actions taken
Outcome
Resolution status
A learning statement for future incidents

The retention call looks like this:

python
hindsight_client.retain(
bank_id=BANK_ID,
content=learning_record,
context="OpsMind SRE incident learning",
metadata={
"incident_id": str(incident_id),
"service": str(service),
"type": "incident_outcome",
"successful": str(successful).lower(),
},
)

The metadata is useful because it provides structured information alongside the learning record.

INC-008 as a Learning Event

INC-008 provides a concrete example.

The Payment API experienced:

6.1-second latency
26% HTTP 500 errors
97% database connection utilization
Requests waiting for database connections

OpsMind diagnosed database connection pool exhaustion.

After the remediation was approved and simulated successfully, the outcome was retained in Hindsight.

Figure 2 — The successful INC-008 resolution is retained as an organizational learning record.

At that point, INC-008 changed from a current incident into historical organizational knowledge.

That transition is the core behavior we wanted from persistent memory.

Testing the Memory Loop

The strongest test was to investigate another incident after INC-008 had been retained.

When INC-007 was analyzed, Hindsight returned INC-008 as historical context.

Figure 3 — A later investigation retrieves the newly retained INC-008 experience.

This gave us a complete memory lifecycle:

INC-008 → Resolve → Retain → INC-007 → Recall INC-008

The important observation is that the newly retained experience was not manually copied into the second investigation.

It was available through the memory layer.

An Unexpected Engineering Issue

The memory integration also exposed an asynchronous execution issue.

The initial implementation produced:

text
Timeout context manager should be used inside a task

The problem appeared when the synchronous Hindsight recall path interacted with the asynchronous FastAPI request environment.

We changed the implementation to use an asynchronous Hindsight client and arecall() inside a dedicated asynchronous function.

The client is explicitly closed afterward:

python
await hindsight_client.aclose()
This fixed the event-loop problem and also avoided leaving the underlying client session open.

It was a useful reminder that an agent-memory integration has to fit the application's execution model, not just its logical architecture.

What We Learned

1. Recall and retain are one system

Retrieval without learning produces a static knowledge base.

Retention without retrieval produces information that cannot influence future reasoning.

The value comes from connecting both.

2. Memory quality depends on context

A vague query can return broadly related incidents.

Including service behavior, symptoms, metrics, errors, and resource saturation gives the memory system more useful retrieval context.

3. Historical context needs boundaries

Previous incidents should inform diagnosis without becoming unquestioned instructions.

The current incident remains the primary evidence source.

4. Successful outcomes are especially valuable

OpsMind retains the outcome of successful remediation because that gives future investigations information about what previously worked.

Conclusion

The central memory design in OpsMind is simple:

Recall before reasoning. Retain after learning.

Hindsight provides the persistent layer that connects those two operations.

INC-008 demonstrated the complete lifecycle. Its symptoms were analyzed using current evidence and historical context. After successful resolution, the experience was retained. A later investigation could then recall INC-008 as historical context.

That makes memory part of the incident-response lifecycle rather than a separate documentation system.

DEV Community: Devalla Harika