How Hindsight Changed Our Incident Response Workflow

srivani golla — Mon, 28 Sep 2026 10:31:34 +0000

How Hindsight Changed Our Incident Response Workflow

An incident isn't really solved if the next engineer has to rediscover the same solution.

That was the problem I wanted to address while building OpsMind, an AI SRE incident response agent designed to combine current incident evidence with persistent organizational memory. Instead of treating every incident as an isolated investigation, OpsMind records successful incident outcomes and makes them available when similar failures appear again.

The interesting part is not simply adding memory to an AI agent. It is deciding what the agent should trust, when memory should be consulted, when a human should intervene, and how a resolved incident becomes useful knowledge for the next one.

What OpsMind Does

The basic incident-response workflow is straightforward:

Incident
   ↓
Current Logs + Metrics
   ↓
Hindsight Recall
   ↓
AI SRE Diagnosis
   ↓
Human Approval
   ↓
Simulated Remediation
   ↓
Hindsight RETAIN
   ↓
Future Incident

OpsMind uses two different information sources during diagnosis.

The first is the current incident evidence: logs, metrics, symptoms, severity, latency, error rates, and resource utilization. This is the primary evidence because it describes what is happening now.

The second is historical memory stored in Hindsight. This provides supporting context from previous incidents with related symptoms, services, failure patterns, remediation steps, and outcomes.

The distinction is important. Historical memory should help an agent investigate an incident, not replace the evidence from the current incident.

OpsMind separates current incident evidence from historical memory, adds a human approval gate before remediation, and retains successful outcomes for future investigations.*

Current Evidence Comes First

For every incident, OpsMind first collects the available evidence from the incident data, logs, and metrics.

It then constructs a query for Hindsight using the current incident's characteristics. For example, the query can contain the service, severity, symptoms, latency, error rate, resource saturation, and other observable behavior.

The recall operation is implemented asynchronously:

response = await hindsight_client.arecall(
    bank_id=HINDSIGHT_BANK_ID,
    query=query,
)

The returned memories are processed before they are passed to the reasoning layer. The current incident is excluded from the historical results, and incident identifiers are extracted so that the system can present useful historical context rather than an unstructured collection of memories.

This separation also prevents a subtle failure mode: allowing a historical incident to become the answer to the current incident.

OpsMind deliberately sends the reasoning model the current incident's symptoms and telemetry without directly providing its stored root cause, resolution, or outcome. The model therefore has to reason from the current evidence while using Hindsight memories as additional context.

That design reflects a simple principle:

Memory should improve investigation, not replace investigation.

A Concrete Incident: INC-008

The behavior becomes clearer with a real incident from the system: INC-008.

INC-008 affected the payment-api service. The incident contained several strong signals:

API latency increased to 6.1 seconds
HTTP 500 errors increased to 26%
Database connection utilization reached 97%
Requests were waiting longer to acquire database connections
Logs reported delays while acquiring database connections

The current evidence pointed toward a database connection pool problem.

At the same time, OpsMind queried Hindsight for previous incidents involving similar symptoms and service behavior. The returned historical context included incidents such as INC-007, INC-006, and INC-001.

The AI SRE agent then combined the two sources.

The current logs and metrics supported the diagnosis. The historical incidents provided additional information about how similar situations had been investigated and resolved previously.

INC-008 analysis showing current incident evidence, AI diagnosis, confidence, and historical context retrieved through Hindsight.*

The resulting diagnosis identified database connection pool exhaustion causing connection contention and timeouts.

The important detail is that Hindsight was not treated as the source of truth. The diagnosis was grounded in the current incident's database connection utilization, connection-acquisition delays, HTTP errors, and latency.

Diagnosis Should Not Automatically Mean Execution

Finding a likely root cause is only one part of incident response.

OpsMind separates diagnosis from remediation.

The analysis endpoint returns a diagnosis and explicitly waits for approval:

"awaiting_approval": True

The agent can recommend actions, but it does not automatically execute them.

For INC-008, the recommended remediation was to increase database connection pool capacity and restart the affected Payment API.

The workflow therefore becomes:

AI Diagnosis
      ↓
Recommended Runbook
      ↓
Human Approval
      ↓
Simulated Remediation

This human approval gate is intentional. An AI system can identify a plausible remediation without having enough operational context to safely execute it. Keeping the approval decision outside the model makes the boundary explicit.

The current implementation also keeps remediation simulated. No production infrastructure is changed by the runbook execution.

That makes the system useful for demonstrating the reasoning and memory workflow while maintaining a clear operational safety boundary.

The Resolution Becomes Memory

The most important part of the workflow happens after the incident is successfully resolved.

OpsMind does not stop at:

Incident → Diagnosis → Resolution

It records the experience back into Hindsight.

The learning layer creates an incident learning record containing information such as the incident ID, service, AI diagnosis, actions taken, outcome, and whether the resolution was successful.

The record is then retained:

hindsight_client.retain(
    bank_id=BANK_ID,
    content=learning_record,
    context="OpsMind SRE incident learning",
    metadata={
        "incident_id": str(incident_id),
        "service": str(service),
        "type": "incident_outcome",
        "successful": str(successful).lower(),
    },
)

For INC-008, the successful outcome becomes part of OpsMind's organizational memory.

INC-008 after successful simulated remediation, showing the incident resolved and the outcome retained as organizational memory.*

This changes the role of memory.

It is no longer just a database of old incidents. It becomes a feedback mechanism:

Incident
   ↓
Investigation
   ↓
Resolution
   ↓
Learning
   ↓
Persistent Memory

The Real Test: Can a Future Incident Use It?

Storing a memory is not enough.

The more meaningful test is whether a later investigation can actually use that experience.

After INC-008 was successfully retained, I investigated INC-007.

This time, Hindsight returned INC-008 as part of the historical context.

The system could therefore see the relationship between the current incident and a recently resolved historical incident:

INC-008
   ↓
Successful Resolution
   ↓
Hindsight RETAIN
   ↓
Historical Memory
   ↓
INC-007 Investigation

The learned remediation from the previous incident could now be considered during the new investigation.

INC-007 analysis showing the previously retained INC-008 experience being retrieved as historical context.*

This was the behavior I was looking for.

The value of persistent memory is not that the agent remembers something. The value is that the remembered experience changes what the agent can consider during a future investigation.

One Engineering Problem I Had to Solve

The Hindsight integration also exposed an implementation issue that was easy to miss initially.

The first version of the memory layer used synchronous Hindsight recall inside the FastAPI request flow. This produced an event-loop error:

Timeout context manager should be used inside a task

The problem was related to how the synchronous client interacted with the asynchronous web application lifecycle.

I changed the recall path to use Hindsight's asynchronous API and created a fresh client for the asynchronous operation:

response = await hindsight_client.arecall(
    bank_id=HINDSIGHT_BANK_ID,
    query=query,
)

The client is explicitly closed after the operation:

await hindsight_client.aclose()

The public memory function can then run the asynchronous operation safely from the surrounding synchronous code where required.

This was a useful reminder that integrating an AI memory system is not only about prompt design. Client lifecycle, asynchronous execution, error handling, and data boundaries matter just as much.

What Changed in the Workflow?

Without persistent memory, the basic pattern looks like this:

Incident
   ↓
Investigate
   ↓
Fix
   ↓
Close

With OpsMind, the workflow becomes:

Incident
   ↓
Current Evidence
   +
Historical Memory
   ↓
AI Diagnosis
   ↓
Human Approval
   ↓
Remediation
   ↓
Successful Outcome
   ↓
Hindsight RETAIN
   ↓
Future Incident
   ↓
Historical Learning Reused

The difference is not simply another AI component.

The difference is the feedback loop.

Lessons Learned

1. Memory needs a clear role

Adding retrieval does not automatically make an agent better. Hindsight works more reliably in this workflow because its role is explicitly defined as supporting context.

Current logs and metrics remain the primary evidence.

2. Historical experience should not become an automatic answer

A previous incident may look similar without being identical.

OpsMind therefore asks the reasoning model to use historical incidents as supporting evidence rather than blindly copying their diagnosis or remediation.

3. Diagnosis and execution should be separate

An agent can recommend an action without being allowed to execute it.

The explicit human approval stage provides a clear control boundary between AI reasoning and operational changes.

4. Successful outcomes are more valuable when they can be reused

The strongest demonstration of memory was not retaining INC-008.

It was seeing INC-008 appear when investigating INC-007.

That turns memory from passive storage into reusable operational knowledge.

5. Real engineering problems exist outside the model

The Hindsight event-loop issue was unrelated to the quality of the model's reasoning. It came from integrating asynchronous memory operations into the application lifecycle.

AI systems still depend on ordinary software engineering fundamentals.

Limitations and Next Steps

OpsMind currently focuses on the incident reasoning and persistent-memory workflow rather than directly controlling production infrastructure.

The incident logs and metrics used by the system are represented through the project's incident data, and remediation remains simulated. A production deployment would require integrations with real observability systems, infrastructure platforms, authentication, authorization, audit logging, stronger failure handling, and carefully scoped execution permissions.

Another limitation is memory quality.

If irrelevant or poorly structured incidents are retained, retrieval can become less useful. Persistent memory therefore needs the same engineering discipline as any other data system: useful records should be retained, metadata should be meaningful, and retrieved context should be evaluated rather than trusted automatically.

A production version could also expand the memory model to include richer post-incident information such as contributing factors, validation steps, rollback decisions, and operator feedback.

Conclusion

The main idea behind OpsMind is simple:

Solve an incident
      ↓
Remember what worked
      ↓
Reuse that experience
      ↓
Improve the next investigation

The interesting part of an AI SRE agent is therefore not only whether it can diagnose one incident.

It is whether the system can learn from an incident without losing the distinction between current evidence and historical experience.

In OpsMind, INC-008 demonstrates that loop: current telemetry leads to a diagnosis, a human approves the simulated remediation, the successful outcome is retained in Hindsight, and a later investigation of INC-007 can retrieve that experience.

That is the behavior I wanted from persistent agent memory: not just remembering the past, but making the past useful when the next incident arrives.

DEV Community: srivani golla

How Hindsight Changed Our Incident Response Workflow

How Hindsight Changed Our Incident Response Workflow

What OpsMind Does

Current Evidence Comes First

A Concrete Incident: INC-008

Diagnosis Should Not Automatically Mean Execution

The Resolution Becomes Memory

The Real Test: Can a Future Incident Use It?

One Engineering Problem I Had to Solve

What Changed in the Workflow?

Lessons Learned

1. Memory needs a clear role

2. Historical experience should not become an automatic answer

3. Diagnosis and execution should be separate

4. Successful outcomes are more valuable when they can be reused

5. Real engineering problems exist outside the model

Limitations and Next Steps

Conclusion

Learn More