How I Used Hindsight to Turn SRE Incidents Into Memory

Nadiya Nimra — Mon, 28 Sep 2026 07:12:29 +0000

At 2:00 AM, an incident rarely starts with a completely new problem. More often, the symptoms look familiar: a service suddenly becomes slow, database connections climb, error rates increase, and someone remembers that something similar happened a few weeks ago.

The difficult part is finding that previous incident quickly enough to make it useful.

I built Resilify.AI around this problem. It is an SRE incident investigation and remediation system that combines live telemetry with persistent organizational memory. Instead of treating every production incident as an isolated prompt, Resilify.AI uses Hindsight to recall relevant incidents, reason over their outcomes, and retain what the team learned after recovery.

The design goal was simple:

When the next incident looks like something we've already experienced, the agent should be able to remember what happened last time.

What Resilify.AI Does

The important design decision here is that telemetry becomes a structured incident before it reaches the reasoning layer.

That gives the rest of the system a consistent representation of an outage.

The Interesting Part: Giving the Agent Organizational Memory

A stateless language model can reason about the current incident, but it does not automatically know how the same organization handled similar incidents in the past.

That was the main reason I introduced Hindsight.

I use Hindsight as the persistent memory layer for Resilify.AI. Its documentation describes the memory primitives I use for recalling previous incidents and retaining the results of resolved incidents. I also found Vectorize's explanation of agent memory useful when thinking about how operational experience should become part of an agent's future context.

I use an incident memory bank:

src-incidents-bank

The memory service supports both cloud-backed Hindsight and a local development fallback.

One design decision I found useful was making the storage mode configurable rather than hard-coding a single environment.

The service checks the environment before deciding which implementation to use:

get mode() {
const envMode =
(process.env.HINDSIGHT_MODE || '').toLowerCase();

if (
envMode === 'cloud' &&
process.env.HINDSIGHT_API_KEY &&
!process.env.HINDSIGHT_API_KEY.startsWith('paste_')
) {
return 'cloud';
}

return 'local';
}

This kept local development simple while allowing the same service to connect to Hindsight Cloud.

Recalling Historical Incidents

When a new incident arrives, the agent searches the historical memory for relevant experiences.

The cloud path sends the incident query to the Hindsight memory bank:

if (this.mode === 'cloud' && this.apiKey) {
try {
const response = await fetch(
${this.apiUrl}/v1/banks/${bank}/recall,
{
method: 'POST',
headers: {
'Content-Type': 'application/json',
'Authorization': Bearer ${this.apiKey}
},
body: JSON.stringify({
query,
limit: options.limit || 3
})
}
);

if (response.ok) {
  const cloudResult = await response.json();

  return {
    ...cloudResult,
    mode: 'cloud'
  };
}

} catch (err) {
// fallback handling
}
}

The important point is not simply that old incidents are stored.

The current incident is actively used as the query for finding relevant historical experience.

For example, an incident involving:

payment-gateway
high latency
database connections
recent deployment

can retrieve an earlier incident involving similar symptoms.

Why Memory Changes the Investigation

The important question was not:

"Can I store an incident in memory?"

The more interesting question was:

"Does remembering the incident actually change what the agent does next?"

That became the central design principle.

Suppose the current telemetry shows:

DB Connections: 100/100
Error Rate: 38.7%
P99 Latency: 4820 ms
Recent Deployment: v2.8.1

The agent retrieves a historical incident with a similar pattern.

The previous incident contained:

Root Cause:
Hikari CP connection pool starvation

Remediation:
RB-PAY-04

Outcome:
Recovery verified

The current agent can now use that historical evidence when generating its hypothesis.

Instead of simply saying:

"Check the database."

it can explain why connection-pool exhaustion is a likely cause and show the historical incident supporting that hypothesis.

This is the difference between storing memory and using memory.

Separating Evidence From Diagnosis

I deliberately avoid presenting the model's diagnosis as absolute truth.

The interface uses:

LIKELY ROOT CAUSE

rather than:

ROOT CAUSE

The reasoning layer combines current telemetry with historical evidence.

For example:

Likely Root Cause:
Database connection pool exhaustion

Evidence:

DB connections reached 100/100
P99 latency increased sharply
Error rate increased
Recent deployment occurred
Similar historical incident was resolved using the same remediation pattern

This makes the reasoning inspectable.

An engineer can see both the recommendation and the evidence behind it.

Keeping an Engineer in the Loop

I did not want the system to blindly execute infrastructure changes.

The workflow therefore includes an explicit approval boundary.

The engineer can inspect the recommendation before allowing the remediation to proceed.

For example:

Recommended Runbook: RB-PAY-04

Increase database connection capacity
Restart the connection service
Monitor error rate
Monitor latency
Verify service health

[ APPROVE & EXECUTE ]

[ REJECT ]

Remediation runs through a controlled execution layer rather than directly modifying production infrastructure.

Verifying Recovery Instead of Assuming It

Executing a remediation command does not prove that an incident is fixed.

Resilify.AI therefore checks the telemetry after remediation.

A successful recovery should be visible in the system:

Error Rate

38.7% → 24.2% → 8.1% → 1.4% → 0.3%

P99 Latency

4820 ms → 45 ms

Health probes are also checked before the incident is marked as recovered.

The final state becomes:

RECOVERY VERIFIED

This gives the system an actual outcome to remember.

Closing the Loop

The post-mortem captures what happened and what the system learned.

Lesson:

High database connection utilization combined with increased latency after deployment is a strong indicator of connection pool exhaustion.

The post-mortem is then retained in Hindsight.

The code path that closes the loop is conceptually:

if (this.mode === 'cloud' && this.apiKey) {
try {
const response = await fetch(
${this.apiUrl}/v1/banks/${bank}/memories,
{
method: 'POST',
headers: {
'Content-Type': 'application/json',
'Authorization': Bearer ${this.apiKey}
},
body: JSON.stringify({
content,
metadata
})
}
);

if (response.ok) {
  const cloudData = await response.json();

  this.saveLocalMemory(
    bank,
    content,
    metadata
  );

  return {
    ...cloudData,
    mode: 'cloud'
  };
}

} catch (err) {
// fallback handling
}
}

The important idea is that the resolution becomes part of the agent's future context.

The next time a similar incident occurs, this experience becomes available during investigation.

The Complete Learning Loop

The resulting architecture is more than:

Incident → LLM → Answer

It becomes:

Remember → Investigate → Act → Verify → Learn

The system therefore treats every resolved incident as another piece of operational experience.

That is the part of the design I found most interesting.

The agent does not become useful simply because it can generate troubleshooting text.

It becomes more useful when previous outcomes influence future investigations.

What I Learned

Memory is useful only when it changes decisions

It is easy to build a system that retrieves old information.

The harder and more important question is whether that information changes the current recommendation.

For Resilify.AI, historical incidents are useful only when they provide evidence that affects the current investigation.

Retrieval and reasoning are different problems

Finding a similar incident and deciding what that incident means are not the same operation.

That is why the system separates historical retrieval from reasoning over historical experience.

Evidence makes AI recommendations easier to trust

Instead of presenting a diagnosis as certain, I show the evidence supporting a likely root cause.

This gives an engineer something concrete to inspect before approving remediation.

The outcome is part of the memory

A remediation without an outcome is incomplete.

The system needs to know whether the recommended action actually worked.

That outcome is what makes the historical incident useful to future investigations.

The best memory systems close the loop

The most important architectural decision was connecting the beginning and end of the incident lifecycle.

The system therefore has a simple mechanism for accumulating operational experience over time.

LIMITATION

One limitation I ran into was that persistent memory is only useful when the retrieved experience is relevant to the current incident.

A historical incident can provide strong supporting evidence, but it cannot replace current telemetry.

That is why Resilify.AI treats historical memory as evidence for investigation rather than as an unquestionable diagnosis.

The current incident still has to be verified through its own telemetry and recovery checks.

Conclusion

Building Resilify.AI changed how I think about AI agents for operational systems.

A language model can already explain logs, suggest commands, and generate troubleshooting steps.

The harder problem is giving it useful organizational experience.

With Hindsight, a resolved incident can become persistent memory that is available during future investigations.

That changes the role of the agent from a system that simply answers questions into a system that can use the organization's previous experience.

The core loop is simple:

Remember.

Investigate.

Act.

Verify.

Learn.

That is the foundation of Resilify.AI.

AI Assistance Disclosure

I used AI tools to assist with editing and structuring this article. The project implementation, technical decisions, and results described here are based on my own work.

DEV Community: Nadiya Nimra

[Boost]