OnCallMemory: Building a Persistent AI Incident Response Agent with Hindsight

Sunitha — Tue, 29 Sep 2026 17:13:08 +0000

Introduction
Modern engineering teams deal with production incidents that often resemble problems they have already solved. However, incident knowledge is frequently scattered across tickets, logs, documentation and team conversations.
This inspired us to build OnCallMemory, an AI incident-response agent that uses persistent memory to help on-call engineers leverage previous incident experience.
The Problem
A typical AI assistant can analyze the current alert, but without persistent memory it has no knowledge of how similar incidents were handled previously.
This can result in generic troubleshooting recommendations even when the engineering team has already solved the same problem before.
Our Solution
OnCallMemory gives the incident-response agent persistent operational memory using Vectorize Hindsight.
The system follows three major memory operations:
Retain → Recall → Reflect
Retain
When an incident is resolved, important information such as symptoms, root cause, resolution steps and outcome is stored in Hindsight.
Recall
When a new incident arrives, OnCallMemory searches its historical memory for relevant previous incidents.
Reflect
Hindsight Reflect allows the system to reason across accumulated incident memories and identify recurring operational patterns.
Memory OFF vs Memory ON
One of the key features of our project is a side-by-side comparison.
With Memory OFF, the agent analyzes only the current incident and provides generic troubleshooting suggestions.
With Memory ON, Hindsight retrieves relevant historical incidents, allowing the agent to use previous root causes and successful resolution approaches.
Example
Consider a SEV1 checkout API latency incident where the p99 response time reaches 9200ms and application logs indicate connection-pool starvation.
Without historical memory, the agent may identify several possible causes.
With Hindsight memory, the agent can recall previous checkout API incidents and identify whether connection-pool exhaustion has previously caused similar failures and what resolution worked.
Architecture
The system consists of:
Streamlit interface
AI incident-response agent
Vectorize Hindsight memory
LLM
Incident dataset
Hindsight Retain, Recall and Reflect operations
Why Persistent Memory Matters
The key idea behind OnCallMemory is that every resolved incident can become useful knowledge for future incidents.
Instead of starting from zero, the agent can build on previous operational experience.
Conclusion
OnCallMemory demonstrates how persistent memory can make AI agents more useful for real-world engineering workflows.
The goal is simple: turn previous incident experience into persistent operational knowledge.
Built for HackwithHyderabad 3.0 using Vectorize Hindsight.

DEV Community: Sunitha

OnCallMemory: Building a Persistent AI Incident Response Agent with Hindsight