AI Incident Response Agent That Learns From Past Incidents Using Hindsight

How I Built an AI Incident Response Agent That Learns From Past Incidents Using Hindsight

Building a memory-driven assistant that helps on-call engineers investigate production issues using historical incident knowledge.

Introduction

When a production service fails, engineers need to identify the cause quickly. Searching previous incident reports, comparing symptoms, and finding successful resolutions can take valuable time.

Our team built OnCall Memory, an AI-assisted incident response system that uses historical incident knowledge to support engineers during new production issues.

The project uses Hindsight as its memory layer, a Python backend for incident processing, and Groq for language-model inference.

The Problem

Imagine a payment service experiencing database connection errors during checkout. Engineers need to investigate whether the issue is caused by connection-pool exhaustion, traffic spikes, database failures, or recent deployments.

Similar problems may have occurred previously, but finding and comparing those incidents can be difficult.

Our goal was to build a system that retrieves relevant historical incidents, compares them with current symptoms, and generates an organised investigation briefing.

Introducing OnCall Memory

The application follows this workflow:

1. An engineer submits a production error.
2. The backend searches Hindsight for related memories.
3. Retrieved information is used as historical evidence.
4. Groq analyses the current error using that evidence.
5. The application displays an investigation report.

The system supports engineers rather than making production changes automatically.

Architecture and Technologies

• Python: Backend logic and memory integration.
• Hindsight: Retaining, recalling, and reflecting on incident information.
• Groq: Language-model inference for incident analysis.
• React and Vite: Frontend interface.
• Streamlit: Initial interface and testing workflow.

The key design principle is to use retrieved historical information as evidence instead of allowing the language model to invent incident history.

My Contribution: Hindsight Memory Layer

My primary responsibility was developing the memory integration in memory.py.

1. Retaining Incidents

I implemented functionality to store incident information in Hindsight so historical symptoms, affected services, and resolution details could be reused.

2. Recording Outcome Feedback

The system supports recording whether a proposed resolution worked. This adds useful context about previous troubleshooting attempts.

3. Recalling Similar Incidents

For a new error, the application retrieves potentially relevant memories. I also implemented result limits and exact duplicate removal to avoid unnecessary repetition.

4. Reflecting on Patterns

Reflection helps explore recurring patterns across stored memories, such as repeated connection failures or similar incident symptoms.

5. Reliability and Error Handling

I worked on input validation, retry handling, fallback behaviour, and secure certificate configuration using certifi.

These improvements help the application handle memory-service failures without falsely reporting that a historical search succeeded.

Exploring Hindsight World Facts

World Facts provides a way to inspect general knowledge retained by Hindsight. It helps us understand how information from historical incidents is represented.

Hindsight provides three visualization options:

• Timeline: Explores retained information chronologically.
• Table: Displays structured information and associated metadata.
• Constellation: Visualises relationships between memories and entities.

Figure 1: Hindsight World Facts Timeline view.

Figure 2: Hindsight World Facts Table view.!

Figure 3: Hindsight World Facts Constellation view.

Exploring Hindsight Observations

Observations provide a more detailed view of individual pieces of information retained by Hindsight, including incident symptoms, technical details, and resolution context.

The same three visualization options help us inspect these records:

• Timeline: Understands the sequence of recorded observations.
• Table: Makes individual observations easier to inspect.
• Constellation: Shows connections between observations and related entities.

Figure 4: Hindsight Observations Timeline view.

Figure 5: Hindsight Observations Table view.

Figure 6: Hindsight Observations Constellation view.

Practical Example: Payment-Service Incident

We tested the workflow using a payment-service scenario involving database connection errors.

The engineer submits the incident details, and the backend retrieves potentially relevant historical memories. Groq then generates a structured briefing containing:

1. Possible root causes.
2. Historical incident comparison.
3. Ordered investigation steps.
4. Possible solutions after verification.
5. Confidence and limitations.
6. Next immediate actions.

The investigation encourages engineers to verify connection-pool metrics, database connections, traffic patterns, deployments, and logs before applying a fix.

A historical match is a lead, not proof of the current root cause.

Testing and Challenges

I tested the memory layer through focused checks covering:

• Exact duplicate removal.
• Retry handling.
• Reflection and fallback behaviour.
• Outcome feedback.
• Input validation and Python syntax.
• Hindsight-to-agent-to-Groq integration.

The focused tests reported passing results, and we also tested the complete incident-analysis workflow.

One important challenge was semantic retrieval. Hindsight may return relevant memories without retrieving every incident expected. This taught us to distinguish historical evidence from verified facts and recommendations.

What Comes Next

Future improvements include better retrieval evaluation, stronger source references, confidence calibration, live dashboard metrics, authentication, and production monitoring.

Conclusion

OnCall Memory demonstrates how AI can use organisational memory to support incident investigation. Hindsight provides the memory capabilities, Python connects the system components, and Groq generates readable investigation briefings.

Working on the memory layer taught me that useful AI systems need reliable memory, relevant retrieval, proper error handling, and human verification.

The goal is not to replace on-call engineers, but to help them learn from previous incidents.

---

Project: OnCall Memory — AI Incident Response Agent powered by Hindsight.