The first challenge in building an AI-powered incident investigation system is not always generating an answer. Sometimes, it is deciding which information the AI should actually use.
Consider a Payment Service experiencing a database connection timeout. An AI memory system may retrieve hundreds of records related to the same issue. Some contain useful historical information, while others are repeated descriptions of incidents already stored.
Giving all those records directly to an LLM can make its analysis unnecessarily complicated.
This is the problem addressed by RECALLOPS, an intelligent incident investigation system that combines historical memory retrieval, duplicate detection, relevance scoring and AI-powered reasoning.
The system uses Hindsight to remember previous incidents, processes the retrieved information and uses Groq to generate an investigation report.
The interesting part is what happens between retrieving a memory and sending it to the AI model.
A LOOK AT THE RECALLOPS DASHBOARD:
WHAT RECALLOPS DOES?
RECALLOPS follows a simple but structured workflow.
An incident is submitted through the React frontend with details such as service name, problem description, symptoms and severity.
The FastAPI backend processes the request and communicates with Hindsight to retrieve related historical information.
The retrieved memories then pass through a preparation pipeline before being supplied to the AI model.
THE OVERALL STRUCTURE
Incident Form (React)
|
v
FastAPI Backend
|
v
Hindsight Memory Bank
|
v
Retrieved Memories
|
v
Deduplicate and Classify
|
v
Relevance Scoring
|
v
Group by Incident ID
|
v
Groq AI Model
|
v
Incident Analysis Report
|
v
SQLite Database
|
v
Retain for Future Recall
The technology stack consists of:
- FastAPI: Handles backend requests and API communication.
- React: Provides the interactive frontend.
- Hindsight: Manages long-term AI memory.
- Groq: Generates incident investigation reports.
- SQLite: Stores incident information and dashboard records.
- Python: Connects the different components.
Hindsight provides three important capabilities: retain, recall and reflect.
retain stores information, recall retrieves relevant memories, and reflect identifies patterns across stored memories.
Retaining incidents as structured memories
Simply storing an incident description is not enough.
The system needs to retain information in a format that can be useful during future investigations.
An incident memory contains:
- Incident ID
- Service name
- Problem description
- Symptoms
- Confirmed root cause
- Confirmed resolution
- Outcome
- AI investigation
For example, a Payment Service incident can be represented as:
incident_memory = f"""
NEW PRODUCTION INCIDENT
Incident ID: {incident.incident_id}
Service: {incident.service}
Problem: {incident.problem}
Symptoms: {incident.symptoms}
Confirmed Root Cause:
{incident.root_cause or "Not yet identified"}
Confirmed Resolution:
{incident.resolution or "Not yet applied"}
Confirmed Outcome:
{incident.outcome or "Pending"}
AI Investigation:
{analysis}
AI-generated findings are hypotheses
unless verified by an engineer.
"""
client.retain(
bank_id=BANK_ID,
content=incident_memory
)
One important detail is the Incident ID.
It helps connect different memories belonging to the same incident. Without a stable identifier, the AI may interpret multiple descriptions of one incident as separate incidents.
Another important consideration is separating confirmed information from AI-generated suggestions. An AI hypothesis should not automatically become a verified historical fact.
The interesting challenge: 126 memories to 15
RECALLOPS performs historical similarity analysis by comparing the current incident with previously stored memory records. In this test, Hindsight retrieved 126 records, from which the system selected 15 memories for further investigation.
The Similarity Intelligence Report groups the selected memories into linked historical incidents, unlinked evidence, investigation guidance and unrelated incidents.
This classification helps distinguish genuinely related incidents from records that merely share similar symptoms. It also prevents unrelated historical information from being treated as confirmed evidence.
This is one of the most important parts of RECALLOPS.
During a Payment Service connection timeout investigation, Hindsight returned 126 candidate memory records.
At first, this looked useful. However, many records contained repeated information.
New Incident – Users submit incident details for AI-powered investigation.
Why does this happen?
When an incident is retained, Hindsight can extract multiple facts from the same content. Repeated investigations may also retain related information again.
As a result, one incident can produce several similar memory records.
Sending all 126 records to Groq would introduce unnecessary repetition.
The solution is a memory preparation function called prepare_historical_memories.
Its responsibilities include:
- Removing unnecessary date and location metadata.
- Detecting exact duplicates.
- Identifying near-duplicate memories.
- Classifying memory categories.
- Calculating relevance scores.
- Selecting the top 15 memories.
The duplicate detection uses conservative similarity conditions.
def are_conservative_duplicates(
containment_score,
jaccard_score,
sequence_score
):
return (
containment_score >= 0.90
and (
jaccard_score >= 0.75
or sequence_score >= 0.85
)
)
The relevance scoring process gives additional importance to memories connected to the requested service.
For example:
if service and service in memory_lower:
relevance_score += 5
if category == "HISTORICAL_INCIDENT":
relevance_score += 3
if incident_ids:
relevance_score += 2
After processing, the system selects the highest-ranked memories:
selected_memories = processed_memories[:15]
The result is a smaller collection of information that is more useful for the current investigation.
The duplicate detection is deliberately conservative. Two memories may look similar but still contain different technical details. Removing such information could affect the final investigation.
Making AI analysis traceable
Selecting relevant memories is only one part of the problem.
The next challenge is ensuring that the AI model uses the retrieved information responsibly.
RECALLOPS assigns each selected memory a reference such as:
- HM-1
- HM-2
- HM-3
- HM-4
These references allow the AI-generated report to identify the historical information supporting its conclusions.
The system also groups memories using their incident IDs.
This is important because several memories may describe different aspects of the same production incident.
The AI is instructed to follow evidence-based rules:
1. Support historical factual claims
with memory references.
2. Never invent memory IDs
or incident IDs.
3. Similar symptoms do not prove
identical root causes.
4. A resolution is confirmed only
when the memory says it was applied.
5. Missing information must be
reported as "Not documented".
The model is also configured with a low temperature of 0.1.
Automatic production changes are not recommended. Engineers must review and approve suggested fixes.
This makes the system more suitable for assisting incident investigation rather than independently modifying production environments.
AI ANALYSIS:
Reflect: Finding recurring problems
Recall and Reflect serve different purposes.
Recall retrieves individual memories relevant to a particular question.
RECALLOPS provides a dedicated AI Memory Intelligence interface that allows users to search previously stored incident information. Using Hindsight Memory Search, users can enter queries related to specific services or problems and retrieve relevant historical knowledge.
The interface also provides an AI Insights feature for generating broader insights from stored information. Alongside memory search, current incident records are displayed to help users understand the relationship between ongoing problems and previously documented incidents.
Reflect examines information across multiple memories to identify broader patterns.
RECALLOPS uses AI Memory Insights to synthesize information from previously stored incidents. Instead of examining each memory independently, the system identifies recurring problems, summarizes incident history and highlights possible common causes.
In the Payment Service example, the analysis identifies repeated database connection timeouts and connection pool exhaustion across multiple incidents. It also highlights possible underlying factors, including database overload, network latency, connection leaks and cloud-provider resource limits.
These insights help engineers understand recurring operational patterns and prioritize further investigation. The identified causes remain hypotheses until verified.
For example, consider repeated Payment Service database connection failures.
A normal memory search may retrieve individual incidents. However, reflection can help identify recurring operational patterns across those incidents.
The integration is straightforward:
result = client.reflect(
bank_id=BANK_ID,
query=request.query
)
return {
"success": True,
"query": request.query,
"answer": result.text
}
RECALLOPS connects this functionality to an AI Insights feature.
Instead of simply displaying historical records, the system can generate a broader interpretation of recurring problems.
This is particularly useful when engineers want to understand whether repeated incidents are isolated events or symptoms of a common operational issue.
Testing: Before and after memory processing
Consider the following test case:
Service: Payment Service
Problem: Database Connection Timeout
Symptoms: Payment failures and connection pool reaching its maximum capacity.
Without historical memory, the AI receives the incident details without useful previous incident evidence.
With Hindsight integration, the system retrieves historical records, filters them and provides selected information to Groq.
The observed test results were:
| Measurement | Result |
|---|---|
| Retrieved memories | 126 |
| Selected memories | 15 |
| Memories linked to one confirmed incident | 3 |
| Unlinked Payment Service records | 10 |
| Investigation guidance records | 1 |
| Unrelated Authentication Service record | 1 |
The important observation is that 15 memories do not necessarily represent 15 different incidents.
Some records belong to the same incident, while others do not have a confirmed incident identifier.
Grouping these records helps prevent the AI from treating every memory as an independent production failure.
The resulting investigation can identify previously observed symptoms while keeping the root cause marked as unknown when there is insufficient evidence.
This distinction is important: the system should provide possible explanations rather than inventing a confirmed root cause.
What makes this approach useful?
There are several important engineering ideas demonstrated by RECALLOPS.
1. More memory is not always better.
A large collection of repeated information can make AI analysis less useful. Filtering and ranking are essential.
2. Incident IDs matter.
Stable identifiers help connect related memories and prevent incorrect incident counting.
3. AI responses need evidence.
Memory references make it easier for engineers to trace historical claims.
4. AI-generated suggestions need verification.
A possible root cause should not be presented as a confirmed fact.
5. Recall and Reflect are different.
Recall provides historical evidence, while Reflect helps identify recurring patterns.
What can be improved next?
Although the current implementation demonstrates the main workflow, there are opportunities for further improvement.
- Introduce a feedback mechanism to verify whether recommended resolutions actually work.
- Improve automatic classification of different memory types.
- Benchmark duplicate detection with larger datasets.
- Evaluate memory retrieval accuracy across multiple services.
- Improve incident trend visualization.
- Separate AI-generated investigation suggestions from verified incident records at the time of storage.
One particularly useful improvement would be evaluating the duplicate detection thresholds against a larger collection of real incident records instead of relying entirely on manually selected thresholds.
Final thoughts
RECALLOPS demonstrates how an AI memory layer can support production incident investigation.
The combination of Hindsight, FastAPI, React, SQLite and Groq creates a workflow in which historical information can contribute to future investigations.
The most interesting takeaway is that building an intelligent AI agent is not just about connecting an LLM to a backend.
It also involves deciding what information deserves to be remembered, what should be retrieved, what can be trusted and how the final answer should be supported by evidence.
ANALYTICS DASHBOARD
The goal is simple: make every useful past incident contribute to a better-informed future investigation.
Author: INDHUJA A
Project: RECALLOPS – AI-Powered Production Incident Analysis
Acknowledgement: Credit to the original project developer and contributors for the implementation and technical work.
Top comments (0)