Managing AI Agent Memories: From 126 Retrieved Records to 15 Relevant Memories

Introduction

AI agents can store information from previous interactions and use it to solve future problems. However, storing information is only one part of building an effective memory system. The bigger challenge is retrieving useful information without overwhelming the AI model with repeated or irrelevant records.

In this article, I explore how an AI-powered incident investigation system called RECALLOPS manages historical memory records. The system uses Hindsight for memory storage and retrieval, FastAPI for backend development, React for the user interface, and Groq for AI-based analysis.

The main idea is simple: instead of sending every retrieved memory directly to the AI model, we first process the records, remove unnecessary repetitions, and select the most relevant information.

1. Understanding the System Architecture

RECALLOPS is designed to help investigate production incidents using information from previous incidents.

The application follows a sequence of operations:

1. The user enters incident details through the React interface.
2. FastAPI receives the incident information.
3. Hindsight retrieves related historical memories.
4. The retrieved records are filtered and organized.
5. Groq analyses the selected memories and generates an investigation report.
6. Incident information is stored in SQLite and retained in Hindsight for future investigations.

This process helps the system use previous incident information while investigating new problems.

1. Storing Incident Information

An important part of the system is retaining useful incident details.

incident_memory = f"""
NEW PRODUCTION INCIDENT
Incident ID: {incident.incident_id}
Service: {incident.service}
Problem: {incident.problem}
Symptoms: {incident.symptoms}
"""
client.retain(
    bank_id=BANK_ID,
    content=incident_memory
)

Each incident is stored with information such as:

• Incident ID
• Service name
• Problem description
• Symptoms
• Confirmed root cause
• Resolution
• Final outcome

Keeping these details separate makes it easier to identify what actually happened during an incident.

For example, if the root cause has not been identified, the system stores it as "Not yet identified" instead of leaving the field empty.

This distinction helps prevent the AI from treating missing information as confirmed information.

1. The Challenge: 126 Memories Retrieved

During one investigation involving a Payment Service connection timeout, Hindsight returned 126 memory records.

However, many of these records contained similar information with small differences.

This happened because a single incident could generate multiple extracted memory records. Repeated investigations also contributed additional information.

Sending all 126 records directly to an AI model could introduce unnecessary repetition and make the analysis less focused.

To address this issue, the system applies a memory filtering process.

1. Selecting the Most Relevant 15 Memories

Before sending information to Groq, the retrieved records go through several processing stages.

def   are_conservative_duplicates():
    return (
        containment_score >= 0.90
        and (
            jaccard_score >= 0.75
            or sequence_score >= 0.85
        )
    )

These include:

• Removing duplicate records.
• Identifying the category of each memory.
• Calculating relevance scores.
• Grouping records using incident IDs.
• Selecting the highest-ranked memories.

The duplicate detection process uses text similarity measurements to identify records that are almost identical.

The system also assigns higher relevance scores to records related to the same service and historical incidents.

After processing, only the top 15 memories are selected.

This reduces unnecessary information while preserving potentially useful historical evidence.

1. Making AI Analysis More Reliable

Selecting relevant memories does not automatically guarantee accurate results.

The AI model must also be able to identify where its information came from.

For this purpose, each selected memory receives a unique reference such as HM-1, HM-2, and HM-3.

The model is instructed to:

1. Include memory references when making historical claims.
2. Avoid generating fake incident IDs.
3. Distinguish similar symptoms from confirmed root causes.
4. Report missing information as "Not documented."
5. Avoid presenting unverified recommendations as confirmed solutions.

This approach makes it easier for engineers to check the information used by the AI.

1. Using Recall and Reflect

Hindsight provides two useful capabilities: recall and reflect.

Recall retrieves information related to a particular query. It helps identify what happened in previous incidents.

Reflect analyses information across multiple memories to identify repeated patterns and relationships.

For example, when investigating repeated Payment Service problems, reflect can help identify whether similar issues occurred on different dates and whether the same temporary solutions were used.

result = client.reflect(
    bank_id=BANK_ID,
    query=request.query
)

return {
    "success": True,
    "query": request.query,
    "answer": result.text
}

This makes it useful for understanding recurring problems instead of simply retrieving individual records.

1. Before and After Memory Filtering

Consider a Payment Service incident involving database connection timeouts and payment failures.

Without historical memory, the AI can only analyse the information supplied in the current incident report.

With historical memory, it can compare the new problem against previous records.

In the example, 126 records were retrieved, but only 15 were selected for further analysis.

The selected records were then grouped using incident IDs to avoid treating multiple memory entries as separate incidents.

The AI could identify known information, distinguish unknown root causes, and generate possible explanations based on historical records.

1. Important Learnings

While exploring this system, several important points become clear:

• Memory retrieval should not be treated as the final answer.
• Duplicate records should be filtered before reaching the AI model.
• Every incident should have a stable identifier.
• AI-generated assumptions should not be stored as confirmed facts.
• Historical claims should be connected to their original evidence.
• Recall and reflect serve different purposes in an AI memory system.

1. Current Limitations

Although the system demonstrates a useful approach, some improvements are still needed.

The memory database contains limited historical information, and the duplicate detection thresholds have not been evaluated using a large dataset.

There is also a need for a feedback mechanism to verify whether suggested solutions actually resolve incidents.

Improving these areas could make the system more reliable for real-world incident investigation.

Conclusion

Building an AI-powered incident investigation system involves more than connecting a language model to a memory database.

The quality of the information supplied to the model plays an important role in generating useful results.

By filtering repeated memories, ranking relevant records, grouping incidents, and requiring evidence references, RECALLOPS demonstrates how an AI agent can make better use of historical information.

The key takeaway is that retrieving more memories does not necessarily mean obtaining better answers. The important part is selecting the right information before asking the AI model to reason over it.

Technologies Discussed

• Python
• FastAPI
• React
• Hindsight
• Groq API
• SQLite
• Large Language Models (LLMs)