From Remembering to Searching: How ReFind Challenges AI Agent Memory
Everyone is building AI memory systems.
But a fundamental question remains:
How should an AI agent actually remember?
As agents move from simple conversations to long-term collaboration, memory becomes a critical capability.
However, building memory is not just about storing more information.
The harder problem is:
When an agent needs something from the past, how can it find the right information?
Many memory systems try to solve this by creating structured memories in advance:
- summarizing conversations;
- extracting facts;
- building knowledge structures;
- maintaining memory entries.
But this creates a difficult trade-off:
Before knowing what the user will ask in the future, how can a system know what information will matter?
This leads to another possible direction:
Instead of trying to remember everything beforehand, what if an agent could search its own history when needed?
This is the idea behind ReFind.
In the first Agent Memory Leaderboard (AML) Open Leaderboard, ReFind achieved a score of 44.97, ranking #2 among open-source memory systems.
Rather than pre-building complex memory representations, ReFind keeps raw conversation history and lets agents actively search for relevant evidence.
The Core Idea: Let Agents Search Their Own History
Humans do not perfectly remember every conversation.
When we forget something from an old chat, we usually do not reconstruct the entire conversation from memory.
We search.
We open the conversation.
We try keywords.
We check surrounding messages.
If necessary, we search again.
ReFind takes inspiration from this behavior.
Instead of deciding during the write stage what information will be important, ReFind keeps the original conversation records and allows the agent to explore them when a query arrives.
The goal changes from:
“Build the perfect memory beforehand.”
to:
“Find the right evidence when it is needed.”
How ReFind Works
ReFind combines several mechanisms to simulate human-like search behavior:
1. Multi-turn Search
Traditional retrieval often follows:
Query → Retrieve → Answer
But real information search is usually iterative.
A user may search one keyword, inspect results, refine the query, and search again.
ReFind allows the retrieval agent to decide:
- what to search;
- whether more searching is needed;
- which evidence should be kept.
This turns retrieval from a one-shot operation into an exploration process.
2. Context Browsing
A retrieved message is not always enough.
A single sentence may depend on surrounding conversation:
- previous decisions;
- explanations;
- conditions;
- follow-up results.
For example:
“Let’s use the previous approach.”
Without context, the sentence has little meaning.
ReFind therefore allows the agent to inspect surrounding messages instead of treating retrieved memories as isolated fragments.
The key idea:
Useful memory is not always one matching sentence. Sometimes it is the context around it.
3. Moving Computation from Write Time to Query Time
One of ReFind's most important design choices is where computation happens.
Traditional memory systems often process information when it is created:
Conversation
↓
Extract / Summarize
↓
Structured Memory
↓
Retrieve
ReFind takes another approach:
Conversation
↓
Keep Raw Records
↓
Search When Needed
↓
Retrieve Evidence
This creates a clear trade-off.
Pre-built memory can make future retrieval faster, but requires the system to decide early what information should be preserved.
ReFind delays this decision.
Instead of paying the cost for every conversation, it spends more computation only when a query actually requires deeper search.
The Limitation: Search Is Not Always Enough
ReFind shows that active search can be a powerful memory strategy.
But it also reveals a deeper challenge.
A search system still needs a connection between the current query and the memory it should retrieve.
Consider:
Current question:
“I want to adopt a cat.”
Historical memory:
“I have many lilies at home.”
These two statements may have little semantic similarity.
A retrieval system may search for:
- cats;
- pets;
- adoption.
But it may never find the information about lilies.
However, that memory could be critical because lilies can be dangerous for cats.
The problem is not reasoning.
The problem is:
The agent never retrieved the memory it needed.
Beyond Retrieval: The Future of Agent Memory
This challenge is explored by the ReFind team's additional benchmark, InMind.
ReFind and InMind represent two different questions:
ReFind:
How can agents search their history better?
InMind:
What if the important memory cannot be found through similarity alone?
Together, they highlight a broader challenge for Agent Memory:
Future systems may need not only better retrieval, but also better understanding of:
- when a past event matters;
- which memories should be recalled;
- and how hidden connections between memories should be discovered.
Conclusion
ReFind starts from a simple idea:
Agents may not need to remember everything. They need the ability to find what matters.
By preserving raw conversation history and enabling active search, ReFind provides an alternative direction for building AI memory systems.
The first AML results show that search-based memory can be highly competitive.
At the same time, they raise a bigger question:
Should agents only retrieve memories when asked — or should they also know when something from the past matters?
Thanks to the ReFind Team
We would like to thank the ReFind team for sharing their approach and contributing to the AML Technical Deep Dive series.
The goal of this series is to make different memory systems easier to understand, compare, and learn from — not only through leaderboard scores, but also through the ideas and engineering choices behind them.
This is the second technical deep dive in the AML series. More breakdowns of top-performing memory systems are coming soon.
ReFind GitHub Repository
https://github.com/imlrz/ReFind
InMind Benchmark
Keep It InMind: Benchmarking the Implicit-Association Blind Spot in Agent Memory
https://arxiv.org/abs/2607.24368
InMind GitHub Repository
https://github.com/imlrz/InMind
Agent Memory Leaderboard (AML)
https://huggingface.co/spaces/agent-memory-leaderboard/leaderboard
Top comments (1)
The framing assumes search and memory are different things. Search is a form of memory. It just trades state for latency. A vector index of past interactions is a memory with a retrieval cost. A persistent state is a memory with a storage cost. The real question is not whether agents need memory. It is whether the retrieval layer can answer the question the agent actually has, or whether the agent needs to carry state because the retrieval layer cannot. Most 'search instead of memory' proposals work beautifully until the question is 'what was I about to do before I got interrupted'. Which is a state question, not a retrieval question.