Retrieving memory for an AI agent sounds simple:
Given the current query, find the most relevant memories.
In practice, however, semantic similarity is not always enough.
A retrieved memory may be highly similar to the query while still missing the surrounding context needed for the agent to make the right decision.
This is the problem InvMem approaches from a different angle:
Memory retrieval should not only find similar information. It should recover enough relevant context to reconstruct a useful memory.
1. Fine-grained Chunking: Breaking Memory into Smaller Units
One part of InvMem's approach is fine-grained chunking.
Instead of treating a long piece of historical information as one indivisible memory unit, InvMem breaks it down into smaller pieces.
This gives the retrieval system more flexibility:
- different parts of the same historical interaction can be retrieved independently;
- more precise information can be matched against the current query;
- irrelevant parts of a long memory are less likely to dominate retrieval.
But finer-grained retrieval also creates a new problem:
What if the most relevant chunk does not contain enough context on its own?
This leads to the next part of the design.
2. Hybrid Retrieval: Dense Retrieval + BM25
InvMem combines dense retrieval with BM25.
These two retrieval approaches capture different types of relevance.
Dense retrieval is useful for semantic similarity. It can identify memories that express similar concepts even when the wording is different.
BM25, on the other hand, is particularly useful for lexical matching and exact terms.
For memory retrieval, combining the two provides a broader retrieval signal than relying on either method alone.
The goal is not simply to find:
“Which memory looks most similar?”
but rather:
“Which memories are most likely to contain useful evidence for the current query?”
3. Weighted RRF: Combining Different Retrieval Signals
After obtaining candidates from different retrieval methods, InvMem uses Weighted Reciprocal Rank Fusion (Weighted RRF) to combine their rankings.
This allows the system to integrate signals from dense retrieval and BM25 rather than choosing one retrieval method over the other.
Conceptually, the pipeline becomes:
Query → Dense Retrieval
Query → BM25
↓
Weighted RRF
↓
Unified candidate ranking
This matters because memory relevance is rarely one-dimensional.
A memory can be semantically relevant while lacking an important keyword, or contain an exact keyword while being less relevant to the overall meaning of the query.
Combining the two signals helps balance these cases.
4. Same-session Adjacency Expansion: Recovering Context
This is one of the more interesting parts of InvMem's approach.
A highly relevant memory chunk does not necessarily contain the entire context needed by the agent.
Historical conversations often contain information that is distributed across neighboring turns.
For example:
Turn A: The user describes a problem.
Turn B: The agent proposes a solution.
Turn C: The user confirms what worked.
If retrieval only returns Turn B, the agent may know the solution but not fully understand why it was proposed or whether it was actually validated.
InvMem therefore performs same-session adjacency expansion after retrieval.
The retrieved chunk can bring in neighboring information from the same session, helping reconstruct a more complete context.
This reflects an important distinction:
The most similar chunk is not necessarily the most useful memory.
Sometimes the useful memory is the retrieved chunk plus the context surrounding it.
5. From Similarity to Completeness
Putting these components together, InvMem's retrieval pipeline can be understood as:
Fine-grained Chunking
↓
Dense Retrieval + BM25
↓
Weighted RRF
↓
Same-session Adjacency Expansion
↓
More complete memory context
The key idea is not to maximize the amount of memory retrieved.
It is to improve the chance that the retrieved information contains the complete evidence needed by the agent.
This leads to a broader question for AI memory systems:
Should memory retrieval be optimized for similarity, or for usefulness?
InvMem's approach suggests that these two objectives are not always the same.
Why This Matters for Agent Memory
Long-term memory is becoming an increasingly important component of agent systems.
But as memory systems become more sophisticated, the retrieval problem becomes more than a standard vector-search problem.
An agent may need:
- a specific fact;
- a previous decision;
- the reasoning behind that decision;
- evidence that a solution worked;
- or the surrounding context needed to interpret a retrieved memory correctly.
This makes memory completeness an important dimension to consider alongside retrieval relevance.
The InvMem result is therefore interesting not only because it achieved Rank #1 with 45.06 on the first AML Open Leaderboard, but also because it illustrates one possible direction for moving beyond simple similarity-based retrieval.
Thanks to the InvMem Team
We’d like to thank the InvMem team for sharing their approach and contributing to the AML technical deep dive series.
The goal of this series is to make different memory systems easier to understand, compare, and learn from — not just through leaderboard scores, but through the ideas and engineering decisions behind them.
This is the first deep dive in the series. More technical breakdowns are coming soon.
Top comments (0)