How ActiveMemoryIndex Ranks #3 Without Memory Governance
AI memory systems often try to make memory smarter.
They summarize conversations, extract facts, resolve conflicts, update old values, build hierarchies, and decide which memories should remain active.
But what if a memory system did almost none of that?
ActiveMemoryIndex takes a surprisingly different approach: preserve more, process less, and let the answer model decide.
In the first AML Open Leaderboard, ActiveMemoryIndex ranked #3 on the Open-Source Text track with a score of 44.84.
More interestingly, it ranked #1 on “new value overrides and current state” (54.50) and #1 on “contradiction detection and conflict resolution” (24.79) — despite having no explicit memory governance mechanism.
So how does it work?
1. A Memory System That Doesn't “Manage” Memory
ActiveMemoryIndex has only two storage operations:
CREATE and INSERT OR REPLACE.
There is no UPDATE.
There is no DELETE.
It does not merge memories, invalidate old memories, detect contradictions, or explicitly resolve conflicts.
Every stored record remains available.
Instead of deciding during the write stage which information is still valid, the system preserves the original evidence — including timestamps — and lets the reading model determine what matters.
This leads to a simple principle:
Don't decide too early what the agent will need later.
The approach trades sophisticated memory governance for information preservation.
2. Two Copies: Raw Messages + Atomic Facts
For every Add request, ActiveMemoryIndex stores two versions of the information.
Raw conversation
The original message is preserved verbatim, with:
- timestamp
- speaker role
- original wording
Extracted facts
A second representation is generated by GPT-4o-mini as first-person atomic facts.
Both are embedded and stored in the same SQLite table.
The raw version preserves information fidelity.
The fact version provides a more compact representation that can improve retrieval.
But the system does not assume that the extracted version is always better.
In fact, its own ablation experiments suggest that extraction contributes only modestly.
This reflects a broader design choice:
An imperfect summary should never become the only copy of the memory.
3. Retrieval: Make the Query Speak the Same Language as the Memory
There is another interesting problem.
The stored conversations are written from the user's perspective:
“I want to move to London.”
But the benchmark question might ask:
“Where does John want to move?”
These two expressions may refer to the same information while living in different linguistic frames.
ActiveMemoryIndex therefore rewrites the query into a first-person memory-oriented question.
For example:
“Did I mention where John wanted to move?”
The rewritten query is then combined with the original query for retrieval.
Conceptually:
Original Query
↓
Memory-oriented Query Rewrite
↓
Embedding Retrieval
↓
Combined Ranking
The authors note that this is closely related to HyDE, except that the generated representation is a memory-oriented question rather than a hypothetical answer.
4. Why Return Raw Evidence First?
After retrieval, ActiveMemoryIndex returns up to 100 pieces of evidence.
But the ordering matters.
The system places:
Raw messages first → Extracted facts second
The collection itself is not changed.
Only the ordering changes.
Why?
Because the raw message has not gone through another lossy transformation.
Consider two retrieved pieces:
“I decided to move to London because my company is opening a new office there.”
and:
“John plans to move to London.”
The second is easier to process, but the first contains the actual evidence and context.
If the answer model only receives a compressed representation, information that seemed irrelevant during extraction may already be gone.
So ActiveMemoryIndex takes a conservative approach:
Preserve the evidence. Let the model interpret it later.
5. The Counterintuitive Result: Governance Isn't Always Better
This is perhaps the most interesting finding from the system.
Memory governance sounds inherently useful.
A system could:
- detect contradictions;
- downgrade stale information;
- delete outdated memories;
- merge related facts;
- maintain a “current state.”
But ActiveMemoryIndex does none of these.
And yet it ranked #1 on the AML state-update and conflict-resolution dimensions.
The authors propose a possible explanation:
Governance only helps if it actually improves what reaches the answer model.
For example, a system might detect that an old value conflicts with a new value and reduce its ranking.
But the old value may still enter the model's context.
The model still has to decide what is current.
By contrast, preserving complete, timestamped evidence may give the reading model enough information to make the decision itself.
This is an important distinction:
Changing the ranking of evidence is not necessarily the same as resolving the conflict.
The authors present this as a hypothesis rather than a universal conclusion.
6. A Surprising Lesson About Context Size
The ActiveMemoryIndex experiments also reveal an interesting benchmark-level issue.
The authors tested returning a larger “parent block” around retrieved messages.
It initially appeared to improve performance significantly.
But when the comparison was controlled for the amount of text returned, the advantage largely disappeared.
In other words:
More context can look like better memory simply because the model was allowed to read more.
This exposes a potential limitation of evaluation setups based primarily on the number of retrieved items.
If one system returns 100 short messages while another returns 100 large blocks, the two systems are technically respecting the same item limit but giving the answer model very different amounts of information.
The authors therefore suggest that future benchmarks could consider reporting:
tokens/query + accuracy
rather than relying only on a fixed number of retrieved items.
This is an important question for memory evaluation more broadly:
Are we measuring memory quality, or how much context the system is allowed to spend?
7. What ActiveMemoryIndex Doesn't Do
The simplicity of the architecture is also its limitation.
ActiveMemoryIndex does not currently use:
- knowledge graphs
- hierarchical summaries
- entity disambiguation
- explicit memory governance
- sophisticated state management
Instead, it relies heavily on:
preserving raw evidence + retrieval + answer-model reasoning
This makes the approach lightweight, but it also leaves some difficult problems unresolved.
The weakest capability in the current submission was temporal reasoning, where the system scored 18.35.
The authors identify this as a major area for future work.
8. What's Next: From Retrieval to Temporal Memory
The next direction is not simply “more retrieval.”
The team is exploring how to represent temporal information more explicitly.
For example:
“I moved last week.”
A memory system should ideally convert that relative expression into an anchored date.
Similarly, if a user's state changes over time:
“I live in Beijing.”
followed later by:
“I moved to Shanghai.”
the system should understand the relationship between the two states rather than treating them as two unrelated facts.
This suggests a possible next step:
Raw evidence → temporal anchoring → versioned facts → better state reasoning
Interestingly, this moves ActiveMemoryIndex closer to the memory-governance layer it deliberately avoided in its current version.
9. What Can We Learn From ActiveMemoryIndex?
ActiveMemoryIndex offers a different perspective on Agent Memory.
Instead of asking:
How should we organize and govern memory?
it asks:
How much information should we preserve and let the model reason over later?
Its current results suggest that a lightweight system can remain highly competitive when it:
- preserves original evidence;
- avoids premature information loss;
- improves retrieval through query rewriting;
- gives the answer model enough context to reason about state and conflicts.
But the approach also exposes an important boundary.
Preserving everything does not automatically mean understanding everything.
As memory tasks become more temporal, stateful, and long-running, systems may eventually need to decide what should be remembered, what should expire, and how different versions of the same fact relate to one another.
That makes the trade-off between preservation and governance an interesting direction for future Agent Memory research.
Conclusion
ActiveMemoryIndex shows that a memory system does not necessarily need a complicated memory-management layer to be competitive.
Sometimes, the better strategy may be surprisingly simple:
Keep the evidence. Retrieve it well. Let the model reason.
Its #3 ranking on the first AML Open Leaderboard makes this approach particularly interesting — not because it proves that “less governance is better,” but because it raises a deeper question:
When should a memory system make decisions for the model, and when should it preserve information and let the model decide?
That is one of the trade-offs we hope to explore through the AML technical deep dive series.
Learn More
ActiveMemoryIndex GitHub:
https://github.com/linxuhao/ActiveMemoryIndex
Underlying research:
An Index, Not a Store: The Model Does Remember — It Just Needs Its Notebook
HyDE:
https://arxiv.org/abs/2212.10496
About AML
The Agent Memory Leaderboard (AML) is an open evaluation platform for AI Agent Memory systems.
Through technical deep dives like this one, we aim to look beyond leaderboard scores and understand the architectures, engineering choices, trade-offs, and limitations behind different memory systems.
One Benchmark. Real Memory.
Top comments (0)