AI memory is becoming an increasingly important part of agent infrastructure.
But there is still a basic problem:
How should we actually measure memory?
It is easy to demonstrate that an agent can remember something.
It is much harder to determine whether one memory architecture is genuinely better than another.
Different memory systems can use different datasets, models, prompts, retrieval strategies, and evaluation pipelines.
That makes many existing comparisons difficult to interpret.
The first season of the** Agent Memory Leaderboard (AML)** was designed around this problem.
After evaluating 69 representative memory systems from 136 registered teams, we want to share some of the technical observations from the first season.
1. The Problem With Evaluating Memory
Consider a simple agent:
Conversation History
↓
Memory System
↓
Retrieved Memories
↓
Answer Model
↓
Evaluation
If the agent gives the correct answer, which component deserves the credit?
The memory system may have retrieved excellent evidence.
But the answer model may also be stronger.
The prompt may be better.
The judge may use a different rubric.
Even the underlying dataset may favor one architecture over another.
This creates a fundamental attribution problem.
A memory benchmark therefore needs to control as many variables as possible outside the memory layer.
2. Separating Memory From the Rest of the Agent
AML uses a clearer system boundary.
The participating memory system is responsible for:
Add → Search
The benchmark platform handles:
Answer → Eval
The idea is straightforward.
If different systems are evaluated with the same downstream answering and evaluation pipeline, differences in the final results can be more meaningfully attributed to the memory layer.
This does not make every source of bias disappear.
A benchmark still needs to consider dataset composition, judge calibration, private test sets, system versions, and other factors.
But establishing a clear evaluation contract is an important starting point.
3. Why Retrieval Alone Is Not Enough
One of the most important lessons from the first season is that AI memory is not simply a retrieval problem.
Imagine an agent remembers:
“The project uses PostgreSQL.”
Three months later, the architecture changes to:
“The project migrated to MySQL.”
A useful memory system needs more than semantic similarity.
It needs to understand that the newer information changes the state of the previous memory.
The same problem appears with:
- changing user preferences
- outdated instructions
- conflicting facts
- evolving project requirements
- previous decisions and rejected alternatives
This is why future memory systems will need stronger mechanisms for:
formation → organization → retrieval → updating → forgetting
Memory management becomes part of the intelligence of the agent.
4. Measuring Multiple Memory Capabilities
AML evaluates memory across multiple dimensions rather than reducing everything to retrieval.
The first evaluation includes dimensions such as:
Fact Recall
Can the system recover information that appeared previously?
Compositional Reasoning
Can the agent connect multiple pieces of historical information?
Temporal Reasoning
Can the system understand changes and relationships across time?
Personalization
Can it maintain an accurate representation of a user's preferences and characteristics?
Memory Governance
Can the system manage memory appropriately?
Rule & Workflow Execution
Can historical information help the agent follow established rules and workflows?
Safety & Privacy
Can memory remain useful without introducing unwanted information or interference?
These dimensions matter because memory systems can have very different capability profiles.
5. What Did the First Leaderboard Show?
Commercial Products
The first Commercial Products — Text Memory ranking was:
Rank System Score
1 MemoraX 58.02
2 MemOS 45.89
3 NTES-MEMORY-SMART 44.21
The gap between the top systems illustrates one of the benefits of having a unified evaluation environment.
Rather than comparing isolated benchmark claims from different projects, the systems are evaluated under the same framework.
But the overall score is only part of the story.
A system's capability profile can reveal much more about its underlying strengths and weaknesses.
6. Open-source Methods Show a Different Pattern
The open-source ranking was considerably tighter:
Rank System Score
1 InvMem 45.10
2 ReFind 45.00
3 ActiveMemoryIndex 44.80
The difference between the top three is relatively small.
This suggests that there is no single dominant approach to memory yet.
Different systems are exploring different combinations of retrieval, indexing, reasoning, memory organization, and query processing.
That diversity is valuable for a young research area.
A benchmark should not only identify winners.
It should help researchers understand which approaches work, under which capabilities, and where the remaining gaps are.
7. Long Context Does Not Equal Long-Term Memory
The rapid expansion of context windows raises another important question.
If a model can process millions of tokens, why do we need a separate memory layer?
Because context and memory solve different problems.
A context window answers:
What information can the model access right now?
Memory asks:
What information should the agent retain and reuse over time?
For a long-running agent, simply keeping more information in context is not necessarily a scalable solution.
The system still needs to determine:
- what is important,
- what is outdated,
- what is relevant to the current task,
- and what should influence future behavior.
The memory layer therefore becomes a mechanism for managing persistent state rather than simply extending the prompt.
8. The Next Challenge: Memory for Long-Running Agents
The hardest memory problems are likely to appear when agents operate continuously.
Consider a coding agent working on the same repository for weeks.
It may need to remember:
- architectural decisions
- coding conventions
- previous debugging attempts
- dependencies
- user preferences
- rejected approaches
- known failure modes
Without persistent memory, the agent repeatedly rediscovers this information.
With poor memory, it may retrieve outdated or contradictory information.
This makes coding agents an especially interesting environment for studying long-term memory.
The problem becomes:
How can an agent accumulate useful experience without accumulating noise?
9. What a Future Memory Benchmark Should Measure
The first season also raises questions for future evaluation.
A more complete benchmark may need to examine:
Memory quality
Does the system retrieve the right information?
Memory evolution
Can it update outdated information?
Conflict resolution
What happens when memories contradict each other?
Abstention
Can the system recognize when something is not known?
Efficiency
What are the storage, latency, and inference costs?
Long-horizon stability
Does performance degrade as memory accumulates?
Version consistency
Does a commercial memory API behave consistently as the underlying system evolves?
These are difficult problems, but they are increasingly important as memory moves from research prototypes into production agents.
10. The First Season Is Only a Starting Point
The goal of AML is not to declare a permanent winner.
AI memory is still evolving rapidly.
The first season gives us a common starting point for comparing systems and identifying where different approaches succeed or struggle.
Future seasons will need to become more comprehensive as agents become more capable and more autonomous.
We hope AML can provide an open environment where researchers and developers can:
- evaluate new memory architectures
- compare different approaches
- reproduce results
- identify weaknesses
- and build better long-term agents
The field needs more systems, more experiments, and better evaluation.
The first leaderboard is only the beginning.
Explore the First AML Season
Leaderboard:
Agent Memory Leaderboard
GitHub:
AML GitHub
Hugging Face:
AML on Hugging Face
Top comments (0)