Agent Memory Leaderboard

Posted on Aug 13

Building a Fair Benchmark for AI Agent Memory Systems

#ai #opensource #machinelearning #agents

Everyone is building AI memory systems.

But how do we know which ones actually work?

As AI agents move from one-off interactions toward long-term collaboration, memory is becoming a core capability. Yet evaluating memory systems fairly is surprisingly difficult.

Different systems often use different datasets, answer models, prompts, and evaluation methods. When the final score changes, it can be hard to tell whether the difference comes from the memory system itself or from the evaluation setup.

That's why we built Agent Memory Leaderboard (AML).

Why Do We Need a Memory Benchmark?

Memory is more than storing conversation history.

A useful memory system needs to retrieve relevant information, connect information across time, handle changing states, and provide useful context for an agent's current task.

But there hasn't been a common evaluation environment where different memory approaches can be compared under the same conditions.

AML was created to provide that common ground.

The first edition was jointly initiated by nearly 30 universities and research institutions and covers two evaluation tracks:

Open-source Methods — Text Memory
Commercial Products — Text Memory

As of August 12, 2026:

136 teams registered for the competition
67 representative memory frameworks successfully completed the first evaluation
The AML website surpassed 200,000 clicks
The website passed 100,000 clicks within the first 10 days

The first leaderboard results are now live.

Making Memory Systems More Comparable

One of the main challenges we wanted to address was evaluation consistency.

In a typical setup, a memory system may be evaluated together with a particular answer model, prompt, or judge.

That makes direct comparison difficult.

A higher score could come from a better memory system — but it could also come from a stronger downstream model or a different evaluation setup.

AML tries to separate these components.

The core interface for participating memory systems is:

Memory System
Add → Search

The memory system receives long-term history through Add, and returns relevant memories through Search.

Then AML handles:

AML Evaluation Platform
Answer → Eval

Answer generation and evaluation are completed by the benchmark platform under the same evaluation process.

This helps reduce the impact of different answer models, prompts, judges, and scoring conventions.

The goal is simple:

Compare memory systems under the same conditions as much as possible.

Memory Is More Than Retrieval

A memory system shouldn't be judged only by whether it can retrieve something that looks similar. Memory quality is not only about retrieving similar information, but about understanding relevance, context, time, and task requirements.

For text memory, AML evaluates multiple capabilities, including:

Factual recall
Relational and multi-hop reasoning
Temporal and event understanding
Memory governance
Personalization and care
Rules and workflow execution
Epistemic safety and privacy

This matters because real-world agent memory is not just a search problem.

An agent may need to understand:

What happened before?
When did it happen?
What changed afterward?
Which previous experience is relevant now?
Which information should be trusted?
How should that memory affect the current task?

A useful memory system needs to handle these questions together.

The First Results

The first AML evaluation has now been completed.

67 representative memory frameworks successfully completed the evaluation across two tracks covering text memory for both open-source methods and commercial products.

The complete rankings, scores, and system versions are available on the leaderboard.

Leaderboard:
[https://agentmemoryleaderboard.ai/leaderboard/industry/textual ]

What Comes Next?

The first leaderboard is not the finish line.

We plan to keep AML running as a long-term evaluation and public leaderboard for agent memory systems.

Going forward, we will publish deeper technical analyses of the first-round results, including:

Technical architectures that performed well
Interesting evaluation patterns
Performance across individual memory capabilities
Differences between text and code memory
What current results tell us about the evolution of agent memory

We also want the benchmark itself to evolve.

If you are working on AI agents, memory systems, evaluation, or related research, we would love to hear what you think a useful memory benchmark should measure next.

Four Evaluation Tracks

To better reflect different types of memory systems, AML organizes evaluation into two tracks:
Open-source Methods