DEV Community: Agent Memory Leaderboard

From “Managing” to “Preserving”: How ActiveMemoryIndex Keeps Memory Simple

Agent Memory Leaderboard — Wed, 26 Aug 2026 03:43:25 +0000

How ActiveMemoryIndex Ranks #3 Without Memory Governance

AI memory systems often try to make memory smarter.

They summarize conversations, extract facts, resolve conflicts, update old values, build hierarchies, and decide which memories should remain active.

But what if a memory system did almost none of that?

ActiveMemoryIndex takes a surprisingly different approach: preserve more, process less, and let the answer model decide.

In the first AML Open Leaderboard, ActiveMemoryIndex ranked #3 on the Open-Source Text track with a score of 44.84.

More interestingly, it ranked #1 on “new value overrides and current state” (54.50) and #1 on “contradiction detection and conflict resolution” (24.79) — despite having no explicit memory governance mechanism.

So how does it work?

1. A Memory System That Doesn't “Manage” Memory

ActiveMemoryIndex has only two storage operations:

CREATE and INSERT OR REPLACE.

There is no UPDATE.

There is no DELETE.

It does not merge memories, invalidate old memories, detect contradictions, or explicitly resolve conflicts.

Every stored record remains available.

Instead of deciding during the write stage which information is still valid, the system preserves the original evidence — including timestamps — and lets the reading model determine what matters.

This leads to a simple principle:

Don't decide too early what the agent will need later.

The approach trades sophisticated memory governance for information preservation.

2. Two Copies: Raw Messages + Atomic Facts

For every Add request, ActiveMemoryIndex stores two versions of the information.

Raw conversation

The original message is preserved verbatim, with:

timestamp
speaker role
original wording

Extracted facts

A second representation is generated by GPT-4o-mini as first-person atomic facts.

Both are embedded and stored in the same SQLite table.

The raw version preserves information fidelity.

The fact version provides a more compact representation that can improve retrieval.

But the system does not assume that the extracted version is always better.

In fact, its own ablation experiments suggest that extraction contributes only modestly.

This reflects a broader design choice:

An imperfect summary should never become the only copy of the memory.

3. Retrieval: Make the Query Speak the Same Language as the Memory

There is another interesting problem.

The stored conversations are written from the user's perspective:

“I want to move to London.”

But the benchmark question might ask:

“Where does John want to move?”

These two expressions may refer to the same information while living in different linguistic frames.

ActiveMemoryIndex therefore rewrites the query into a first-person memory-oriented question.

For example:

“Did I mention where John wanted to move?”

The rewritten query is then combined with the original query for retrieval.

Conceptually:

Original Query
↓
Memory-oriented Query Rewrite
↓
Embedding Retrieval
↓
Combined Ranking

The authors note that this is closely related to HyDE, except that the generated representation is a memory-oriented question rather than a hypothetical answer.

4. Why Return Raw Evidence First?

After retrieval, ActiveMemoryIndex returns up to 100 pieces of evidence.

But the ordering matters.

The system places:

Raw messages first → Extracted facts second

The collection itself is not changed.

Only the ordering changes.

Why?

Because the raw message has not gone through another lossy transformation.

Consider two retrieved pieces:

“I decided to move to London because my company is opening a new office there.”

and:

“John plans to move to London.”

The second is easier to process, but the first contains the actual evidence and context.

If the answer model only receives a compressed representation, information that seemed irrelevant during extraction may already be gone.

So ActiveMemoryIndex takes a conservative approach:

Preserve the evidence. Let the model interpret it later.

5. The Counterintuitive Result: Governance Isn't Always Better

This is perhaps the most interesting finding from the system.

Memory governance sounds inherently useful.

A system could:

detect contradictions;
downgrade stale information;
delete outdated memories;
merge related facts;
maintain a “current state.”

But ActiveMemoryIndex does none of these.

And yet it ranked #1 on the AML state-update and conflict-resolution dimensions.

The authors propose a possible explanation:

Governance only helps if it actually improves what reaches the answer model.

For example, a system might detect that an old value conflicts with a new value and reduce its ranking.

But the old value may still enter the model's context.

The model still has to decide what is current.

By contrast, preserving complete, timestamped evidence may give the reading model enough information to make the decision itself.

This is an important distinction:

Changing the ranking of evidence is not necessarily the same as resolving the conflict.

The authors present this as a hypothesis rather than a universal conclusion.

6. A Surprising Lesson About Context Size

The ActiveMemoryIndex experiments also reveal an interesting benchmark-level issue.

The authors tested returning a larger “parent block” around retrieved messages.

It initially appeared to improve performance significantly.

But when the comparison was controlled for the amount of text returned, the advantage largely disappeared.

In other words:

More context can look like better memory simply because the model was allowed to read more.

This exposes a potential limitation of evaluation setups based primarily on the number of retrieved items.

If one system returns 100 short messages while another returns 100 large blocks, the two systems are technically respecting the same item limit but giving the answer model very different amounts of information.

The authors therefore suggest that future benchmarks could consider reporting:

tokens/query + accuracy

rather than relying only on a fixed number of retrieved items.

This is an important question for memory evaluation more broadly:

Are we measuring memory quality, or how much context the system is allowed to spend?

7. What ActiveMemoryIndex Doesn't Do

The simplicity of the architecture is also its limitation.

ActiveMemoryIndex does not currently use:

knowledge graphs
hierarchical summaries
entity disambiguation
explicit memory governance
sophisticated state management

Instead, it relies heavily on:

preserving raw evidence + retrieval + answer-model reasoning

This makes the approach lightweight, but it also leaves some difficult problems unresolved.

The weakest capability in the current submission was temporal reasoning, where the system scored 18.35.

The authors identify this as a major area for future work.

8. What's Next: From Retrieval to Temporal Memory

The next direction is not simply “more retrieval.”

The team is exploring how to represent temporal information more explicitly.

For example:

“I moved last week.”

A memory system should ideally convert that relative expression into an anchored date.

Similarly, if a user's state changes over time:

“I live in Beijing.”

followed later by:

“I moved to Shanghai.”

the system should understand the relationship between the two states rather than treating them as two unrelated facts.

This suggests a possible next step:

Raw evidence → temporal anchoring → versioned facts → better state reasoning

Interestingly, this moves ActiveMemoryIndex closer to the memory-governance layer it deliberately avoided in its current version.

9. What Can We Learn From ActiveMemoryIndex?

ActiveMemoryIndex offers a different perspective on Agent Memory.

Instead of asking:

How should we organize and govern memory?

it asks:

How much information should we preserve and let the model reason over later?

Its current results suggest that a lightweight system can remain highly competitive when it:

preserves original evidence;
avoids premature information loss;
improves retrieval through query rewriting;
gives the answer model enough context to reason about state and conflicts.

But the approach also exposes an important boundary.

Preserving everything does not automatically mean understanding everything.

As memory tasks become more temporal, stateful, and long-running, systems may eventually need to decide what should be remembered, what should expire, and how different versions of the same fact relate to one another.

That makes the trade-off between preservation and governance an interesting direction for future Agent Memory research.

Conclusion

ActiveMemoryIndex shows that a memory system does not necessarily need a complicated memory-management layer to be competitive.

Sometimes, the better strategy may be surprisingly simple:

Keep the evidence. Retrieve it well. Let the model reason.

Its #3 ranking on the first AML Open Leaderboard makes this approach particularly interesting — not because it proves that “less governance is better,” but because it raises a deeper question:

When should a memory system make decisions for the model, and when should it preserve information and let the model decide?

That is one of the trade-offs we hope to explore through the AML technical deep dive series.

Learn More

ActiveMemoryIndex GitHub:
https://github.com/linxuhao/ActiveMemoryIndex

Underlying research:
An Index, Not a Store: The Model Does Remember — It Just Needs Its Notebook

HyDE:
https://arxiv.org/abs/2212.10496

About AML

The Agent Memory Leaderboard (AML) is an open evaluation platform for AI Agent Memory systems.

Through technical deep dives like this one, we aim to look beyond leaderboard scores and understand the architectures, engineering choices, trade-offs, and limitations behind different memory systems.

One Benchmark. Real Memory.

What If AI Agents Didn’t Need Memory? They Could Just Search Their Past

Agent Memory Leaderboard — Fri, 21 Aug 2026 06:56:06 +0000

From Remembering to Searching: How ReFind Challenges AI Agent Memory

Everyone is building AI memory systems.

But a fundamental question remains:

How should an AI agent actually remember?

As agents move from simple conversations to long-term collaboration, memory becomes a critical capability.

However, building memory is not just about storing more information.

The harder problem is:

When an agent needs something from the past, how can it find the right information?

Many memory systems try to solve this by creating structured memories in advance:

summarizing conversations;
extracting facts;
building knowledge structures;
maintaining memory entries.

But this creates a difficult trade-off:

Before knowing what the user will ask in the future, how can a system know what information will matter?

This leads to another possible direction:

Instead of trying to remember everything beforehand, what if an agent could search its own history when needed?

This is the idea behind ReFind.

In the first Agent Memory Leaderboard (AML) Open Leaderboard, ReFind achieved a score of 44.97, ranking #2 among open-source memory systems.

Rather than pre-building complex memory representations, ReFind keeps raw conversation history and lets agents actively search for relevant evidence.

The Core Idea: Let Agents Search Their Own History

Humans do not perfectly remember every conversation.

When we forget something from an old chat, we usually do not reconstruct the entire conversation from memory.

We search.

We open the conversation.

We try keywords.

We check surrounding messages.

If necessary, we search again.

ReFind takes inspiration from this behavior.

Instead of deciding during the write stage what information will be important, ReFind keeps the original conversation records and allows the agent to explore them when a query arrives.

The goal changes from:

“Build the perfect memory beforehand.”

to:

“Find the right evidence when it is needed.”

How ReFind Works

ReFind combines several mechanisms to simulate human-like search behavior:

1. Multi-turn Search

Traditional retrieval often follows:

Query → Retrieve → Answer

But real information search is usually iterative.

A user may search one keyword, inspect results, refine the query, and search again.

ReFind allows the retrieval agent to decide:

what to search;
whether more searching is needed;
which evidence should be kept.

This turns retrieval from a one-shot operation into an exploration process.

2. Context Browsing

A retrieved message is not always enough.

A single sentence may depend on surrounding conversation:

previous decisions;
explanations;
conditions;
follow-up results.

For example:

“Let’s use the previous approach.”

Without context, the sentence has little meaning.

ReFind therefore allows the agent to inspect surrounding messages instead of treating retrieved memories as isolated fragments.

The key idea:

Useful memory is not always one matching sentence. Sometimes it is the context around it.

3. Moving Computation from Write Time to Query Time

One of ReFind's most important design choices is where computation happens.

Traditional memory systems often process information when it is created:

Conversation
↓
Extract / Summarize
↓
Structured Memory
↓
Retrieve

ReFind takes another approach:

Conversation
↓
Keep Raw Records
↓
Search When Needed
↓
Retrieve Evidence

This creates a clear trade-off.

Pre-built memory can make future retrieval faster, but requires the system to decide early what information should be preserved.

ReFind delays this decision.

Instead of paying the cost for every conversation, it spends more computation only when a query actually requires deeper search.

The Limitation: Search Is Not Always Enough

ReFind shows that active search can be a powerful memory strategy.

But it also reveals a deeper challenge.

A search system still needs a connection between the current query and the memory it should retrieve.

Consider:

Current question:

“I want to adopt a cat.”

Historical memory:

“I have many lilies at home.”

These two statements may have little semantic similarity.

A retrieval system may search for:

cats;
pets;
adoption.

But it may never find the information about lilies.

However, that memory could be critical because lilies can be dangerous for cats.

The problem is not reasoning.

The problem is:

The agent never retrieved the memory it needed.

Beyond Retrieval: The Future of Agent Memory

This challenge is explored by the ReFind team's additional benchmark, InMind.

ReFind and InMind represent two different questions:

ReFind:

How can agents search their history better?

InMind:

What if the important memory cannot be found through similarity alone?

Together, they highlight a broader challenge for Agent Memory:

Future systems may need not only better retrieval, but also better understanding of:

when a past event matters;
which memories should be recalled;
and how hidden connections between memories should be discovered.

Conclusion

ReFind starts from a simple idea:

Agents may not need to remember everything. They need the ability to find what matters.

By preserving raw conversation history and enabling active search, ReFind provides an alternative direction for building AI memory systems.

The first AML results show that search-based memory can be highly competitive.

At the same time, they raise a bigger question:

Should agents only retrieve memories when asked — or should they also know when something from the past matters?

Thanks to the ReFind Team

We would like to thank the ReFind team for sharing their approach and contributing to the AML Technical Deep Dive series.

The goal of this series is to make different memory systems easier to understand, compare, and learn from — not only through leaderboard scores, but also through the ideas and engineering choices behind them.

This is the second technical deep dive in the AML series. More breakdowns of top-performing memory systems are coming soon.

ReFind GitHub Repository
https://github.com/imlrz/ReFind

InMind Benchmark
Keep It InMind: Benchmarking the Implicit-Association Blind Spot in Agent Memory
https://arxiv.org/abs/2607.24368

InMind GitHub Repository
https://github.com/imlrz/InMind

Agent Memory Leaderboard (AML)
https://huggingface.co/spaces/agent-memory-leaderboard/leaderboard

AML Technical Deep Dive #1: From “Similarity” to “Completeness” — How InvMem Retrieves Useful Long-Term Memory

Agent Memory Leaderboard — Thu, 20 Aug 2026 03:33:05 +0000

Retrieving memory for an AI agent sounds simple:

Given the current query, find the most relevant memories.

In practice, however, semantic similarity is not always enough.

A retrieved memory may be highly similar to the query while still missing the surrounding context needed for the agent to make the right decision.

This is the problem InvMem approaches from a different angle:

Memory retrieval should not only find similar information. It should recover enough relevant context to reconstruct a useful memory.

1. Fine-grained Chunking: Breaking Memory into Smaller Units

One part of InvMem's approach is fine-grained chunking.

Instead of treating a long piece of historical information as one indivisible memory unit, InvMem breaks it down into smaller pieces.

This gives the retrieval system more flexibility:

different parts of the same historical interaction can be retrieved independently;
more precise information can be matched against the current query;
irrelevant parts of a long memory are less likely to dominate retrieval.

But finer-grained retrieval also creates a new problem:

What if the most relevant chunk does not contain enough context on its own?

This leads to the next part of the design.

2. Hybrid Retrieval: Dense Retrieval + BM25

InvMem combines dense retrieval with BM25.

These two retrieval approaches capture different types of relevance.

Dense retrieval is useful for semantic similarity. It can identify memories that express similar concepts even when the wording is different.

BM25, on the other hand, is particularly useful for lexical matching and exact terms.

For memory retrieval, combining the two provides a broader retrieval signal than relying on either method alone.

The goal is not simply to find:

“Which memory looks most similar?”

but rather:

“Which memories are most likely to contain useful evidence for the current query?”

3. Weighted RRF: Combining Different Retrieval Signals

After obtaining candidates from different retrieval methods, InvMem uses Weighted Reciprocal Rank Fusion (Weighted RRF) to combine their rankings.

This allows the system to integrate signals from dense retrieval and BM25 rather than choosing one retrieval method over the other.

Conceptually, the pipeline becomes:

Query → Dense Retrieval
Query → BM25
↓
Weighted RRF
↓
Unified candidate ranking

This matters because memory relevance is rarely one-dimensional.

A memory can be semantically relevant while lacking an important keyword, or contain an exact keyword while being less relevant to the overall meaning of the query.

Combining the two signals helps balance these cases.

4. Same-session Adjacency Expansion: Recovering Context

This is one of the more interesting parts of InvMem's approach.

A highly relevant memory chunk does not necessarily contain the entire context needed by the agent.

Historical conversations often contain information that is distributed across neighboring turns.

For example:

Turn A: The user describes a problem.
Turn B: The agent proposes a solution.
Turn C: The user confirms what worked.

If retrieval only returns Turn B, the agent may know the solution but not fully understand why it was proposed or whether it was actually validated.

InvMem therefore performs same-session adjacency expansion after retrieval.

The retrieved chunk can bring in neighboring information from the same session, helping reconstruct a more complete context.

This reflects an important distinction:

The most similar chunk is not necessarily the most useful memory.

Sometimes the useful memory is the retrieved chunk plus the context surrounding it.

5. From Similarity to Completeness

Putting these components together, InvMem's retrieval pipeline can be understood as:

Fine-grained Chunking
↓
Dense Retrieval + BM25
↓
Weighted RRF
↓
Same-session Adjacency Expansion
↓
More complete memory context

The key idea is not to maximize the amount of memory retrieved.

It is to improve the chance that the retrieved information contains the complete evidence needed by the agent.

This leads to a broader question for AI memory systems:

Should memory retrieval be optimized for similarity, or for usefulness?

InvMem's approach suggests that these two objectives are not always the same.

Why This Matters for Agent Memory

Long-term memory is becoming an increasingly important component of agent systems.

But as memory systems become more sophisticated, the retrieval problem becomes more than a standard vector-search problem.

An agent may need:

a specific fact;
a previous decision;
the reasoning behind that decision;
evidence that a solution worked;
or the surrounding context needed to interpret a retrieved memory correctly.

This makes memory completeness an important dimension to consider alongside retrieval relevance.

The InvMem result is therefore interesting not only because it achieved Rank #1 with 45.06 on the first AML Open Leaderboard, but also because it illustrates one possible direction for moving beyond simple similarity-based retrieval.

Thanks to the InvMem Team

We’d like to thank the InvMem team for sharing their approach and contributing to the AML technical deep dive series.

The goal of this series is to make different memory systems easier to understand, compare, and learn from — not just through leaderboard scores, but through the ideas and engineering decisions behind them.

This is the first deep dive in the series. More technical breakdowns are coming soon.

Beyond Retrieval: What We Learned From the First Agent Memory Leaderboard

Agent Memory Leaderboard — Mon, 17 Aug 2026 06:40:15 +0000

AI memory is becoming an increasingly important part of agent infrastructure.

But there is still a basic problem:

How should we actually measure memory?

It is easy to demonstrate that an agent can remember something.

It is much harder to determine whether one memory architecture is genuinely better than another.

Different memory systems can use different datasets, models, prompts, retrieval strategies, and evaluation pipelines.

That makes many existing comparisons difficult to interpret.

The first season of the** Agent Memory Leaderboard (AML)** was designed around this problem.

After evaluating 69 representative memory systems from 136 registered teams, we want to share some of the technical observations from the first season.

1. The Problem With Evaluating Memory

Consider a simple agent:

Conversation History
↓
Memory System
↓
Retrieved Memories
↓
Answer Model
↓
Evaluation

If the agent gives the correct answer, which component deserves the credit?

The memory system may have retrieved excellent evidence.

But the answer model may also be stronger.

The prompt may be better.

The judge may use a different rubric.

Even the underlying dataset may favor one architecture over another.

This creates a fundamental attribution problem.

A memory benchmark therefore needs to control as many variables as possible outside the memory layer.

2. Separating Memory From the Rest of the Agent

AML uses a clearer system boundary.

The participating memory system is responsible for:

Add → Search

The benchmark platform handles:

Answer → Eval

The idea is straightforward.

If different systems are evaluated with the same downstream answering and evaluation pipeline, differences in the final results can be more meaningfully attributed to the memory layer.

This does not make every source of bias disappear.

A benchmark still needs to consider dataset composition, judge calibration, private test sets, system versions, and other factors.

But establishing a clear evaluation contract is an important starting point.

3. Why Retrieval Alone Is Not Enough

One of the most important lessons from the first season is that AI memory is not simply a retrieval problem.

Imagine an agent remembers:

“The project uses PostgreSQL.”

Three months later, the architecture changes to:

“The project migrated to MySQL.”

A useful memory system needs more than semantic similarity.

It needs to understand that the newer information changes the state of the previous memory.

The same problem appears with:

changing user preferences
outdated instructions
conflicting facts
evolving project requirements
previous decisions and rejected alternatives

This is why future memory systems will need stronger mechanisms for:

formation → organization → retrieval → updating → forgetting

Memory management becomes part of the intelligence of the agent.

4. Measuring Multiple Memory Capabilities

AML evaluates memory across multiple dimensions rather than reducing everything to retrieval.

The first evaluation includes dimensions such as:

Fact Recall

Can the system recover information that appeared previously?

Compositional Reasoning

Can the agent connect multiple pieces of historical information?

Temporal Reasoning

Can the system understand changes and relationships across time?

Personalization

Can it maintain an accurate representation of a user's preferences and characteristics?

Memory Governance

Can the system manage memory appropriately?

Rule & Workflow Execution

Can historical information help the agent follow established rules and workflows?

Safety & Privacy

Can memory remain useful without introducing unwanted information or interference?

These dimensions matter because memory systems can have very different capability profiles.

5. What Did the First Leaderboard Show?
Commercial Products

The first Commercial Products — Text Memory ranking was:

Rank System Score
1 MemoraX 58.02
2 MemOS 45.89
3 NTES-MEMORY-SMART 44.21

The gap between the top systems illustrates one of the benefits of having a unified evaluation environment.

Rather than comparing isolated benchmark claims from different projects, the systems are evaluated under the same framework.

But the overall score is only part of the story.

A system's capability profile can reveal much more about its underlying strengths and weaknesses.

6. Open-source Methods Show a Different Pattern

The open-source ranking was considerably tighter:

Rank System Score
1 InvMem 45.10
2 ReFind 45.00
3 ActiveMemoryIndex 44.80

The difference between the top three is relatively small.

This suggests that there is no single dominant approach to memory yet.

Different systems are exploring different combinations of retrieval, indexing, reasoning, memory organization, and query processing.

That diversity is valuable for a young research area.

A benchmark should not only identify winners.

It should help researchers understand which approaches work, under which capabilities, and where the remaining gaps are.

7. Long Context Does Not Equal Long-Term Memory

The rapid expansion of context windows raises another important question.

If a model can process millions of tokens, why do we need a separate memory layer?

Because context and memory solve different problems.

A context window answers:

What information can the model access right now?

Memory asks:

What information should the agent retain and reuse over time?

For a long-running agent, simply keeping more information in context is not necessarily a scalable solution.

The system still needs to determine:

what is important,
what is outdated,
what is relevant to the current task,
and what should influence future behavior.

The memory layer therefore becomes a mechanism for managing persistent state rather than simply extending the prompt.

8. The Next Challenge: Memory for Long-Running Agents

The hardest memory problems are likely to appear when agents operate continuously.

Consider a coding agent working on the same repository for weeks.

It may need to remember:

architectural decisions
coding conventions
previous debugging attempts
dependencies
user preferences
rejected approaches
known failure modes

Without persistent memory, the agent repeatedly rediscovers this information.

With poor memory, it may retrieve outdated or contradictory information.

This makes coding agents an especially interesting environment for studying long-term memory.

The problem becomes:

How can an agent accumulate useful experience without accumulating noise?

9. What a Future Memory Benchmark Should Measure

The first season also raises questions for future evaluation.

A more complete benchmark may need to examine:

Memory quality

Does the system retrieve the right information?

Memory evolution

Can it update outdated information?

Conflict resolution

What happens when memories contradict each other?

Abstention

Can the system recognize when something is not known?

Efficiency

What are the storage, latency, and inference costs?

Long-horizon stability

Does performance degrade as memory accumulates?

Version consistency

Does a commercial memory API behave consistently as the underlying system evolves?

These are difficult problems, but they are increasingly important as memory moves from research prototypes into production agents.

10. The First Season Is Only a Starting Point

The goal of AML is not to declare a permanent winner.

AI memory is still evolving rapidly.

The first season gives us a common starting point for comparing systems and identifying where different approaches succeed or struggle.

Future seasons will need to become more comprehensive as agents become more capable and more autonomous.

We hope AML can provide an open environment where researchers and developers can:

evaluate new memory architectures
compare different approaches
reproduce results
identify weaknesses
and build better long-term agents

The field needs more systems, more experiments, and better evaluation.

The first leaderboard is only the beginning.

Explore the First AML Season

Leaderboard:
Agent Memory Leaderboard

GitHub:
AML GitHub

Hugging Face:
AML on Hugging Face

Building a Fair Benchmark for AI Agent Memory Systems

Agent Memory Leaderboard — Thu, 13 Aug 2026 02:51:12 +0000

Everyone is building AI memory systems.

But how do we know which ones actually work?

As AI agents move from one-off interactions toward long-term collaboration, memory is becoming a core capability. Yet evaluating memory systems fairly is surprisingly difficult.

Different systems often use different datasets, answer models, prompts, and evaluation methods. When the final score changes, it can be hard to tell whether the difference comes from the memory system itself or from the evaluation setup.

That's why we built Agent Memory Leaderboard (AML).

Why Do We Need a Memory Benchmark?

Memory is more than storing conversation history.

A useful memory system needs to retrieve relevant information, connect information across time, handle changing states, and provide useful context for an agent's current task.

But there hasn't been a common evaluation environment where different memory approaches can be compared under the same conditions.

AML was created to provide that common ground.

The first edition was jointly initiated by nearly 30 universities and research institutions and covers two evaluation tracks:

Open-source Methods — Text Memory
Commercial Products — Text Memory

As of August 12, 2026:

136 teams registered for the competition
67 representative memory frameworks successfully completed the first evaluation
The AML website surpassed 200,000 clicks
The website passed 100,000 clicks within the first 10 days

The first leaderboard results are now live.

Making Memory Systems More Comparable

One of the main challenges we wanted to address was evaluation consistency.

In a typical setup, a memory system may be evaluated together with a particular answer model, prompt, or judge.

That makes direct comparison difficult.

A higher score could come from a better memory system — but it could also come from a stronger downstream model or a different evaluation setup.

AML tries to separate these components.

The core interface for participating memory systems is:

Memory System
Add → Search

The memory system receives long-term history through Add, and returns relevant memories through Search.

Then AML handles:

AML Evaluation Platform
Answer → Eval

Answer generation and evaluation are completed by the benchmark platform under the same evaluation process.

This helps reduce the impact of different answer models, prompts, judges, and scoring conventions.

The goal is simple:

Compare memory systems under the same conditions as much as possible.

Memory Is More Than Retrieval

A memory system shouldn't be judged only by whether it can retrieve something that looks similar. Memory quality is not only about retrieving similar information, but about understanding relevance, context, time, and task requirements.

For text memory, AML evaluates multiple capabilities, including:

Factual recall
Relational and multi-hop reasoning
Temporal and event understanding
Memory governance
Personalization and care
Rules and workflow execution
Epistemic safety and privacy

This matters because real-world agent memory is not just a search problem.

An agent may need to understand:

What happened before?
When did it happen?
What changed afterward?
Which previous experience is relevant now?
Which information should be trusted?
How should that memory affect the current task?

A useful memory system needs to handle these questions together.

The First Results

The first AML evaluation has now been completed.

67 representative memory frameworks successfully completed the evaluation across two tracks covering text memory for both open-source methods and commercial products.

The complete rankings, scores, and system versions are available on the leaderboard.

Leaderboard:
[https://agentmemoryleaderboard.ai/leaderboard/industry/textual ]

What Comes Next?

The first leaderboard is not the finish line.

We plan to keep AML running as a long-term evaluation and public leaderboard for agent memory systems.

Going forward, we will publish deeper technical analyses of the first-round results, including:

Technical architectures that performed well
Interesting evaluation patterns
Performance across individual memory capabilities
Differences between text and code memory
What current results tell us about the evolution of agent memory

We also want the benchmark itself to evolve.

If you are working on AI agents, memory systems, evaluation, or related research, we would love to hear what you think a useful memory benchmark should measure next.

Four Evaluation Tracks

To better reflect different types of memory systems, AML organizes evaluation into two tracks:
Open-source Methods

Text Memory

Commercial Products

Text Memory

Each track evaluates systems under the same benchmark framework.

Explore AML

Leaderboard:
[https://agentmemoryleaderboard.ai/leaderboard/industry/textual]

GitHub:
[https://github.com/AML-memory/agent-memory-leaderboard]

Hugging Face Space:
[https://huggingface.co/agent-memory-leaderboard]

X:
[https://x.com/AgentMemoryL/status/2087544165433590240]

Contact:
contactus@agentmemoryleaderboard.ai

The first results are out.

Now the real work begins:

Make memory measurable. Make progress verifiable.

Why AI Agents Need Memory Benchmarks？

Agent Memory Leaderboard — Mon, 10 Aug 2026 08:18:18 +0000

AI agents are getting better at reasoning, coding, and tool use.

But one question remains open:

How do we know if an agent actually remembers and learns from previous interactions?

Most current evaluations focus on immediate task performance. However, real-world agents often need more than that.

They need to remember:

previous decisions and why they were made
failed approaches and lessons learned
project-specific patterns and context

Memory is becoming a key capability for long-running AI agents, but evaluating it fairly is still challenging.

Different systems often use different datasets, models, and evaluation methods, making direct comparisons difficult.

We believe open and reproducible evaluation is an important step toward building better AI agents.

This week, we will share the first results from an open evaluation effort for AI Agent Memory systems.

More updates coming soon.