Why Agent Memory Benchmarks Need More Than Retrieval Accuracy

Most AI agents can retrieve information from a previous conversation or a project history.

That is useful, but it is not the same as having reliable memory.

A memory system may retrieve a fact that is technically relevant but no longer current. It may return two conflicting pieces of evidence without knowing which one should be trusted. It may find the right note, while the answer model fails to use it correctly.

This raises a more important question for AI memory evaluation:

Did the retrieved memory actually help the agent complete the task?

Retrieval is only one part of the memory problem

Many Agent Memory systems are evaluated by asking whether they can retrieve the correct historical information.

This is a reasonable starting point. If a system cannot find relevant information, the agent cannot use it.

But retrieval accuracy alone does not describe the full behavior of a memory-enabled agent.

Consider a coding agent working on a long-running repository. A previous note says that the application uses PostgreSQL. Several weeks later, the project has migrated to MySQL. If the agent retrieves the old note because it is semantically relevant, the retrieval step may be counted as successful. However, using that memory in the current task could lead to an incorrect implementation.

The same issue appears in other settings:

• A user preference has changed;
• A team rule has been updated;
• A previous solution was later replaced;
• Two historical notes contain conflicting information;
• A large amount of irrelevant history distracts the agent from the current task.

In these cases, “did the system find the note?” is not enough. We also need to ask whether the note is still valid, whether it applies to the current task, and whether the agent used it appropriately.

Memory quality depends on time and state

Long-term memory is not just a larger context window.

A useful memory system needs to track how information changes over time. Some memories remain stable, while others are temporary. Some facts should be updated. Others should be deleted or marked as outdated.

This is especially important for software projects and organizational knowledge.

A project may change its database, architecture, deployment process, or ownership. A team may replace an engineering convention. A user may change a preference that was previously recorded as a long-term fact.

If the memory system treats every historical record as equally valid, additional context can make the agent less reliable rather than more reliable.

This is why a useful Agent Memory benchmark should include temporal changes, conflicting evidence, and noisy history.

The answer model matters too

Another difficulty is that memory evaluation is not limited to the memory layer.

A typical memory-enabled agent pipeline contains several stages:

1. The system receives and stores new information.
2. It searches for relevant memories when a task arrives.
3. The answer model receives the retrieved evidence.
4. The agent generates a response or takes an action.
5. An evaluator judges the final result.

A system can fail at any stage.

It may store the wrong information, retrieve the wrong evidence, return too much context, or retrieve the correct evidence but fail to use it in the final answer.

This means that the final task score reflects the entire pipeline, not just the retrieval component.

A benchmark that only measures retrieval can miss failures that users experience in practice.

Why results are difficult to compare

Different Agent Memory systems are often evaluated with different datasets, models, prompts, retrieval settings, and judges.

As a result, the same system may receive different scores across different reports.

This does not necessarily mean that one report is wrong. It may mean that the benchmarks are measuring different capabilities under different conditions.

For example, one evaluation may focus on direct fact recall, while another tests multi-hop reasoning over long conversations. One may use a clean dataset, while another introduces irrelevant or conflicting history. One may evaluate the retrieved evidence directly, while another measures whether the final answer is correct.

Without a shared protocol, it becomes difficult to tell whether a score reflects the memory system itself or the surrounding evaluation setup.

What should a stronger Agent Memory benchmark measure?

A more complete benchmark should evaluate several connected capabilities.

First, it should test whether important information is written correctly. A system cannot retrieve information that it failed to store.

Second, it should measure retrieval relevance. The system should return evidence that is useful for the current task, rather than simply returning text that looks similar.

Third, it should evaluate temporal validity. The benchmark should include facts that change over time and test whether the agent can prioritize the current state.

Fourth, it should include conflicting and noisy history. Long-running agents will rarely operate on perfectly clean memory.

Finally, it should measure downstream task success. The key question is whether memory improves the agent’s ability to answer, code, reason, or act.

This last step is particularly important. A memory system should not receive full credit merely because it returned a relevant note. The retrieved note should help the agent do something better.

A shared evaluation protocol

Agent Memory Leaderboard (AML) is one public attempt to create a more standardized evaluation environment for this problem.

In AML’s public evaluation design, participating systems provide standardized Add and Search interfaces. The platform then handles downstream Answer and Eval under a shared protocol, followed by result review and public reporting.

This separation creates a clearer boundary between the memory system and the rest of the evaluation pipeline.

The participating system is responsible for receiving and retrieving memory. The platform is responsible for applying a consistent downstream process to assess whether that memory supports the task.

AML’s second public cycle covers three evaluation tracks:

• Textual Memory;
• Coding Memory;
• Multimodal Memory.

These tracks represent different requirements. Textual Memory involves long conversations, cross-session history, temporal events, personalization, and rule following. Coding Memory focuses on repository context, engineering decisions, debugging history, and previous failures. Multimodal Memory examines how systems write, retrieve, and use evidence across text and images.

The separation between tracks matters because a system that performs well on conversational memory may not perform equally well on long-running coding tasks or multimodal evidence.

From “Can it retrieve?” to “Can it use memory correctly?”

The next stage of Agent Memory evaluation should move beyond the question of whether a system can find historical information.

A stronger benchmark should ask:

• Is the memory still valid?
• Does it apply to the current task?
• Can the system resolve conflicts between old and new evidence?
• Can it ignore irrelevant history?
• Does the agent use the memory correctly?
• Does memory improve the final task outcome?

This is a more demanding standard, but it is also closer to how memory is used in real systems.

As AI agents move from one-shot interactions to long-running work, the value of memory will depend less on how much information is stored and more on whether the agent can use the right experience at the right time.

The future of Agent Memory benchmarks may therefore be less about ranking retrieval systems in isolation and more about measuring reliable memory use across complete agent tasks.

AML Cycle 2 materials submission has closed, and the official evaluation is scheduled to conclude on November 4, 2026 at 23:59 UTC+8. The official results are expected in mid-November.