Beyond Retrieval: What Should an AI Agent Remember?

As AI agents move from short interactions to long-running tasks, memory becomes more than a retrieval problem.

An agent working on a software project for several days may need to remember previous design decisions, debugging attempts, failed approaches, and changes in requirements. A conversational agent may need to distinguish between a user’s current preference and something they said weeks ago.

The challenge is not simply whether an agent can retrieve historical information. It is whether the agent can identify which information is still relevant, which has become outdated, and which conflicts with newer evidence.

More memory does not always mean better memory

A memory system can retrieve a large amount of historical context and still make an agent perform worse.

For example, an AI coding agent may retrieve:

• an old implementation decision;
• a previously fixed bug;
• an outdated configuration;
• a failed debugging approach;
• information from a similar but unrelated module.

All of these may look relevant at first glance. But if the agent cannot distinguish current information from stale information, retrieval can introduce noise instead of providing useful context.

This creates several important questions:

• When should a memory be updated?
• How should conflicting evidence be handled?
• Can the system recognize that an old memory is no longer valid?
• Does retrieved memory actually improve task performance?
• How much irrelevant history can an agent tolerate?

These questions are difficult to answer with a retrieval-accuracy score alone.

Evaluating memory under realistic conditions

A useful memory benchmark should test more than whether a system can find a matching sentence.

It should also examine how the system behaves when:

• historical information is incomplete;
• several pieces of evidence conflict;
• irrelevant memories appear alongside useful ones;
• the same project evolves over time;
• a task requires experience from multiple earlier events;
• the agent must continue work after a long interruption.

This is especially important for coding agents. In a real repository, the most recent piece of information is not always the correct one, and the most similar piece of text is not always the most useful evidence.

A strong system should help the agent understand not only what happened, but also why it happened and whether it still applies.

A shared evaluation process

_Agent Memory Leaderboard (AML) _provides an evaluation framework for comparing memory systems across a common process.

Participants bring their own memory infrastructure and expose two standard interfaces:

• Add: receive and store information;
• Search: return memories or evidence relevant to a query.

AML then standardizes the downstream stages, including answer generation, evaluation, review, and public reporting.

This separation is useful because it keeps the focus on the memory system itself. Participants can use different storage methods, retrieval strategies, ranking models, or memory architectures, while the evaluation process remains comparable.

The goal is not to prescribe one implementation. It is to make different approaches easier to study under shared conditions.

Three areas of Agent Memory

The second AML evaluation cycle covers three types of memory.

Textual Memory

This track focuses on long conversations and cross-session history. It includes factual recall, relationships, temporal events, personalization, rule following, and streaming memory.

The key challenge is maintaining a useful representation of information as conversations continue to change.

Coding Memory

Coding Memory focuses on software engineering tasks.

A coding agent may need to use:

• project structure and implementation context;
• previous debugging sessions;
• historical design decisions;
• failed solutions;
• test results;
• relationships between modules.

The important question is not only whether the agent can retrieve repository history, but whether that history helps it complete a new task more effectively.

Multimodal Memory

Many real-world agents work with both text and images.

The Multimodal Memory track examines how systems store, retrieve, and use information across these modalities. This includes connecting textual explanations with visual evidence and using both types of information in later tasks.

What should a useful benchmark measure?

Retrieval quality remains important, but it should be treated as one part of a broader evaluation.

A more complete benchmark may consider:

1. Relevance — Is the retrieved memory related to the current task?
2. Accuracy — Does it represent the original information correctly?
3. Freshness — Is the information still valid?
4. Conflict handling — Can the system distinguish between competing memories?
5. Robustness to noise — Does irrelevant history significantly reduce performance?
6. Task impact — Does memory actually improve the final result?
7. Efficiency — How much context and computation are required?

This shifts the focus from “How much can the system retrieve?” to “How effectively can the agent use past experience?”

An open question for the field

Agent Memory is still an emerging research area. There is no single accepted architecture for deciding what an agent should remember, how memories should be updated, or when they should be removed.

That makes shared evaluation particularly valuable.

A public benchmark can help researchers compare different assumptions, expose failure cases, and identify which memory capabilities matter most in real tasks.

The long-term goal is not to build agents that remember everything. It is to build agents that can use the right experience at the right time.

AML Cycle 2 is currently open to researchers, developers, open-source teams, commercial AI teams, and individual contributors worldwide.

The evaluation includes Textual Memory, Coding Memory, and Multimodal Memory. Complete submissions are due by October 31, 2026, at 23:59 UTC+8.

Learn more and apply at Agent Memory Leaderboard.