AI agents are becoming better at working across multiple steps, sessions, and tasks. But as agents take on longer-running work, another problem becomes increasingly important:
What happens when the memory they retrieve is no longer correct?
A memory system may successfully retrieve information from the past. However, that does not necessarily mean the information is useful for the current task.
An old design decision may have been replaced. A debugging approach may have already failed. A user preference may no longer apply. Two historical records may provide conflicting evidence.
In these cases, retrieving more context can make an agent less reliable rather than more capable.
Retrieval is only the first step
Many memory systems are evaluated by whether they can retrieve relevant information. This is an important capability, but it does not fully describe how memory affects an agent.
A retrieved memory also needs to be:
- relevant to the current task;
- consistent with newer evidence;
- valid in the current project state;
- understandable to the downstream agent;
- useful for completing the task.
Consider a coding agent working on a long-running repository. It may retrieve a previous implementation decision that was correct several months ago. Since then, the architecture may have changed. If the agent treats the old decision as current, the memory system has introduced a failure mode.
The problem is not that retrieval failed. The problem is that the system did not understand the status of the retrieved information.
Three difficult types of memory
Stale memory
A stale memory was once correct but is no longer valid.
Examples include:
- an old API specification;
- a deprecated configuration;
- a previous product requirement;
- a bug that has already been fixed;
- a design decision replaced by a newer approach.
A useful memory system should be able to recognize that time and state matter.
Conflicting memory
Different historical records may disagree.
For example, one project document may say that a service uses one database, while a later engineering decision moves it to another. Both records may be retrieved for the same query.
The agent then needs to determine which evidence is newer, more authoritative, or more relevant to the current task.
Noisy history
Long-running projects generate a large amount of information that is not relevant to every task.
A repository may contain unrelated commits, abandoned experiments, failed approaches, and duplicated discussions. If the memory system returns too much of this history, the agent may spend more effort interpreting context than solving the problem.
This is particularly important for Coding Memory. The goal is not to retrieve the entire history of a repository. The goal is to retrieve the right engineering experience for the current task.
What should a benchmark measure?
A more complete Agent Memory evaluation should test not only retrieval, but also downstream task performance.
Useful questions include:
- Can the system retrieve information related to the task?
- Can it distinguish current information from outdated information?
- Can it handle conflicting evidence?
- Can it remain useful when irrelevant history is added?
- Does the retrieved memory improve the final answer or implementation?
- Does it reduce repeated explanations and failed attempts?
These questions require controlled evaluation conditions. Otherwise, it becomes difficult to determine whether an improvement comes from the memory system, the language model, the prompt, or the evaluation process itself.
A shared evaluation framework
Agent Memory Leaderboard (AML) provides a common evaluation process for different memory systems.
Participants provide their own memory infrastructure through standardized Add and Search interfaces. The AML platform then handles downstream Answer, Evaluation, review, and public reporting.
This setup allows different architectures and retrieval methods to be compared within a shared evaluation boundary.
The second AML cycle covers three areas:
- Textual Memory for long conversations and cross-session history;
- Coding Memory for software engineering tasks and repository experience;
- Multimodal Memory for systems working with text and image-based evidence.
The purpose is not to define one correct memory architecture. It is to make the strengths and weaknesses of different approaches easier to observe and discuss.
From remembering more to remembering better
A reliable agent does not need to remember everything.
It needs to:
- retain information that remains useful;
- identify when an old memory has changed;
- resolve conflicts between historical records;
- ignore irrelevant context;
- apply past experience to new tasks.
This is why Agent Memory evaluation should move beyond simple retrieval accuracy.
The more important question is whether memory helps an agent make better decisions under changing conditions.
AML Cycle 2 is currently open to researchers, developers, open-source teams, commercial AI teams, and individual contributors. The evaluation covers Textual, Coding, and Multimodal Memory.
Complete submissions are due by October 31, 2026, at 23:59 UTC+8.
Learn more and participate:
Top comments (3)
Stale context costs twice. The agent resends it on every turn, and a wrong fact in it can still win when two notes conflict. A memory that carries a date and a source lets the agent drop the older note instead of guessing.
Exactly—stale context can be more dangerous than missing context. In practice, what signals do you rely on most to resolve conflicts between memories: timestamps, source reliability, explicit user updates, or something else?
Thanks for this post. I found four issues while running Cycle 2.
Abstention lowered my Textual score, so I disabled it in my submission. As it is currently scored, the benchmark does not test whether a memory system can recognise that the evidence it holds does not support an answer. In my tests a retrieval score could not separate unanswerable questions from answerable ones, so that decision needs a reader that checks the evidence, which the memory layer does not control here.
Many times during my runs, the memory layer retrieved the correct information and the problem was in the reader.
Controlling only Add and Search, without the ability for the memory layer to inject context at specific moments, unfortunately cannot fully test the system. In many cases the agent will not search or look for memory by itself.
On stale memory: my memory system resolves it before the reader. When a new memory is written, it declares which earlier fact it replaces, the old one is marked as superseded, and Search answers from the current one. On FactConsolidation this answered 97 of 100 questions, against 51 when only dates are passed to the reader, because readers do not reliably apply “latest date wins”.
In AML, Add receives raw conversation, so those declarations have to be extracted by an LLM at write time, which in my experiments is the only good place to do it. The mandatory gpt-4o-mini is not good enough for this role, while DeepSeek, Luna and Opus can do it.