An agent retrieves a project decision from last month. The record is relevant, detailed, and correctly quoted.
There is one problem: the decision was reversed last week.
Retrieval succeeded. The agent may still make the wrong recommendation.
This is why evaluating agent memory requires more than asking whether a system can find a related record. We also need to examine whether the retrieved evidence supports the current task, reflects changes, and helps the agent produce a useful answer or action.
The second cycle of Agent Memory Leaderboard (AML) has now published its results across text, code, and multimodal memory. This article looks at how to read those results—and what a ranking alone cannot tell us.
Disclosure: This article is published by the AML team and discusses our evaluation and results.
Three tracks, three different questions
“Agent memory” covers several kinds of work.
For a conversational assistant, the challenge may be recalling a preference while recognizing that it has changed. For a coding agent, it may be finding an earlier debugging attempt and deciding whether its conclusion still applies. For a multimodal agent, it may involve connecting a question to evidence distributed across text and images.
AML Cycle 2 evaluates these settings through three tracks:
- Text memory: retrieving and using information from interaction histories, including facts, relationships, updates, and rules.
- Code memory: using historical engineering experience to support subsequent coding tasks.
- Multimodal memory: retrieving and using evidence across modalities.
The tracks use different tasks, metrics, and scoring rules. Their scores should therefore be interpreted within each track.
A score of 63 in code memory does not establish that a system has “better memory” than one scoring 53 in multimodal memory.
What the published results show
Cycle 2 received 402 Full evaluation task submissions. Of these, 333 runs completed successfully. Deduplication produced 309 records, and review excluded 17 non-compliant records, leaving 292 valid leaderboard entries.
These figures describe different stages of evaluation. They are not interchangeable with a count of participating teams or unique systems. A system evaluated in multiple tracks contributes separate track entries.
Participating projects were submitted by teams from China and abroad, including teams from Alibaba Cloud DMS, Shopee, OPPO, KT, and OceanBase, alongside university researchers, memory startups, open-source contributors, and independent developers.
That range of participants makes a shared evaluation useful. It brings research implementations and deployed products into a common framework, while retaining separate leaderboard categories.
Each track publishes open-source and closed-source rankings. A commercial products leaderboard identifies a subset of the closed-source entries.
A returning leader—and several results worth examining
On the Text Memory Commercial Products leaderboard, the top three are:
| Rank | System | Overall score |
|---|---|---|
| 1 | MemoraX | 69.26 |
| 2 | AIDBS-Groundata | 64.32 |
| 3 | Gedo Memory | 62.28 |
MemoraX retained first place in this category for the second consecutive cycle. It also ranked first on this cycle’s Code and Multimodal Commercial Products leaderboards.
That is a result about performance under the published evaluation conditions. It does not establish that the system leads on every capability or is the best fit for every deployment.
The open-source leaders were:
| Track | System | Overall score |
|---|---|---|
| Text memory | Resurface | 61.22 |
| Code memory | InsightMem | 63.33 |
| Multimodal memory | Khipu | 43.26 |
The new Multimodal Open-Source leaderboard also includes starwingMem and Multimodal-Raw-Rag-AML in its top three.
These rankings provide starting points for investigation. The next step is to examine the capability breakdowns and the evidence behind particular successes and failures.
Separate the memory system from the downstream model
A recurring difficulty in memory evaluation is attribution.
If one system uses a stronger answer model, a higher final score may reflect that model’s ability to compensate for incomplete evidence. Conversely, useful retrieved evidence may lead to a poor answer because the downstream model misinterprets it.
AML separates these responsibilities:
- Participants provide Add and Search interfaces for writing and retrieving memory.
- The platform handles downstream answering or agent execution and evaluation under the track’s shared protocol.
- Results undergo completeness, version, and compliance review before publication.
This reduces variation from participant-selected downstream components. It also makes the evaluation easier to inspect: what did the memory system return, and what happened when the downstream system used it?
It does not remove every source of uncertainty. Results remain tied to the evaluated tasks, system versions, and execution conditions.
Read a leaderboard as a shortlist for investigation
For a developer choosing a memory system, the overall ranking is useful—but insufficient.
Start with the track closest to your workload. Then examine the capabilities and failure cases that matter to your application.
For an assistant handling changing customer requirements, useful questions include:
- Can it retrieve the current requirement and distinguish it from an earlier version?
- Can it explain the historical state when asked?
- Can it avoid applying a temporary instruction outside its intended scope?
For a coding agent:
- Does the retrieved experience concern the current repository state?
- Is a previous workaround still valid?
- Can the agent distinguish a useful failed attempt from a solution it should repeat?
For a multimodal workflow:
- Does retrieval find the evidence needed to answer the question?
- Can the downstream system connect evidence across modalities?
- Does the answer remain grounded when the evidence is incomplete?
These questions help turn a leaderboard position into a practical evaluation plan.
The next step is explaining the failures
A final answer can reveal that something went wrong. Fixing it requires locating the failure.
Was the necessary information stored? Was it retrieved? Did a later update appear alongside an obsolete record? Did the downstream agent receive the right evidence but follow the wrong instruction?
Those possibilities call for different interventions. Increasing retrieval depth, for example, may help when evidence is missing. It may also add more conflicting history for the answer model to resolve.
AML will publish further capability-level analyses and case studies to examine these distinctions.
The full rankings, system versions, and capability breakdowns are available at:
https://agentmemoryleaderboard.ai
Updates: https://x.com/AgentMemoryL
When you evaluate agent memory, which failure is hardest to diagnose: missing evidence, outdated evidence, or correct evidence used incorrectly?


Top comments (0)