DEV Community: Agent Memory Leaderboard

From Storing Context to Building Experience: What 50+ Teams Tell Us About Agent Memory

Agent Memory Leaderboard — Wed, 23 Sep 2026 09:27:49 +0000

More than 50 teams have registered for the second Agent Memory Challenge shortly after the cycle opened.

They come from different backgrounds: universities, open-source projects, independent developers, and commercial teams. They are building different kinds of memory systems, using different engineering approaches, and targeting different agent workflows.

This number does not tell us which system is the best.

It does tell us something important: Agent Memory is becoming a serious research and engineering problem.

Memory is moving beyond chat history

Early discussions around AI memory often focused on a simple question:

Can an agent remember what happened earlier?

That question is still useful, but it is no longer sufficient for long-running agents.

A production agent may need to work across days or weeks. It may interact with multiple users, repositories, documents, tools, and task states. During that time, facts can change, decisions can be revised, and previous solutions can become invalid.

A useful memory system therefore needs to do more than store additional context. It needs to help an agent:

retrieve evidence from previous interactions;
connect related facts across multiple sessions;
understand when information is relevant to the current task;
recognize when an earlier decision has been superseded;
avoid relying on stale or conflicting context;
reuse previous experience without blindly copying it.

In other words, the goal is not simply to make an agent remember more.

The goal is to help an agent use the right experience at the right time.

Why interest in Agent Memory is growing

The rapid growth of agentic systems has changed the role of memory.

For a one-shot question-answering system, the context window may be enough. But for an agent that maintains a project, debugs software, manages a workflow, or collaborates with a user over a long period, the context window is only one part of the problem.

The agent must also decide:

what should be remembered;
what should be ignored;
what should be updated;
what should be retrieved later;
which evidence is still valid;
and how much historical context should be passed into the next task.

This creates a broad design space.

Some systems focus on structured facts. Others build episodic or procedural memory. Some use graphs, timelines, summaries, or topic documents. Others attempt to learn memory policies or evolve their own internal representations.

The diversity of approaches is valuable, but it also creates a comparability problem.

The evaluation problem

Different memory systems are often evaluated with different datasets, answer models, prompts, retrieval methods, and scoring rules.

As a result, two systems may report impressive results while answering fundamentally different questions.

One system may be optimized for factual recall. Another may focus on multi-hop reasoning. A third may target coding history or multimodal evidence. Without a shared evaluation boundary, it is difficult to understand what the scores actually mean.

This is the problem that Agent Memory Leaderboard is trying to address.

AML separates the memory layer from the downstream answer and evaluation process:

Participants provide an Add interface for writing and updating memory.
Participants provide a Search interface for retrieving relevant memory evidence.
AML standardizes the downstream Answer and Eval pipeline.
Results are reviewed and published by track, division, submitted version, and evaluation cycle.

This does not create one universal definition of memory.

Instead, it creates a common environment where different approaches can be compared more transparently.

What 50+ registrations may indicate

The early response to Cycle 2 should not be interpreted as proof that the field has already converged on a single architecture.

In fact, the opposite may be true.

The number of participating teams suggests that the field is still exploring several important questions:

1. Memory is becoming a system-level capability

Many teams are no longer treating memory as a small feature added to an agent framework. They are designing dedicated systems for storing, organizing, updating, and retrieving experience.

This reflects a shift from prompt-level context management to memory infrastructure.

2. Different agents need different forms of memory

A conversational agent may need user preferences, personal facts, and temporal events.

A coding agent may need repository history, debugging traces, failed approaches, design decisions, and testing evidence.

A multimodal agent may need to connect text with images, captions, visual events, and ordered context.

These requirements cannot always be solved by the same retrieval strategy.

3. Freshness is as important as recall

A memory system can retrieve a fact correctly and still produce the wrong result if that fact is outdated.

For example:

a project decision may have been changed;
a user preference may no longer apply;
a dependency may have been upgraded;
a previous debugging hypothesis may have been disproven.

This means future memory evaluations need to test not only whether a system can retrieve historical information, but also whether it can identify current and valid evidence.

4. Agent Memory needs reproducible benchmarks

As more teams build memory systems, informal demos become less sufficient.

A demo can show that a system works in one scenario. A benchmark can help reveal how it behaves across different tasks, histories, updates, noise conditions, and evidence requirements.

The field needs both: practical demonstrations and controlled evaluation.

What Cycle 2 evaluates

The second Agent Memory Challenge includes three tracks.

Textual Memory

The Textual Memory track evaluates long conversations and cross-session history, including:

explicit fact retrieval;
multi-hop relationships;
temporal events;
personalization;
rule and workflow following;
memory governance;
safety and privacy boundaries.

Cycle 2 also includes streaming memory settings, where Add and Search operations occur as events unfold. This tests whether a system can use information available at the correct point in time, rather than relying only on a static archive.

Coding Memory

The Coding Memory track focuses on long-running software engineering tasks.

It evaluates whether an agent can use historical development experience, including:

repository context;
implementation decisions;
debugging trails;
failed approaches;
module relationships;
testing evidence;
previously validated fixes.

The formal coding evaluation includes 150 software engineering tasks under both relevant-history and noisy-history conditions.

Multimodal Memory

The Multimodal Memory track evaluates how systems write, retrieve, and use memory involving text and images.

Participants may process original images, captions, or both, but must correctly handle ordered multimodal content and disclose which media types their systems use.

What happens next

The early participation in Cycle 2 is encouraging, but the more important work is still ahead.

The field now needs to move toward clearer definitions and stronger evaluation questions:

How should memory systems represent changing facts?
How should they handle conflicting evidence?
How can provenance be preserved?
When should a memory be updated, decayed, or deleted?
How should we measure stale retrieval?
How can we evaluate memory under noisy or adversarial histories?
What should a coding agent remember from a previous task?
How should multimodal evidence be linked to later decisions?

These are not only benchmark questions. They are design questions for the next generation of agents.

Join the benchmark

Agent Memory Challenge Cycle 2 is now open to:

universities and research institutions;
open-source maintainers;
independent research teams;
commercial product teams;
individual developers;
cross-organization teams.

Participation is free. The challenge includes separate Open-source Methods and Commercial Products divisions across the three tracks.

The goal is not to declare one universal winner.

The goal is to make long-term Agent Memory more measurable, more comparable, and easier to improve.

More than 50 teams have already registered. We hope to see more researchers and developers contribute systems, feedback, benchmarks, and ideas to this growing area.

Agent Memory Challenge 2026 Cycle 2

Website:

https://agentmemoryleaderboard.ai/

Evaluation:

https://agentmemoryleaderboard.ai/evaluation

Rules:

https://agentmemoryleaderboard.ai/rules

GitHub:

https://github.com/AML-memory/agent-memory-leaderboard

Twitter Official Link:
https://x.com/AgentMemoryL

Evaluating Long-Term Memory for AI Agents: Agent Memory Challenge Cycle 2 Is Now Open

Agent Memory Leaderboard — Mon, 21 Sep 2026 03:16:27 +0000

Long-term Agent Memory is not only about retaining more history. It is about retrieving evidence that remains useful when facts, decisions, and tasks change.

Agent Memory Challenge 2026 — Cycle 2 is now open.

Textual Memory · Coding Memory · Multimodal Memory

Open-source Methods · Commercial Products

Join the evaluation

Everyone agrees that AI agents need memory.

But “memory” can mean very different things in practice.

One system may preserve raw conversation history. Another may summarize it into structured facts. A third may retrieve repository traces, prior debugging attempts, or images from earlier sessions. Some systems use dense retrieval; others use graphs, databases, rerankers, or generative memory consolidation.

The difficult question is not whether an Agent can store information.

The difficult question is whether it can retrieve the information that should govern its next action.

A memory system can retrieve a highly similar statement from the past and still mislead an agent if that statement has since been corrected. It can return every related record and still fail if the agent cannot distinguish a current decision from an obsolete one.

That is the problem Agent Memory Challenge 2026 Cycle 2 is designed to examine.

What is the Agent Memory Challenge?

The Agent Memory Challenge is the public evaluation program of the Agent Memory Leaderboard (AML), an open benchmark for long-term Agent Memory.

Participants provide the memory layer through two interfaces:

Add — receives content that the system should store, organize, index, update, or consolidate.
Search — returns relevant memory evidence for a later query and user scope.

AML then runs the shared downstream workflow:

Answer — generates task responses from retrieved evidence.
Eval — scores results under a standardized protocol.
Review — verifies submitted versions, evaluation conditions, and compliance before publication.

This separation matters.

In many public comparisons, different systems use different answer models, prompts, datasets, scoring methods, and evaluation procedures. It can become difficult to tell whether a result reflects a better memory system, a stronger answer model, or simply a different test setup.

AML creates a clearer evaluation boundary:

Participants build the memory layer.

AML standardizes the downstream Answer and Eval process.

The goal is not to claim that one architecture is universally the “best” memory system. It is to make different approaches more comparable under a shared, reproducible evaluation flow.

Why “current evidence” matters

Long-running agents do not operate in a static world.

A release date changes. A preference is corrected. A debugging hypothesis is disproven. A repository evolves. A previous decision becomes invalid after new evidence appears.

In these situations, retrieving the most similar old memory is not enough.

A useful memory system should help an agent answer questions such as:

What was previously believed?
What changed later?
Which source supports the current state?
Which older information remains historically relevant?
Which information should no longer influence the next task?
Can the returned evidence be traced back to its source?

This is why Agent Memory should not be evaluated only as a search problem.

It is also a problem of temporal reasoning, provenance, relevance, scope, and decision support.

Three tracks in Cycle 2

Cycle 2 expands the evaluation across three independent tracks.

Textual Memory

The Textual Memory track evaluates long conversations and cross-session history.

It includes capabilities such as:

Explicit fact recall
Multi-hop relations and reasoning
Temporal events and changing states
User personalization
Rule and workflow following
Memory governance and safety boundaries

Cycle 2 also includes Streaming Memory conditions, where Add and Search calls occur as events unfold. This tests whether a system can use the right information at the right time, rather than relying only on a static archive of the full conversation.

Coding Memory

Coding agents accumulate a different kind of history.

Past implementation decisions, repository conventions, failed debugging attempts, test results, error messages, module relationships, and development trajectories can all matter to a later task.

The Coding Memory track evaluates whether historical engineering experience can help an agent solve subsequent software-engineering tasks.

The formal evaluation contains 150 software-engineering tasks, tested under both relevant-history and noisy-history settings—for 300 task-condition units in total.

The question is not simply whether a system can find a file path or a past error message.

It is whether it can retrieve useful engineering evidence without overwhelming the agent with irrelevant historical noise.

Multimodal Memory

Agents increasingly work across text, images, screenshots, documents, and other visual context.

The Multimodal Memory track evaluates how systems write, retrieve, and use text-and-image memory. It tests whether a system can preserve ordered multimodal context, connect evidence across modalities, and retrieve the information needed for a later task.

Participants may use original images, captions, or both, while disclosing the media types their systems actually process.

Core principles of the evaluation

The benchmark is built around a few practical rules.

Search returns evidence, not a disguised final answer

The participant’s Search API should return memory evidence relevant to the query. AML handles downstream answer generation and scoring.

This keeps the memory layer distinct from the final response model.

User isolation is a strict boundary

Memory from one user_id, task, sample, team, or evaluation run must not leak into another.

Long-term memory is useful only if it is also appropriately scoped.

A successful write must be searchable

A synchronous Add request should return success only after the content has been persisted and can be retrieved through Search.

Retries should not create duplicate memories

Systems should handle request_id idempotently, so a network retry does not write the same memory multiple times.

Formal evaluations use a fixed system version

Once a Full evaluation begins, the submitted code, image, endpoint, authentication method, and operating conditions must remain stable.

Results are published in the context of the corresponding track, division, system version, evaluation suite, and cycle.

Cycle 1: a shared evaluation is already attracting attention

AML published its first leaderboard cycle on August 12, 2026.

During the first cycle:

136 teams submitted applications
67 representative memory systems completed official evaluation
The AML website received more than 300,000 cumulative visits
AML’s Hugging Face Space reached the top of the Spaces trending list during the first cycle

These figures reflect participation and community interest in Cycle 1. They do not imply that any single system is universally superior outside its submitted version, track, and evaluation conditions.

What they do show is that the Agent Memory community is actively looking for more transparent ways to compare systems.

Who can participate?

Cycle 2 is open globally to:

Universities and research institutions
Open-source maintainers
Independent research teams
Commercial product teams
Individual developers
Cross-organization teams

Each track includes two separate divisions:

Open-source Methods
Commercial Products

Participants may enter multiple tracks, while rankings remain separate by track and division.

Participation is free. Participants operate their own APIs, databases, bandwidth, and compute resources; AML provides the standardized evaluation orchestration, downstream Answer and Eval process, result review, and leaderboard publication.

Key dates

September 20, 2026, 00:00 UTC+8 — Cycle 2 opens
September 20–October 31 — Registration, integration, Smoke tests, Full evaluations, and review
October 31, 2026, 23:59 UTC+8 — Submission-material deadline
November 4, 2026, 23:59 UTC+8 — Evaluation closes
Mid-November 2026 — Official results planned for release

Teams should submit stable APIs early. A Full evaluation typically takes approximately 0.5–2 days, depending on the implementation and evaluation scale.

Open-source awards

Cycle 2 provides a total prize pool of approximately USD 22,400 for eligible Open-source Methods teams across the three tracks.

Each track includes:

First Prize — approximately USD 2,980
Two Second Prizes — approximately USD 1,190 each
Three Third Prizes — approximately USD 450 each
Best Technical Innovation Award — approximately USD 750

Awards are denominated and paid in RMB. USD figures are approximate and may vary with the exchange rate.

Commercial Products are ranked in a separate division and are not eligible for the prize pool.

Explore AML and participate

Long-term Agent Memory should be more than a growing context window.

It should help an agent retrieve the evidence that is relevant now, understand what has changed, and act on a record that can be inspected and verified.

Make memory measurable. Make progress verifiable.

From “Storing” to “Staying Current”: Why Agent Memory Needs a Shared Evaluation

Agent Memory Leaderboard — Sat, 19 Sep 2026 14:06:32 +0000

Agent Memory Challenge 2026 Cycle 2 opens September 20 across Textual, Coding, and Multimodal Memory.

Long-term Agent Memory is not about keeping more history.

It is about retrieving the right evidence from long-running experience, recognizing when a fact or decision has changed, avoiding stale context, and using that evidence reliably in the next task.

That is the question behind the Agent Memory Challenge, the public evaluation program of the Agent Memory Leaderboard (AML).

Cycle 2 opens on September 20, 2026, at 00:00 UTC+8.

Join or learn more at agentmemoryleaderboard.ai

Why memory needs to be evaluated differently

As AI agents move beyond one-off chats toward long-running work, memory becomes a core capability.

An agent may need to retain context across conversations, software repositories, documents, images, and evolving tasks. It may need to remember a user preference expressed weeks ago, recover a debugging path from an earlier development session, or distinguish a superseded project decision from the current one.

But “remembering” is not enough.

Retrieving yesterday’s decision after it has been corrected can be worse than retrieving nothing. Returning a familiar but outdated fact can quietly derail a later task. A memory system needs to preserve evidence while also helping an agent understand what remains current.

This makes public comparisons difficult. Different memory systems are often tested with different datasets, answer models, prompts, scoring rules, and evaluation procedures. A score may reflect not only the memory layer, but also everything surrounding it.

AML creates a clearer boundary.

Participants provide the memory layer through Add and Search APIs. AML standardizes the downstream Answer, Eval, result review, and leaderboard-publication process.

The goal is not to declare one universal “best memory system.” It is to create a more open, reproducible way to compare different approaches under a shared evaluation flow.

What Cycle 1 established

AML published its first leaderboard cycle on August 12, 2026.

The first cycle received 136 team applications, and 67 representative memory systems completed official evaluation. AML’s official website has since received more than 300,000 cumulative visits, while its Hugging Face Space reached #1 on the Spaces trending list during the cycle.

Those numbers reflect participation and community interest—not a claim that any individual system is universally superior outside its submitted version, track, suite, and evaluation conditions.

What they do show is that Agent Memory has become a shared technical question: how should systems store experience, retrieve useful evidence, handle change, and prove that the evidence returned to an agent is actually reliable?

Cycle 2 expands that question across three distinct settings.

Three tracks, one evaluation boundary

Textual Memory

The Textual Memory track evaluates long conversations and cross-session history.

It includes explicit fact recall, multi-hop relations, temporal events, personalization, rule following, and memory governance. The track asks whether a system can retrieve relevant evidence without confusing historical statements with current state.

Cycle 2 also introduces Streaming Memory. Here, Add and Search calls may occur as events unfold. The challenge is not simply to search a completed archive, but to use information that was available at the right point in time.

Coding Memory

The Coding Memory track asks a more practical question: can historical engineering experience help an agent solve a later software task?

Systems may need to retrieve and filter repository history such as implementation context, debugging trails, failure cases, module relationships, testing evidence, and prior development decisions.

The formal Coding evaluation includes 150 software-engineering tasks, each tested under both relevant-history and noisy-history conditions—300 task-condition units in total.

The distinction matters. A useful coding memory system should not only find old information. It should help an agent identify which past experience is relevant, which context is distracting, and which evidence can support the next implementation or debugging decision.

Multimodal Memory

The Multimodal Memory track evaluates how systems write, retrieve, and use text-and-image memory.

Participants may process original images, captions, or both. They must correctly handle ordered multimodal content arrays and clearly disclose which media types their systems actually use.

As agents work increasingly across screenshots, product documents, visual references, and written instructions, memory can no longer be treated as text alone. The question is whether a system can preserve and retrieve the connections between visual and textual evidence.

How AML evaluates memory systems

The evaluation boundary is deliberately simple:

Add receives historical content that a system should remember.
Search retrieves relevant memory evidence for a later query and user scope.
AML then runs the downstream Answer and Eval process under a standardized protocol.

Search is not expected to generate the final answer or return a disguised answer key. Its job is to return useful, traceable memory evidence.

The workflow is:

Choose a track and division.
Deploy stable, publicly reachable Add and Search APIs.
Submit a fixed system version.
Pass a per-track Smoke test.
Start a formal Full evaluation.
Keep the submitted version, endpoints, authentication, and operating conditions stable.
AML reviews result validity, version consistency, and compliance status before leaderboard publication.

This separation makes a meaningful comparison possible. Different systems can make different architectural choices—raw-history retrieval, summaries, structured facts, graphs, hybrid retrieval, temporal reasoning, or learned memory policies—while being evaluated through the same downstream Answer and Eval flow.

Technical principles that protect comparability

A benchmark is only useful if the evaluation contract is clear.

Cycle 2 therefore requires several core principles:

Search returns evidence, not a final answer.
user_id is a strict isolation boundary. Evaluation memory must not be shared across users, tasks, samples, teams, or runs.
Synchronous Add must persist content before returning success. Newly written memory must be immediately searchable.
request_id should be idempotent. Network retries should not create duplicate memory.
Full evaluations use a fixed version and contract.
Scores and rankings apply only to the corresponding submitted version, track, suite, and rules.

These constraints are not intended to prescribe a single memory architecture. They make it possible to inspect what a system actually did, reproduce its operating conditions, and interpret its result within a defined scope.

Who can participate

Cycle 2 is open globally to universities and research institutions, open-source maintainers, independent research teams, commercial product teams, individual developers, and cross-organization teams.

Each track has two separate divisions:

Open-source Methods
Commercial Products

Participants may enter multiple tracks, but each submission selects one division. Rankings are published separately by track and division.

Participation is free. Teams operate their own APIs, databases, bandwidth, and compute resources; AML runs the standardized evaluation orchestration, Answer, Eval, result review, and leaderboard publication process.

Eligible Open-source Methods teams will compete for a total RMB 150,000 prize pool across the three tracks. Commercial Products are ranked in a separate division and are not eligible for the prize pool.

Key dates

September 20, 2026, 00:00 UTC+8: Cycle 2 opens
September 20–October 31: Registration, API integration, Smoke tests, Full evaluations, and result review
October 31, 2026, 23:59 UTC+8: Materials submission deadline
November 4, 2026, 23:59 UTC+8: Evaluation closes
Mid-November 2026: Official results planned for release

Teams should submit stable APIs early. Formal evaluation involves resource scheduling, execution, and necessary follow-up review; stable endpoints and operating conditions are essential throughout the process.

From storing history to using experience

Agent Memory is not a single capability.

It includes how a system writes information, organizes it, retrieves it, handles conflict and change, respects isolation boundaries, and supplies evidence that a downstream agent can actually use.

Cycle 2 is an invitation to test these approaches under a common evaluation boundary—across conversations, codebases, and multimodal context.

The benchmark will not settle every question about long-term memory. But it can make the questions sharper, the results more comparable, and technical progress easier to verify.

Agent Memory Challenge 2026 Cycle 2 opens September 20.

Official website · Evaluation entry · Participation guide · API guide · GitHub

Final eligibility, result review, awards, and scheduling are subject to the latest official rules and organizer announcements.

From “Remembering Code” to “Solving Tasks”: How Coding Memory Helps Agents Reuse Engineering Experie

Agent Memory Leaderboard — Thu, 17 Sep 2026 04:14:49 +0000

Compared with textual memory, Coding Memory asks a more direct question:

Can experience accumulated during past software development actually help an Agent complete the next software-engineering task?

This is the question behind the first Agent Memory Leaderboard (AML) Coding Memory evaluation.

A Coding Agent does not work in an empty repository.

It enters a codebase with a history:

previous feature implementations;
old bug reports;
rejected approaches;
commit messages;
test failures;
error traces;
code-review discussions;
file paths;
function names;
and development sessions that may contain clues about how the project actually works.

Some of this history is useful. Some of it is outdated. Some of it is highly relevant but difficult to find. Some of it may lead an Agent in the wrong direction if retrieved without context.

The purpose of Coding Memory is not simply to store this history.

It is to help an Agent use the right part of it at the right moment.

The first AML Coding Memory benchmark was built from 12 real code repositories. It organized 1,290 annotated historical engineering tasks and evaluated systems on 150 held-out tasks:

51 New Feature tasks
99 Bug Fix tasks

The memory system writes and retrieves relevant historical information. A standardized Coding Agent then uses the retrieved context to solve the current task.

That design makes the final metric unusually concrete.

The question is not only:

Did the system retrieve relevant historical content?

It is:

Did that historical content help the Agent complete the software-engineering task?

The first leaderboard results show two very different states.

In the industry ranking, a visible gap has already appeared at the top.

In the open-source ranking, leading methods remain tightly clustered.

Together, these results suggest that Coding Memory is beginning to develop distinct technical routes—but no single architecture has yet become the final answer.

1. Coding Memory Is Not Just Textual Memory Applied to Code

Textual memory is often framed as a retrieval problem.

A user asks a question. The system retrieves relevant past conversation. An answer model reasons over the returned context.

Coding Memory has to operate under a more demanding version of this pattern.

The information an Agent needs may not appear in one clean natural-language sentence.

It may be distributed across:

an issue describing an earlier failure;
a pull request that introduced a similar feature;
a commit that explains why a particular workaround exists;
an error log showing the actual failure mode;
a test that captures the intended behavior;
and a later code review explaining why the first solution was rejected.

A coding task may also require multiple forms of reasoning at once.

The Agent may need to understand:

repository structure;
dependencies between modules;
naming conventions;
historical implementation patterns;
current test expectations;
and the difference between a previous fix and the current failure.

This means a Coding Memory system cannot be evaluated only by whether it retrieves similar-looking text.

It must help the Agent make a better engineering decision.

The history must eventually influence the outcome of a real task:

a feature is implemented correctly, a bug is fixed, tests pass, and the patch fits the existing project.

2. What AML Measures in Coding Memory

The AML Coding Memory benchmark separates the memory layer from the Coding Agent.

The memory system is responsible for two broad jobs:

writing historical development information;
retrieving relevant information for a new task.

The downstream Coding Agent is responsible for the final implementation work.

This separation matters.

A memory system may retrieve a large amount of accurate history but still fail to help the Agent if the returned context is too broad, too noisy, too old, or not actionable.

Likewise, an Agent may occasionally solve a task without useful historical memory—but that does not prove the memory system is strong.

AML therefore evaluates the result at the task level.

The central metric is Task Solve.

This moves Coding Memory evaluation beyond questions such as:

How many memories were stored?

How high was Recall@K?

How similar was the retrieved content to the query?

Those signals can still be useful for diagnosis. But they are not the final goal.

The final question is whether history improves the Agent’s ability to complete the work.

3. The First Results: Two Different Leaderboard Shapes

The first Coding Memory results reveal a sharp contrast between industry systems and open-source methods.

The industrial ranking already has a clear leading system.

The open-source ranking has a large group of systems separated by less than one percentage point.

This does not mean the industrial systems all use one shared method, or that open-source systems are technically identical.

Instead, it suggests that the two groups are at different stages of differentiation.

Industrial systems are beginning to show more visible product-level separation.

Open-source systems are still exploring a broad design space in which several very different architectures can achieve nearly the same overall result.

That makes the leaderboard more interesting than a simple list of scores.

The central question becomes:

What kinds of memory are different systems trying to build for Coding Agents?

4. Industry Ranking: MemoraX Leads by Ten Points

In the industry ranking, MemoraX achieved the highest overall result.

Its Task Solve rate was 62.00%.

Its task-type results were:

New Feature: 70.59%
Bug Fix: 57.58%

The next group—claude-mem, hs, and MemOS—each achieved 52.00% overall Task Solve.

This creates a ten-percentage-point gap between MemoraX and the second tier.

For a benchmark built around real software-engineering tasks, this is a meaningful difference.

It suggests that the way a system selects, updates, and reuses historical experience can matter materially—not only for retrieval quality, but for whether the downstream Agent completes the task.

The public materials behind the leading systems also reveal different product philosophies.

Some systems emphasize preserving an Agent’s development trajectory and restoring it when needed.

Others try to identify which experiences from long-running development work are worth reusing in future tasks.

5. MemoraX: Memory as Reusable Engineering Experience

Public materials indicate that MemoraX Code does not treat Memory as only a “store and retrieve” layer.

It also considers:

which information is worth writing;
which memory should be updated;
when memory should be recalled;
and how prior engineering work can become useful in a later task.

MemoraX Code combines local repository memory with long-term memory.

Its Memory Model learns mechanisms for memory filtering, updating, and recall.

But the more distinctive part of the approach is what happens on top of historical development trajectories.

The system extracts Procedure Memory.

Procedure Memory is not simply a record of what happened in the past.

It aims to capture reusable engineering experience:

how a similar problem was analyzed;
which files or modules were inspected;
what implementation path was chosen;
what alternatives failed;
how the result was verified;
and which workflow may be useful again.

In one public experiment, MemoraX distilled 15 engineering experiences from 123 historical task segments, then organized them into four categories of Procedure Memory.

This introduces a different question for Coding Memory.

Instead of asking only:

What happened in the past?

the system asks:

Which past engineering experience should influence the next task?

That distinction matters.

A raw historical record can tell an Agent that a bug happened before.

A reusable procedure may help it decide where to begin, what to check, and how to verify the fix.

6. From Event History to Procedure Memory

Consider a simplified example.

A previous task involved an API timeout.

The development trajectory may have included:

a bug report about transient failures;
inspection of a network client;
an unsuccessful first retry implementation;
a revised retry strategy;
added error handling;
and new regression tests.

A conventional memory system may store each event independently.

It can later retrieve the issue, the relevant commit, or the test file.

That can be useful.

But a procedure-oriented memory system attempts to preserve something more general:

When this project encounters transient API failures, inspect the client boundary, distinguish retryable errors from permanent errors, use the established retry pattern, and validate the behavior with the existing test conventions.

The goal is not to replace raw evidence with an unsupported summary.

It is to make past engineering experience easier to reuse when a new task is structurally similar.

This is especially relevant for long-running repositories.

As development history grows, repeatedly rediscovering the same debugging and implementation patterns becomes expensive.

A Coding Agent that can reuse verified procedures may spend less time repeating exploratory work.

7. claude-mem: Preserve the Development Trail

claude-mem emphasizes another important form of Coding Memory.

It continuously records the development activity of a Coding Agent.

This can include tool calls, observations, and other traces created while an Agent works through a task.

Those records are organized into more compact semantic entries and persist across sessions.

When a new session begins, the Agent does not need to load all historical activity into its context window.

Instead, it can:

search relevant records;
inspect a timeline around a promising result;
recover detailed content only when the current task requires it.

This is a layered-retrieval approach to coding history.

It tries to avoid two opposite failure modes.

The first is loading too much past history into context at once.

The second is compressing history so aggressively that the Agent cannot recover the original reasoning trail when it needs more detail.

By preserving a searchable semantic record and a timeline that can expand into fuller context, claude-mem aims to make development sessions resumable.

Its first AML result was:

Overall Task Solve: 52.00%
New Feature: 56.86%
Bug Fix: 49.49%

This illustrates a different Coding Memory thesis:

A useful Agent does not always need a distilled procedure. Sometimes it needs to resume the actual trail of work.

8. Persistent Development History Is Also a Product Capability

Development work often unfolds across multiple sessions.

An Agent may inspect a repository today, identify a likely cause, make a partial change, encounter a failed test, and continue later.

Without memory, the next session may need to rediscover:

which files were relevant;
which commands were already run;
which hypothesis had been rejected;
what error output was observed;
and why a particular implementation path was chosen.

This is not only a retrieval problem.

It is a continuity problem.

A timeline-oriented memory system treats the Agent’s work itself as part of the memory.

The history is not limited to the user’s request or the final patch.

It also includes the path through the repository.

This can be useful when the next task is not identical to the previous one but touches the same modules, architectural decisions, or unfinished investigation.

The broader implication is that Coding Memory can preserve more than facts about a codebase.

It can preserve the development process that produced those facts.

9. Industry Systems Are Beginning to Diverge

The first industrial leaderboard suggests that Coding Memory is no longer one generic product category.

Different systems are beginning to emphasize different kinds of value.

One route focuses on continuous capture and recovery of an Agent’s development trajectory.

Another focuses on filtering and distilling long-term work into reusable engineering procedures.

Both approaches address the same underlying problem:

How can an Agent use past development work without repeatedly loading, reading, and rediscovering everything?

But they make different choices about what memory should look like.

Should memory preserve the sequence of work?

Should it extract generalized experience?

Should it do both?

The first leaderboard does not settle these questions.

But it shows that these design choices are starting to appear in final software-engineering outcomes.

10. Open Source: Eight Methods Tied for First

The open-source ranking looks very different.

Eight methods tied for first with an overall 52.67% Task Solve rate:

AM-Link
AMC-Memory
aml-memory-baseline
aml-memory-mvp
causal-memory
Hybrid Episodic Memory
Memoria
nano-memory

Immediately behind them, agent-memory, ChronoHybridMem, mem0-BQE, and MemoryBear each achieved 52.00%.

Across more than a dozen leading open-source systems, the difference in final task success is less than one percentage point.

At this stage, ranking systems strictly as first, second, or third is less revealing.

The more interesting question is:

How can different memory architectures produce nearly identical task-level results?

The answer appears to be that Coding Memory still has multiple viable routes.

Some systems preserve raw history.

Some rely on lexical and semantic hybrid retrieval.

Some add code-specific retrieval signals.

Some create more structured memory representations.

The first AML results do not show that one route has already won.

11. causal-memory: Preserve History, Then Retrieve It Well

The causal-memory submission primarily preserves original historical information.

At retrieval time, it combines lexical matching with semantic retrieval. Different signals are fused to rank historical content relevant to the current task.

The overall idea is direct:

Do not impose heavy abstraction on the history before storage.

Preserve the original engineering record, then make retrieval effective.

Its results were:

Overall Task Solve: 52.67%
New Feature: 62.75%
Bug Fix: 47.47%

This is important because it challenges a common assumption.

It is tempting to believe that Coding Memory must first convert development history into increasingly sophisticated summaries, entities, graphs, or abstractions.

But the first results suggest that this is not necessarily required for a competitive system.

Raw history can remain valuable.

If the system can retrieve the right issue, commit, code change, error trace, or discussion at the right time, the original record may be more useful than a more aggressively transformed representation.

12. Why Raw History Can Matter in Code

Software-engineering context is often highly specific.

A file path can matter.

A function name can matter.

A single error string can matter.

A note in a previous pull request can matter.

A summary may preserve the broad idea of a previous change while losing the precise information needed to make the next patch.

For example, a summary may say:

The API client was updated to handle transient failures.

But the original engineering record may reveal:

which exception types were retryable;
which function was modified;
where retries were intentionally capped;
which test exposed the failure;
and why another implementation was rejected.

The broad summary is useful for orientation.

The raw record may be necessary for correct implementation.

A Raw-First approach does not eliminate the need for structure.

It simply treats the original historical record as something that should remain available rather than being replaced completely.

13. Memoria: Hybrid Retrieval Optimized for Code

Memoria also uses hybrid retrieval, but adapts it more directly to the details of software engineering.

In addition to semantic retrieval and full-text keyword search, it considers signals such as:

function names;
file paths;
snake_case;
CamelCase;
exception messages;
code tokens;
and historical messages near retrieved records.

This is a practical response to the fact that code is not ordinary prose.

A developer—or a Coding Agent—may search using an error string, a function name, a package path, or a specific identifier that has little meaning outside one repository.

Natural-language semantic similarity alone may not capture these signals reliably.

Memoria’s code-aware retrieval design gives these concrete engineering details a direct role in search.

Its results were:

Overall Task Solve: 52.67%
New Feature: 60.78%
Bug Fix: 48.48%

The system illustrates a broader point:

Coding Memory should reflect how software projects are actually named, navigated, debugged, and changed.

14. Code Tokens Are Not Noise

In some memory tasks, highly specific tokens may look like clutter.

In coding tasks, they are often the most valuable part of the context.

Consider the following kinds of information:

fetchWithRetry
api/client.ts
ECONNRESET
NullPointerException
get_user_profile
src/services/auth
Retry-After
test_rate_limit_timeout

A purely semantic retriever may recognize that a query involves an API failure.

But it may not know which historical record contains the exact function, error signature, or test convention that matters.

Code-aware retrieval can treat these terms as first-class evidence.

It can also use surrounding historical messages to restore context around a matching record.

This is especially useful because software engineering often involves local patterns.

The same repository may use one naming convention, one folder structure, one error-handling pattern, and one test style that differ from another project.

A useful Coding Memory system should help the Agent retrieve repository-specific knowledge rather than relying only on general programming knowledge.

15. agent-memory: A Direct Raw-First Alternative

agent-memory adopts a more direct Raw-First approach.

Historical messages are not first converted into new LLM-generated memory summaries.

They are stored as original records.

At retrieval time, the system combines BM25 and local dense retrieval, then fuses the results.

Its first AML scores were:

Overall Task Solve: 52.00%
New Feature: 50.98%
Bug Fix: 52.53%

This result reinforces the observation from causal-memory.

More complex pre-storage abstraction is not a prerequisite for useful Coding Memory.

A system can preserve raw engineering records, combine lexical and dense retrieval, and still remain competitive in task-level results.

The question is not whether raw history or abstraction is always better.

The question is whether the representation helps the Agent retrieve information that is actionable for the current task.

16. Different Routes, Similar Overall Results

The open-source leaderboard contains several systems with similar overall scores but different design choices.

causal-memory emphasizes original historical information plus hybrid retrieval.

Memoria adds code-specific signals such as paths, identifiers, and exception messages.

agent-memory retains raw messages and fuses BM25 with local dense retrieval.

Their close overall results show that a high-level score can hide meaningful architectural differences.

This is why Coding Memory should not be reduced to one broad label.

Two systems may both claim to “retrieve relevant history,” while one is optimized for exact engineering tokens, another for semantic similarity, and another for preserving the raw development record.

These differences may become more visible as benchmarks expand, repositories become larger, and task types become more diverse.

17. New Features and Bug Fixes May Need Different Memories

The split between New Feature and Bug Fix tasks is particularly revealing.

Consider three representative open-source results:

System	New Feature	Bug Fix
causal-memory	62.75%	47.47%
Memoria	60.78%	48.48%
agent-memory	50.98%	52.53%

These numbers do not prove that one memory architecture is inherently better for one class of task.

The benchmark is not yet large enough to support such a universal conclusion.

But the pattern suggests an important possibility:

New Feature development and Bug Fix work may benefit from different kinds of historical memory.

18. What New Feature Tasks May Need

When implementing a new feature, an Agent may need to understand how the repository prefers to build things.

Useful historical context may include:

similar feature implementations;
module boundaries;
architectural patterns;
data-flow conventions;
interface design;
project-specific abstractions;
and tests that show how new behavior is normally introduced.

The key question may be:

How has this project solved a structurally similar problem before?

In this setting, a previous implementation pattern can be more useful than an isolated error message.

A system that can retrieve related modules, prior feature work, and established project conventions may provide a stronger starting point for the Agent.

19. What Bug Fix Tasks May Need

Bug Fix work often asks for another kind of history.

The useful evidence may include:

error messages;
failing tests;
stack traces;
affected files;
prior failed attempts;
earlier bug reports;
previous fixes;
and the verification process that confirmed a repair.

The question is often not:

How should this repository generally implement a feature?

It is:

Where has this failure appeared before, what did people try, and which path actually worked?

A Bug Fix memory system may therefore benefit from more precise retrieval of diagnostics and local change history.

It may need to find the relevant function, exception signature, test failure, or prior patch with much higher specificity.

This may help explain why systems with similar overall Task Solve rates can differ noticeably when New Feature and Bug Fix tasks are separated.

20. Coding Memory Is Task-Conditioned Memory

This leads to a broader conclusion.

There may not be one universally optimal Coding Memory.

The right retrieval strategy may depend on the task.

A New Feature task may need:

architectural context;
reusable patterns;
related implementation examples;
and project conventions.

A Bug Fix task may need:

diagnostic evidence;
error-specific history;
local modifications;
failed attempts;
and validation traces.

A general-purpose Coding Memory system may eventually need to recognize which kind of task it is facing before deciding:

what to retrieve;
how much context to return;
whether to prioritize broad architectural examples or narrow failure evidence;
and whether a reusable procedure is more valuable than a raw historical record.

This is a harder problem than simply searching a repository history.

It is a problem of matching the shape of memory to the shape of work.

21. From Retrieval Quality to Engineering Outcomes

The first AML Coding Memory leaderboard moves memory evaluation forward in an important way.

It does not stop at asking whether an Agent can remember the past.

It asks whether the past changes what the Agent can accomplish.

A memory system may look impressive if it stores rich trajectories, builds structured records, or retrieves semantically similar text.

But these capabilities matter only if they improve the next engineering decision.

Can the Agent locate the correct part of the repository more quickly?

Can it avoid repeating a failed attempt?

Can it reuse an implementation pattern that was already validated?

Can it understand why a previous fix worked?

Can it add the right tests?

Can it solve the current task with fewer unnecessary steps?

These are the questions that turn memory from a context store into engineering infrastructure.

22. From “Remembering the Past” to “Reducing Repeated Trial and Error”

The first Coding Memory results do not yet show that one technical route has become the definitive answer.

The leading industry score was 62.00%.

The leading open-source tier reached 52.67%.

Both commercial and open-source systems have substantial room to improve.

But the field has already moved beyond a narrower view of memory.

Memory is no longer only about preserving prior interactions.

It is becoming a way to reduce repeated trial and error.

A useful Coding Memory system should not merely tell an Agent:

Here is something related that happened before.

It should help the Agent understand:

Here is the past experience that matters for this task.

Here is the relevant implementation, failure, or procedure.

Here is the context needed to use it correctly.

And here is the evidence that allows you to verify it.

23. The Second Agent Memory Challenge Opens Soon

The first AML Coding Memory results showed both meaningful progress and major open questions.

Industrial systems have begun to show a visible separation at the top of the ranking.

Open-source methods remain highly competitive, but the leading systems are still closely grouped.

Raw-history retrieval, code-aware hybrid search, session restoration, and reusable procedure memory are all active directions.

The next question is not simply:

Which system can retrieve more history?

It is:

Can Agent Memory become more task-aware, more reusable, and more reliable across real long-term engineering work?

The Second Agent Memory Challenge opens on September 20, 2026.

This cycle includes three tracks:

Textual Memory — long-horizon conversation retrieval, temporal reasoning, and user-profile alignment
Coding Memory — development-context retrieval for real software-engineering tasks
Multimodal Memory — cross-modal memory association across text, images, audio, and video

Each track includes both an Open-Source Methods Ranking and an Industry Systems Ranking.

The challenge is globally open to universities, research institutions, companies, open-source teams, and individual developers. There is no registration fee.

The open-source ranking includes a total prize pool of RMB 150,000, with awards including:

First Prize: RMB 20,000
Second Prize: RMB 8,000
Third Prize: RMB 3,000
Best Technical Innovation Award: RMB 5,000

Important dates:

Registration and submission entry opens: September 20, 2026, 00:00
Rolling evaluation period: September 20 – October 31
Submission deadline: October 31, 2026, 23:59
Evaluation queue closes: November 4, 2026, 23:59
Results planned for release: mid-November 2026

Each participant may initiate up to two full official evaluations.

Once evaluation begins, the submitted version is frozen. The platform then evaluates the system automatically through the standardized Add/Search interface.

Whether you are building a memory layer for conversational Agents, Coding Agents, or multimodal systems, the second AML cycle is an opportunity to test what your architecture can contribute under a shared and reproducible evaluation setting.

Registration opens on September 20. We look forward to seeing what the next generation of Agent Memory systems can do.

More information:

https://mp.weixin.qq.com/s?__biz=MzcwOTM5MjU5OA==&mid=2247483788&idx=1&sn=05b1e0c52ab185fe3330361aacbef351&scene=21#wechat_redirect

AML

Agent Memory Leaderboard

https://agentmemoryleaderboard.ai/

Leaderboard

https://huggingface.co/spaces/agent-memory-leaderboard/leaderboard

Content Notice

This article is based on publicly available system materials and the first-cycle AML Coding Memory leaderboard results. It reflects an analytical interpretation of the systems discussed and does not represent an official technical recommendation from AML.

Scores apply only to the referenced evaluation cycle, track, submitted system version, and official leaderboard release. They should not be interpreted as guarantees of performance in every software-engineering scenario.

From “Deciding” to “Retrieving”: How FlowGrid Turns Project History into Agent Memory Evidence

Agent Memory Leaderboard — Wed, 16 Sep 2026 03:27:44 +0000

FlowGrid AML Retriever ranked #8 on the first AML Open Leaderboard with an Overall Score of 43.98.

But what makes its approach interesting?

Many Agent Memory systems begin with a familiar question:

Given the current query, how can an Agent retrieve the most relevant past information?

FlowGrid begins one step earlier.

Before a system can retrieve useful memory, someone needs to decide what counts as a decision, what changed afterward, what remains unresolved, and where the supporting evidence came from.

Its original form was not a vector database or a memory API.

It was a Markdown decision log written for people.

That starting point shaped the system’s core idea:

Agent Memory should not only preserve what was said. It should preserve how project judgments were made, what evidence supported them, and how those judgments changed over time.

In its v1.0 submission to the first Agent Memory Challenge, FlowGrid AML Retriever turned this philosophy into a lightweight Add/Search memory service.

It preserved original messages, created multiple retrieval views, combined deterministic retrieval signals, and returned traceable evidence rather than final answers.

The result was a deliberately constrained system:

The retriever should find evidence.

The answer model should reason over evidence.

Neither should silently replace the historical record.

After the competition, the project continued as FlowGrid Agent Memory, where the original retrieval foundation became part of a broader system for memory states, authorization, replacement, deletion, and minimal-context compilation.

But the competition entry remains useful as a focused case study:

What happens when a human-readable project decision log becomes an Agent Memory retriever?

1. The Core Idea: Memory Is More Than Stored Facts

Consider a familiar project scenario.

A team first records:

The release date is August 10.

A few days later, a new note says:

The release date has moved to August 14 because testing needs four more days.

Later, an Agent is asked:

What is the current release date?

A basic memory system may retrieve one date.

A stronger system may retrieve both dates.

But a useful project-memory system also needs to preserve:

which statement came later;
whether the later statement was actually an update;
why the change happened;
where both statements came from;
and whether the older record should remain available for audit.

This is the distinction FlowGrid starts from.

Its memory system is not designed only to retrieve a sentence that looks relevant.

It is designed to return historical evidence that can still be inspected when a project decision needs to be understood, challenged, or revised.

Instead of asking:

“What should the system remember?”

FlowGrid asks:

“What record would let a person or an Agent understand why the current project state exists?”

2. The Starting Point: A Markdown Decision Log

Before FlowGrid became an AML submission, it was a Markdown decision log.

The log recorded project judgments in a human-readable format:

decision status;
project stage;
background;
core question;
candidate options;
final choice;
reasons for rejecting alternatives;
risk assessment;
follow-up validation;
and review entry points.

This mattered for a simple reason.

A project is not only a collection of facts.

It is a sequence of choices made under uncertainty.

A Markdown file can be opened by a human, reviewed in Git, compared across commits, and read even when no Agent is available.

Someone can see what the team decided, which alternatives existed, why a direction was chosen, and what conditions would trigger reconsideration.

As projects and Agent collaboration expanded, this single-file approach encountered predictable limits.

Decisions became distributed across messages, sessions, and files.

Older states could remain in context after new information appeared.

New evidence could fail to reach the task that needed it.

FlowGrid gradually added source tracing, temporal states, conflict preservation, and retrieval.

The AML branch then reduced this broader system into the competition’s required Add and Search interfaces.

3. From Human Decision Records to an AML Retriever

AML separates memory retrieval from answer generation.

The memory system receives conversation history through Add.

Later, it receives a query through Search.

The system returns retrieved memory evidence.

A standardized answer model then uses that evidence to generate the final answer.

FlowGrid AML Retriever operates within exactly that boundary.

It does not generate the final answer.

It does not claim that a derived summary is more authoritative than the original conversation.

It stores historical messages, retrieves relevant evidence, and returns enough metadata for the next stage to understand where the evidence came from.

This separation allows three different questions to be evaluated independently:

Was the memory written correctly?
Was the right historical evidence retrieved?
Did the answer model reason correctly over that evidence?

For FlowGrid, this boundary is not just an evaluation requirement.

It is part of the system’s reliability model.

4. Add and Search: A Small API with Strict Guarantees

FlowGrid AML Retriever exposes two core operations.

Add receives user, session, and message data.

The system returns success only after synchronous persistence is complete. Once the request succeeds, the newly written message is already searchable.

Repeated requests with the same request_id and user_id are handled idempotently. This prevents a network retry from writing the same memory twice.

Search is always restricted to one exact user_id.

It returns memory evidence ranked by relevance.

The official platform then performs Answer and Eval.

These details may sound operational rather than algorithmic. But memory reliability depends on them.

A system needs to answer more than:

“Did I retrieve useful information?”

It also needs to answer:

“Did I retrieve it from the correct user?”

“Was the message already committed when I said it was stored?”

“Did a retry create duplicate memory?”

“Can the returned result be traced back to a source?”

For FlowGrid, these are part of the same evidence story.

5. Preserve the Original Message

Long conversations rarely arrive in a retrieval-friendly form.

Some questions need one direct fact.

Some require two or three adjacent turns.

Others depend on seeing conditions, actions, and outcomes together.

If a system only retrieves isolated messages, pronouns, causal explanations, and time conditions may become detached from the statement they qualify.

If it stores only large summaries, names, numbers, dates, and exact phrasing can be diluted.

FlowGrid therefore preserves every original message.

The original record remains the source of evidence.

Derived retrieval views may make the record easier to find. They do not replace it.

This produces a basic rule:

Retrieval representations can change. Historical evidence should remain recoverable.

6. Three Retrieval Views, One Evidence Source

FlowGrid creates three retrieval views from the same conversation history.

Single-message view

This view helps locate direct statements, names, numbers, and dates.

It is useful when the answer is explicitly stated in one message.

Sliding-window view

This view includes adjacent dialogue turns.

It helps restore references, conditions, and supplementary information that a single message may not contain on its own.

Session-segment view

This view preserves a larger local event structure.

It is useful when a question depends on seeing a sequence of related statements within the same conversation segment.

The important point is that these views change retrieval granularity, not evidence ownership.

Each derived view retains source_message_ids.

The system can always return from a window or session segment to the original messages that created it.

Conceptually:

Original Message

↓

Single Message / Sliding Window / Session Segment

↓

Retrieved Candidate

↓

Source Message IDs

↓

Original Evidence

This avoids a common memory-system failure mode:

Derived representations become easy to retrieve but difficult to verify.

7. Why Multiple Views Matter

Suppose a user says:

Maya is planning the launch.

Later:

She wants the release moved because the final test is incomplete.

And later still:

What changed about the launch schedule?

A single-message search may find the release change.

But it may not clearly resolve who “she” refers to.

A larger session segment may preserve the identity, the proposed change, and the reason together.

At the same time, a broad segment can be less precise than a single message when the question asks for a date, a number, or a direct statement.

No one retrieval granularity works for every question.

FlowGrid’s answer is not to choose one universal memory unit.

It is to keep several views available while binding them all to the same underlying historical record.

8. Deterministic Hybrid Retrieval

FlowGrid v1.0’s default retrieval path does not use embeddings or external LLM calls.

It relies on Python’s standard library, SQLite, and FTS5.

The system combines several interpretable signals to address common failures in long-conversation retrieval.

SQLite FTS5 provides the lexical retrieval foundation.

English, numbers, and dates are handled as lexical units.

For Chinese text, character unigrams and bigrams help compensate for the limitations of word segmentation.

A lexical approach has clear limits.

It struggles with paraphrases that are semantically similar but use very different wording.

But it remains strong for proper nouns, dates, numbers, direct quotations, and exact terms from the original conversation.

For many project-memory tasks, those details matter.

9. Retrieval Signals Beyond Keywords

After initial candidate retrieval, FlowGrid calculates additional deterministic signals.

These include:

original-text substring matches;
term coverage;
named entities;
numbers;
dates;
answer options;
temporal signals;
and adjacent-context signals.

For multiple-choice queries, the answer options can also participate in retrieval.

This gives the system extra clues for people, locations, and candidate facts.

The point is not to claim that every signal is universally useful.

The point is to reduce dependence on one signal alone.

A keyword match may locate a date.

An entity signal may recover a person.

A window view may restore the condition surrounding that person.

A temporal signal may help distinguish an older statement from a later one.

10. Fusion at the Rank Level

Different retrieval methods produce scores on different scales.

A BM25 score is not directly comparable to a custom entity-match score or a temporal feature score.

FlowGrid avoids treating these values as if they meant the same thing.

Instead, BM25 ranking and deterministic feature ranking enter weighted Reciprocal Rank Fusion.

The fusion happens at the ranking level.

This makes it possible to combine multiple retrieval paths without forcing their raw scores into a shared meaning they do not naturally have.

After fusion, duplicate views and evidence completely covered by stronger results can be removed.

This creates space for more independent sources in the final evidence set.

The downstream model receives not a single opaque score, but a ranked collection of traceable historical records.

11. The Competition Result

In the first Agent Memory Leaderboard academic textual-memory ranking, FlowGrid_AML_Retriever v1.0 achieved:

Rank #8

Overall Score: 43.98

The first-place score was 45.06.

The gap was 1.08 points.

FlowGrid did not rank first in any individual category.

Its scores included:

Explicit fact recall: 55.59
Relational and multi-hop compositional reasoning: 45.19
Personalization and care: 51.29
Temporal and event-sequence reasoning: 21.13
Memory governance: 27.86

The latter two were the system’s relative weaknesses.

This result is interesting because the default submission did not rely on an external model call or a heavyweight retrieval stack.

Its competitiveness came from preserving evidence, using multiple views, combining complementary signals, and keeping every derivation connected to source messages.

12. A State Update Is Not Just a Later Timestamp

Temporal memory is one of the hardest parts of long-term Agent Memory.

Return to the release-date example.

The system can find both August 10 and August 14.

But a later message is not automatically a replacement for an earlier one.

A newer message about the same project may be unrelated.

It may mention a date in a different context.

It may discuss a possible plan rather than confirm an update.

A naive strategy such as “always down-rank old values” can improve some temporal questions while damaging others.

FlowGrid’s post-competition experiments showed exactly this.

A more aggressive old-value penalty improved part of the temporal behavior but reduced overall MRR.

The reason was concrete:

Topic similarity plus recency was not enough to prove a state update.

13. Protected State-Update Reranking

FlowGrid v1.1 introduced a narrower post-competition mechanism: protected state-update reranking.

The system adjusts ranking only when three conditions hold:

the query has temporal intent;
the old and new evidence are highly related;
the newer message contains explicit update, correction, delay, or invalidation semantics.

Only then does the system slightly promote the newer evidence and lower the older one.

The older record is not deleted.

It remains preserved and retrievable.

This design reflects a broader principle:

A system should not infer a new state merely because a similar statement appeared later.

The mechanism is intentionally limited.

When the evidence is insufficient, the system keeps the original ranking.

14. Governance for Humans and Governance for Models

FlowGrid’s original governance problem was larger than an AML retrieval benchmark.

A project owner may need to know:

Which state is currently adopted?
Who changed it?
When did it change?
Why was the previous state preserved?
Is a conflict resolved?
Can the team recover context if the current judgment turns out to be wrong?

A standardized answer model has a narrower need.

If it sees both old and new timestamped evidence, it may be able to infer the current state.

But a leaderboard result raises an important question:

Does storing more governance metadata actually improve the evidence returned to the answer model?

Not always.

A system may have many internal state labels while still returning weak evidence.

An outdated value may remain influential as long as it appears in Top K.

FlowGrid’s post-competition response is to treat governance mechanisms as testable interventions rather than automatic improvements.

A mechanism should enter the default path only if it can show:

what error class it repairs;
what other queries it might harm;
and whether it meaningfully changes the returned evidence set.

15. Official Results and Local Experiments Are Different Evidence

The official AML score of 43.98 and the #8 rank apply to FlowGrid_AML_Retriever v1.0.

They do not apply automatically to subsequent versions.

The public FlowGrid Agent Memory repository includes an AML Add/Search adaptation layer at v1.1.0, including protected state-update reranking.

That newer mechanism does not yet have a new official AML leaderboard result.

Its metrics come from deterministic synthetic experiments designed for local version comparison.

They should not be presented as a new AML overall score.

In local paired experiments using classic, medium, and mixed settings, three fixed seeds, and top_k 100, the v1.0 baseline achieved:

Recall@20: 0.9948

Recall@100: 1.0000

MRR: 0.6728

With protected state updates, v1.1 maintained the same Recall@20 and Recall@100 while increasing MRR to 0.6948.

A more aggressive temporal strategy could raise MRR to around 0.7040.

However, it reduced Recall@20 in one seed from 0.9870 to 0.9844.

FlowGrid did not choose that setting.

The system prioritizes stable high recall rather than exchanging recall robustness for a small increase near the top of the ranking.

16. What the Local Experiments Actually Show

The local experiments support a limited conclusion.

Under the current synthetic evaluation protocol, explicit update semantics can reduce errors introduced by blunt temporal reranking.

They do not prove that the same gain will appear on AML’s hidden data.

This distinction is important.

A good Agent Memory research process should separate:

official leaderboard validation;
local paired experiments;
diagnostic experiments;
and product-engineering acceptance checks.

These forms of evidence answer different questions.

The leaderboard tests a submitted version under a shared evaluation contract.

Local experiments compare alternative mechanisms under controlled conditions.

Diagnostic experiments identify why a specific behavior occurs.

Engineering checks determine whether a feature is stable enough for a product path.

Conflating them makes memory research less trustworthy, not more.

17. Where Rank #8 Came From

FlowGrid’s result did not come from a single decisive model component.

It reduced several common failure modes at once.

Original messages were preserved, so writing did not lose detail through summarization or fact extraction.

Single messages, windows, and session segments provided multiple retrieval scales.

Chinese character fragments, entities, dates, numbers, and answer options supplemented baseline lexical retrieval.

Derived views remained bound to original messages.

Results could be verified and deleted at the user level.

The default path used no third-party Python dependency and no external model call.

Add, Search, idempotency, user isolation, and write-then-search behavior could all be reproduced deterministically.

This is not a claim that lightweight lexical retrieval solves Agent Memory.

It is a demonstration that evidence-first retrieval can remain competitive without making every stage dependent on external models.

18. The Trade-Offs Are Real

FlowGrid’s limitations are as clear as its strengths.

A mostly lexical system has limited ability to recover paraphrases with large semantic distance from the original statement.

Temporal paraphrases can be especially difficult.

SQLite is well suited to a lightweight single-node implementation, but it is not itself a distributed memory-service architecture.

Soft reranking can improve evidence ordering, but it cannot replace complete state resolution.

The system can preserve conflicting history.

It cannot automatically decide every real-world conflict without a stronger policy, additional evidence, or human authorization.

Rank #8 shows that the approach was competitive in the first AML leaderboard.

It does not show that long-term memory governance is finished.

19. After AML: FlowGrid Agent Memory

After the competition, the project’s main direction shifted toward FlowGrid Agent Memory.

The AML Add/Search interface remains as a compatibility and evaluation layer.

The broader product core separates memory into three layers.

Raw events

Immutable records of what happened and where the information came from.

Candidate memories

Memories proposed by rules or models.

Candidate, inferred, and unknown states do not automatically become user-confirmed facts.

Confirmed current state

Information accepted by a user, project owner, or authorized policy.

Superseded, rejected, and deleted information does not return to ordinary Agent continuation context.

However, it can remain available in an authorized audit mode.

20. Current State Resolver and Context Compiler

The general-purpose system introduces two core components.

The Current State Resolver determines what is currently valid.

The Context Compiler creates the smallest appropriate memory package for a task, based on scope, permissions, and token budget.

This is a significant shift from a competition retriever.

A benchmark often asks:

Can the system return relevant evidence?

A real long-term Agent also needs to ask:

Is this evidence currently active?

Is it authorized for this task?

Does the Agent need raw history, a confirmed state, or both?

How much context should be included?

Models can help extract potential memory.

But FlowGrid keeps a source-anchor requirement and limits model output to the candidate stage.

A model can propose a memory.

It cannot silently promote its own proposal into confirmed project state.

21. From Benchmark Adapter to General Memory Infrastructure

This distinction protects the general-purpose product from becoming a collection of benchmark-specific rules.

The competition interface belongs in adapters and evaluation configuration.

The governance core should remain applicable to real projects where memory is not only a retrieval task.

The next evidence target is therefore not merely another leaderboard score.

It is whether memory improves continuation across long-running real tasks.

Can an Agent resume work with less repeated explanation?

Can it distinguish active project decisions from rejected alternatives?

Can it retrieve relevant history without reintroducing invalidated context?

Can a human understand why a memory appeared?

These are harder questions than retrieving one correct sentence.

They are also closer to the problems persistent Agents will face in real work.

22. From Decision Logs to Evidence Systems

FlowGrid began with a simple human need:

Keep a readable record of why a project chose one direction over another.

That led to a Markdown decision log.

The log then developed source tracing, temporal state, conflict preservation, and retrieval.

AML Retriever adapted the approach to Add and Search.

FlowGrid Agent Memory extends the same foundation into raw events, candidate memories, confirmed states, authorization, and minimal context.

Across these stages, the core principle remains consistent:

Interpretation can evolve. Evidence should remain traceable.

23. What FlowGrid Adds to the AML Landscape

The first AML leaderboard includes systems that make different architectural choices.

Some emphasize learned representations.

Some prioritize structured memory.

Some focus on retrieval pipelines.

Some use models heavily during write or search.

FlowGrid represents a different point in the design space:

Preserve the original history.

Create multiple retrieval views.

Use deterministic signals where possible.

Return evidence that can be checked.

Keep state updates narrow until they are supported by evaluation.

Its #8 result does not claim that this is the only correct design.

It shows that a lightweight, evidence-first retrieval system can compete under a shared benchmark.

The more interesting contribution may be its path from human-readable decision records to model-facing evidence retrieval.

24. The Broader Question for Agent Memory

When an Agent says:

“I remember this.”

What should it mean?

Should it mean:

“I generated a plausible summary of the past”?

Or should it mean:

“Here is the historical evidence I found, where it came from, what changed afterward, and why this record is relevant now”?

FlowGrid argues for the second interpretation.

Memory should not become a hidden layer that quietly influences an Agent’s decisions.

It should become an evidence layer between an Agent’s current reasoning and its historical context.

That becomes increasingly important as Agents move from short conversations toward long-running projects, persistent collaboration, and decisions with real consequences.

25. From “I Remember” to “Here Is the Record”

FlowGrid’s central contribution is not a claim that every project decision can be automatically resolved.

It is a commitment to keep the record available.

What was said.

Where it came from.

What changed later.

Which state is currently adopted.

Why a current judgment should be trusted.

The first AML result showed that this evidence-retrieval foundation could be competitive.

The post-competition product direction recognizes that retrieval alone is not enough.

Long-term Agent Memory also requires state, authority, context control, and the ability for people to inspect the path from history to action.

A useful Agent Memory system may need to do more than say:

“I remember this.”

It may need to say:

“Here is the evidence.”

“Here is where it came from.”

“Here is what changed afterward.”

“And here is why this is the current record.”

Thanks to the FlowGrid Team

We would like to thank the FlowGrid team for sharing its system design, post-competition experiments, and broader product direction with the AML technical deep dive series.

Their work highlights an important dimension of Agent Memory:

Long-term memory is not only about retrieval quality. It is also about whether the historical path behind a decision remains readable, inspectable, and recoverable.

The goal of the AML Solution Spotlight series is to make top-performing memory systems easier to understand—not only through leaderboard scores, but through their technical ideas, engineering choices, trade-offs, and unresolved questions.

More technical deep dives into the first AML leaderboard are coming soon.

FlowGrid AML Retriever

Team: Artificial Idiot Research Institute

Project Lead: Lingxiao Du

GitHub: https://github.com/dlxeva/flowgrid-aml-retriever

Post-competition general product: https://github.com/dlxeva/flowgrid-agent-memory

Website: https://aizhiz.com/

X: https://x.com/dlxeva

The Second Agent Memory Challenge opens on September 20, 2026.

This cycle includes three tracks:

Textual Memory — long-horizon conversation retrieval, temporal reasoning, and user-profile alignment

Coding Memory — development-context retrieval for real software-engineering tasks

Multimodal Memory — cross-modal memory association across text, images, audio, and video

Each track includes both an Open-Source Methods Ranking and an Industry Systems Ranking.

The challenge is globally open to universities, research institutions, companies, open-source teams, and individual developers. There is no registration fee.

The open-source ranking includes a total prize pool of RMB 150,000, with awards including:

First Prize: RMB 20,000

Second Prize: RMB 8,000

Third Prize: RMB 3,000

Best Technical Innovation Award: RMB 5,000

Important dates:

Registration and submission entry opens: September 20, 2026, 00:00

Rolling evaluation period: September 20 – October 31

Submission deadline: October 31, 2026, 23:59

Evaluation queue closes: November 4, 2026, 23:59

Results planned for release: mid-November 2026

Each participant may initiate up to two full official evaluations.

Once evaluation begins, the submitted version is frozen. The platform then evaluates the system automatically through the standardized Add/Search interface.

Registration opens on September 20. We look forward to seeing what the next generation of Agent Memory systems can do.

AML
Agent Memory Leaderboard
https://agentmemoryleaderboard.ai/

Leaderboard
https://huggingface.co/spaces/agent-memory-leaderboard/leaderboard

Content Notice

From “Retrieving” to “Verifying”: How ChronoHybridMem Turns Agent Memory into Evidence

Agent Memory Leaderboard — Mon, 07 Sep 2026 06:24:52 +0000

ChronoHybridMem ranked #5 on the first AML Open Leaderboard with an Overall Score of 44.33.

But what makes its approach interesting?

Long-term memory retrieval for an AI agent is often framed as a search problem:

Given the current query, find the most relevant memories.

But finding a relevant piece of information is only part of the problem.

An agent may retrieve something that looks relevant but lacks context. A structured fact may be useful but difficult to verify. A generated summary may contain the right information but provide no clear path back to the original conversation.

This raises a deeper question:

When an Agent retrieves a memory, can it also explain where that memory came from?

ChronoHybridMem approaches this problem from a different angle.

Instead of treating memory as a collection of isolated summaries, it treats memory as a collection of verifiable evidence.

Its core idea is simple:

Agent memory should return not just relevant information, but evidence that can be traced back to its source.

In its v0.2.0 submission to the first Agent Memory Challenge, ChronoHybridMem combined raw message preservation, source-bound structured facts, SQLite FTS5 dual-path retrieval, and constrained candidate reranking.

The result was a system designed around a clear boundary:

The model can choose among existing evidence. It should not be allowed to invent the evidence itself.

After the competition, the team continued experimenting with query planning, collection-aware reranking, evidence graphs, recall-failure diagnosis, and selective gating.

Some of these ideas improved the local baseline.

Others did not.

That distinction is important because the team's post-competition work treats both positive and negative results as part of the engineering process.

1. The Core Idea: An Agent Needs Evidence, Not Just Similar Text

Consider a simple example.

Suppose a conversation history contains two statements:

“Yutu gave a bag of cookies to Tutu.”

and later:

“Yutu works at the court.”

Now the user asks:

“Where does the person who gave the cookies work?”

The answer cannot be obtained reliably by retrieving only one of these messages.

The system first needs to identify that the person who gave the cookies was Yutu.

It then needs to retrieve the information about Yutu's workplace.

More importantly, the resulting answer should be supported by evidence from the original conversation.

This illustrates three requirements for long-term Agent Memory:

Original conversations should not be replaced entirely by opaque summaries.
Retrieval should support both raw textual details and structured representations.
The model should select and rank evidence rather than freely generate new memories.

ChronoHybridMem is designed around these principles.

Instead of asking:

“What should the memory system remember?”

it focuses on another question:

“When the Agent needs to remember something, what evidence can the system provide?”

2. From Conversation to Verifiable Memory Evidence

At a high level, the ChronoHybridMem v0.2.0 pipeline looks like:

Conversation
↓
RAW + FACT Storage
↓
Dual-path FTS5 Retrieval
↓
Candidate Merge & Deduplication
↓
Constrained Reranking
↓
Evidence Set
↓
Answer

The system separates the responsibilities of different components.

During the Add stage, the original conversation is preserved.

The system can also extract structured facts from the conversation.

During Search, the query enters two retrieval paths:

raw-message retrieval;
fact-level retrieval.

The candidates are then merged and deduplicated.

A language model is used only to rank the candidates.

Finally, the system returns the selected evidence together with its source information.

This creates an explicit separation between:

retrieving evidence

and

generating an answer from evidence.

The final answer is produced through AML's standardized answer process rather than directly by ChronoHybridMem.

This separation allows the memory component itself to be evaluated independently.

3. RAW + FACT: Keep the Original Evidence, Add Structure on Top

One of the central design choices in ChronoHybridMem is that structured facts do not replace the original conversation.

When a message is written, the original message is first persisted as a RAW record.

In model mode, the system can additionally extract structured FACT representations.

For example:

RAW

“Yutu works at the court.”

FACT

“Yutu works at the court.”

The important difference is that the FACT is not treated as an independent piece of truth.

Each structured fact contains a:

source_message_id

that points back to the original message.

This creates two complementary representations.

RAW preserves context

The original message retains:

exact wording;
relationships between people;
temporal clues;
surrounding context;
details that may be lost during compression.

FACT improves retrieval

Structured facts provide a more compact representation that can be easier to search.

They can help the system locate information that may be difficult to match directly against the original wording.

The source link connects the two

Because every FACT can point back to its source message, the structured representation does not become an isolated summary.

The system can move from:

FACT → source_message_id → RAW

when verification is needed.

This is an important distinction.

Structure is used to improve retrieval, while the original record remains the source of evidence.

4. SQLite FTS5 Dual-Path Retrieval

ChronoHybridMem does not rely on a single retrieval representation.

The Search stage runs two parallel FTS5 retrieval paths.

Path 1: Raw Message Search

The system searches the original conversation records.

This path is useful when the answer depends on:

exact wording;
relationships;
contextual clues;
or details that may not survive structured extraction.

Path 2: Fact Search

The system separately searches the structured FACT records.

This path provides a more compact representation for direct fact retrieval.

The two candidate sets are then merged and deduplicated.

Conceptually:

Query

↓

Raw Message FTS5 → Raw Candidates

Fact FTS5 → Fact Candidates

↓

Merge + Deduplicate

↓

Candidate Pool

The goal is not to decide that one representation is universally better.

Instead:

The two retrieval paths compensate for each other's weaknesses.

Raw messages preserve context.

Facts provide compact retrieval targets.

Using both reduces the risk that the entire retrieval process becomes dependent on a single representation of memory.

The Search request also carries an exact user_id boundary, ensuring that retrieved records remain within the correct user's memory space.

5. Constrained Reranking: Let the Model Choose, Not Invent

After candidate retrieval, ChronoHybridMem uses a language model to rank the candidates.

But the model operates under strict constraints.

It receives:

the current query;
the retrieved candidate set.

It can then return the IDs of candidates that should be ranked higher.

It cannot:

create a new memory;
invent a candidate;
return an arbitrary ID;
access another user's records;
or replace the evidence with a generated memory.

The system subsequently validates:

whether the returned candidate ID actually exists;
whether it belongs to the current user;
whether a FACT has a valid source message;
and whether the source relationship is legitimate.

This establishes a clear division of responsibility:

The model decides which evidence is useful.

The storage and retrieval layer decides what evidence actually exists.

This distinction becomes particularly important when using LLMs inside memory systems.

A model may be excellent at reasoning over information.

But reasoning ability should not automatically give it permission to create the information it is supposed to retrieve.

ChronoHybridMem therefore treats the model as a candidate selector, rather than the source of truth.

6. Why Verifiability Matters

The purpose of this architecture is not simply to increase the number of retrieved memories.

It is to shorten the distance between:

the answer an Agent wants to produce

and

the historical evidence supporting that answer.

Return to the earlier example.

The Agent needs to answer:

“Where does the person who gave the cookies work?”

A useful evidence set might contain:

the original message identifying who gave the cookies;
the structured fact describing that person's workplace;
the source message associated with that fact.

The downstream answer model can then reason over these pieces of evidence.

If necessary, it can trace the structured fact back to the original conversation.

This makes memory more inspectable.

Instead of simply returning:

“The person works at the court.”

the system can provide the evidence chain behind that statement.

This leads to a broader design principle:

A reliable memory system should make it possible to inspect why a memory was retrieved, not merely what was retrieved.

7. Engineering Boundaries Matter Too

ChronoHybridMem's emphasis on evidence is not limited to retrieval.

The team also treats several engineering boundaries as part of memory reliability.

Idempotent writes

The Add operation is designed to be idempotent, reducing the risk of duplicate memory pollution.

Persistent storage

SQLite WAL is used to support stable persistent reads and writes.

User isolation

Both API and SQL-level operations use user_id as an isolation boundary.

Explicit failure

If required configuration for model mode is missing, the system fails explicitly rather than silently switching to unknown behavior.

These details may appear less interesting than a new retrieval algorithm.

But they matter when memory systems move from benchmark demonstrations toward actual Agent infrastructure.

A memory system needs to answer not only:

“Did I retrieve the right information?”

but also:

“Did I retrieve it from the right user?”

“Can I trace it back to its source?”

“Can I reproduce the same behavior?”

“What happens when part of the system fails?”

For ChronoHybridMem, these are part of the same reliability story.

8. Official Result vs. Post-Competition Research

An important distinction in the ChronoHybridMem team's work is between its official AML result and its later local experiments.

The result reported on the AML leaderboard corresponds to:

ChronoHybridMem v0.2.0

Rank #5

Overall Score: 44.33

The experiments described below were conducted after the official evaluation.

They primarily use the public LoCoMo dataset and a local Qwen3-4B proxy model to investigate retrieval mechanisms and engineering hypotheses.

They are not new AML leaderboard scores.

This distinction matters.

The team uses a layered evidence protocol:

official leaderboard results demonstrate performance validated by AML;
full 1,977-question local runs compare methods under the same local evaluation setup;
smaller fixed sets are used for diagnostics, mechanical checks, and upgrade gates.

Only paired comparisons using the same data, model, query plan, and call budget are used to discuss method-level improvements.

This allows the post-competition work to function as research rather than as an attempt to reinterpret the official leaderboard result.

9. After v0.2.0: Turning Memory Research into Falsifiable Experiments

The post-competition development did not simply add more models or retrieval modules.

Instead, the team framed each stage around a falsifiable question:

What failure mode are we trying to solve?

Does the proposed mechanism actually improve it?

Does it introduce additional model calls?

Can its evidence still be traced?

Can the module be safely disabled if it causes regression?

This produced a sequence of experiments from P1 to P5, preceded by two additional local milestones.

10. research-v0.3.0: Lexical + Dense Retrieval

The first post-competition milestone, research-v0.3.0, explored hybrid retrieval.

It retained the existing Add/Search API and original evidence-return contract.

During Search, the system combined:

Porter-normalized BM25 retrieval;
BAAI/bge-large-en-v1.5 dense retrieval.

The scores from the two candidate sets were normalized and fused.

The top five fused candidates were then reranked using:

answerai-colbert-small-v1

The important constraint remained unchanged:

The reranker could only reorder existing candidates.

It did not generate new evidence or replace the original messages.

On the full 1,977-question LoCoMo local evaluation, v0.3.0 achieved:

Hit@1: 0.4355
Hit@3: 0.6186
Hit@10: 0.7577
MRR: 0.5183

Compared with the corresponding lexical baseline, Hit@1 increased from 0.3359 to 0.4355.

The team also tested different reranking pool sizes.

Interestingly, increasing the ColBERT reranking pool did not automatically improve the top-ranked result.

On the fixed 200-question experiment, Top-5 performed better than Top-10 and Top-20 for the final ranking objective.

The team therefore froze the reranking pool at Top-5.

This is an example of an important engineering principle:

A larger candidate pool is not automatically a better candidate pool.

11. research-v0.4.0: Time-Aware Retrieval + Dedicated Reranking

The next milestone shifted the focus from retrieval coverage toward ranking quality.

research-v0.4.0 added a time-aware dense representation and a dedicated local Qwen reranker.

The reranker used:

Qwen3-Reranker-4B

with a yes/no relevance formulation.

The model returned a relevance probability over existing evidence rather than generating new memory.

The system also introduced a controlled temporal key, adding message date information to the retrieval representation.

An interesting result emerged:

The temporal signal alone did not improve Hit@1 on the full dataset.

However, it increased Top-10 candidate coverage.

When combined with the dedicated Qwen reranker, those additional candidates could sometimes be converted into better top-ranked evidence.

The final local result for v0.4.0 was:

Hit@1: 0.5225
Hit@3: 0.6808
Hit@10: 0.7653
MRR: 0.5856

Compared with v0.3.0:

Hit@1 increased by 0.0870
MRR increased by 0.0673

Again, these are post-competition local research results, not AML leaderboard scores.

12. P1: Structured Query Planning

After the two model-side milestones, the research shifted toward a more fundamental question:

Is retrieval failing because the system does not understand what the query actually needs?

P1 introduced structured query planning.

Instead of treating the query as a flat string, the planner decomposes it into:

intent;
core terms;
expansions;
entities;
temporal cues;
up to four evidence needs.

Different fields can then be used by different retrieval paths.

The evidence needs also become reusable signals for later experiments.

Importantly, P1 does not add another model call during Search and does not change the Add/Search API.

If the planner output is incomplete, the system falls back to a safer first-stage retrieval path.

On a fixed 200-question local screening set:

Hit@1: 0.545 → 0.565

MRR: 0.6145 → 0.6292

Hit@10 remained at 0.740.

On the full 1,977-question local evaluation:

Hit@1: 0.5761

Hit@3: 0.7157

Hit@10: 0.7618

MRR: 0.6479

The result suggested that better understanding of the information need could provide more stable gains than simply adding more retrieval models.

13. P2: More Coverage Does Not Necessarily Mean Better Ranking

P2 explored collection-aware reranking.

Multi-hop questions often require multiple complementary pieces of evidence.

A conventional ranking system, however, may place several highly similar records at the top while pushing complementary evidence lower.

P2 therefore attempted to select candidates based partly on how much additional evidence they covered.

The idea sounds intuitive:

If a candidate covers a new evidence need, shouldn't it become more valuable?

Not necessarily.

On a fixed 20-question experiment, P2 kept Hit@1 unchanged while improving Hit@3 and MRR.

But on a frozen 35-case synthetic stratification:

Hit@1 fell from 1.00 to 0.8571

and

MRR fell from 1.00 to 0.9286.

Why?

Because:

Covering more query requirements does not necessarily mean that a candidate is the best first piece of evidence.

P2 was therefore rejected from the default path.

The code remains available for reproduction and failure analysis.

This negative result changed the team's research direction.

Instead of continuing to optimize diversity in the top-ranked candidates, the team began asking:

Where exactly is the correct evidence being lost?

14. P3: Evidence Graphs Need Evidence Too

P3 explored another attractive idea for memory systems:

Can an evidence graph help connect people, places, organizations, relationships, and temporal updates?

The team constructed graph representations for entities and relations.

But it imposed a strict rule:

Every entity mention and every relationship edge must be independently supported by an original message.

The reason is straightforward.

A graph generated by an LLM can look highly structured while still containing unsupported relationships.

For a memory system centered on verifiability, that would simply move the trust problem somewhere else.

Under the strict evidence constraint, the graph became surprisingly sparse.

On a fixed set of 419 original messages, the strict relation graph produced only 3 independently witnessed relationship edges.

A second experiment, P3-B1, used source-local entity mention anchors.

This achieved coverage across:

363 / 419 messages (86.63%)

But on the fixed 20-question test:

Hit@1 fell from 0.40 to 0.35

and

Hit@10 fell from 0.55 to 0.50.

P3 therefore did not become a global default retrieval channel.

The experiment left an important lesson:

A memory structure should not be considered useful simply because it is structured, interpretable, or highly covered.

Its value still has to be demonstrated through paired retrieval gains and source-level auditing.

15. P4: Diagnose the Recall Failure Before Fixing It

P4 was arguably the most important shift in the post-competition research.

Instead of asking:

“What new retrieval module should we add?”

the team first asked:

“Why are we failing to retrieve the correct evidence?”

A fixed 100-question audit identified 30 Top-10 retrieval failures.

They were divided into three categories:

Fusion miss

20 cases

The relevant evidence had been found by at least one retrieval channel but was lost during candidate fusion.

Channel miss

8 cases

None of the lexical retrieval channels found the relevant evidence.

Reranker drop

2 cases

The correct evidence was retrieved but subsequently ranked too low.

This distribution was revealing.

The dominant problem was not:

“The reranker cannot recognize the correct candidate.”

It was:

“The correct candidate often never survives into the reranking pool.”

This changed the optimization target.

16. P4-A: Turn Evidence Needs into Retrieval Channels

To address the largest failure bucket — fusion misses — P4-A reused the evidence needs already generated by P1.

Each evidence need became an independent, bounded retrieval channel.

Each channel was given a fixed number of candidate positions.

These candidates were then merged with the existing retrieval results and passed into the original reranking pipeline.

Crucially:

no new evidence was generated;
the existing query planner was reused;
Search did not require an additional model call.

The goal was simple:

Give important evidence needs a guaranteed opportunity to enter the candidate pool.

On the full 1,977-question local evaluation, P4-A q2 became the strongest post-competition local proxy baseline.

Compared with P1, it recovered 8 questions whose relevant evidence had previously fallen outside Top-10.

Four of those recovered cases moved directly into Top-1.

Hit@1, Hit@3, Hit@10, and MRR all improved modestly.

Again, this is a local proxy result, not a new official AML score.

But the experiment provided a useful diagnosis:

Improving recall before reranking can matter more than making the reranker itself more sophisticated.

17. P5: When Should the System Override Its Top Result?

P5 explored a different problem.

P4-A showed that candidate replacement could rescue some queries.

But the same mechanism could also replace an already-correct Top-1 result.

The team therefore asked:

Can we identify the cases where a candidate swap is actually beneficial?

Several signals were tested:

channel count;
query-token overlap;
temporal/correction strata;
model-reported confidence.

None passed the predefined fixed-200-question threshold.

For example:

channel count: Hit@1 change -0.005
query overlap: -0.035
strata narrowing: -0.010
confidence gating: did not trigger reliably

More importantly, asking the local model to provide an explicit confidence signal itself reduced Hit@1 by 0.045 in the tested setup.

The conclusion was therefore not that selective gating is impossible.

Rather:

These simple signals are not reliable enough to determine when an evidence swap should occur.

P5 was kept as default-off ablation code, and the team stopped tuning this direction.

Again, a negative result became useful evidence.

18. What These Experiments Suggest About Agent Memory

Taken together, the ChronoHybridMem experiments suggest several broader observations.

1. Query understanding can matter more than retrieval complexity

P1 produced stable gains by making the information need more explicit.

Simply adding another retrieval mechanism is not guaranteed to produce the same effect.

2. More structure does not automatically mean better memory

P2 and P3 both explored more structured candidate selection.

Both demonstrated that additional structure can introduce new failure modes.

3. Recall and ranking are different problems

P4's failure audit showed that many apparent “ranking failures” were actually retrieval failures.

If the correct evidence never enters the candidate pool, a better reranker cannot recover it.

4. Negative results can improve system design

P2, P3, and P5 did not enter the default path.

That is not necessarily wasted work.

By explicitly measuring their failures, the team narrowed the space of plausible design choices.

This is perhaps one of the most interesting aspects of the project:

The system is being developed not by accumulating modules, but by eliminating unsupported assumptions.

19. The Next Challenge: When Lexical Retrieval Cannot Find the Connection

The current architecture works well when useful evidence can be brought into the candidate pool.

But an important class of problems remains.

Consider a user saying:

“I'm thinking about adopting a cat.”

Months earlier, they had said:

“I have a lot of lilies at home.”

The historical statement may be highly relevant to the current decision.

But the connection is not obvious from the query itself.

A retrieval system searching for:

cat;
adoption;
pet;
breed;
food;

may never search for:

lilies

The problem is no longer simply:

“Can the system rank the correct evidence?”

It becomes:

“Can the system discover that this seemingly unrelated memory matters?”

This is a harder problem for query-conditioned retrieval.

If the relevant memory never enters the candidate pool, even a powerful reranker cannot recover it.

This is one reason the ChronoHybridMem team is now investigating channel-miss cases more closely.

20. Toward Source-Constrained Multi-Hop Retrieval

The next stage of the team's research focuses on cases involving:

abstract relationships;
identity;
personality;
decisions;
multi-hop reasoning;
and implicit connections.

The goal is to explore source-constrained bridging retrieval.

Instead of allowing an LLM to freely invent relationships, the system could:

locate relevant entities or messages;
identify source-supported relationships;
perform a limited expansion;
retrieve additional evidence from the same user, session, or explicitly witnessed relationship.

The expansion would remain tightly bounded.

For example:

fixed candidate limits;
limited traversal depth;
deterministic tie-breaking;
strict user isolation;
no model-generated relationship treated as ground truth.

The principle remains unchanged:

Expand the search space without expanding the set of unsupported facts.

21. Temporal State and Corrections

Another important direction is temporal memory.

Long conversations are not static.

Users change their plans.

They correct names.

They update schedules.

They revise preferences.

They add exceptions.

Consider:

“I'm moving to Shanghai next month.”

followed later by:

“Actually, the move has been postponed.”

A memory system should not simply retrieve both statements and leave the downstream model to guess.

The next research direction is therefore to represent state changes as an auditable chain:

Original Statement
↓
Update / Correction
↓
Currently Valid State

But the original evidence should remain available.

The goal is not to delete old memories.

It is to make the current state explainable:

Which earlier information changed?

What evidence caused the update?

Why is this the currently valid state?

This is another extension of the same principle behind RAW + FACT:

Interpretation can evolve, but evidence should remain traceable.

22. Toward a More Complete Memory Evaluation Framework

ChronoHybridMem's research also points to a broader question for Agent Memory evaluation.

Memory quality should not be measured only through:

Hit@K;
MRR;
answer accuracy.

Future evaluations may also need to track:

evidence recall;
candidate-source distribution;
model call count;
fallback behavior;
user isolation;
retrieval stability;
database state;
runtime cost;
and source completeness.

A memory system that achieves higher recall by dramatically increasing inference cost may represent a different engineering trade-off from one that achieves similar performance with a smaller budget.

Likewise, a system that retrieves an answer but cannot identify its source presents a different reliability profile from one that returns a fully traceable evidence chain.

This suggests a broader direction for memory evaluation:

Measure not only whether the Agent remembers, but how it remembers, what it costs, and whether the memory can be verified.

23. From Memory Retrieval to Evidence Systems

ChronoHybridMem began with a relatively simple architectural choice:

Keep the original conversation.

Add structure where useful.

Retrieve through multiple paths.

Let the model rank existing candidates.

Return evidence with its source.

The subsequent P1–P5 experiments made the picture more nuanced.

They showed that:

structured query planning can provide meaningful gains;
more candidate diversity does not automatically improve ranking;
graph structure requires strict evidence constraints;
recall failures can dominate reranking failures;
and simple confidence-based gating is not necessarily reliable.

Together, these results suggest that Agent Memory may be moving toward something broader than conventional retrieval.

A memory system is not simply a database.

It is not simply a vector store.

It is not simply a summarization layer.

It is increasingly becoming an evidence system between an Agent and its history.

The central question becomes:

When an Agent remembers something, can we understand where that memory came from and why it should be trusted?

24. What ChronoHybridMem Adds to the AML Landscape

The first AML leaderboard contains systems that make different architectural choices.

Some emphasize structured memory.

Some emphasize retrieval.

Some rely heavily on learned representations.

ChronoHybridMem represents another point in this design space:

Preserve evidence first, then add controlled structure and retrieval mechanisms around it.

Its Rank #5 result — 44.33 Overall — demonstrates that this evidence-oriented architecture is competitive under the first AML evaluation.

But the more interesting contribution may be the research process that followed.

Rather than assuming that every additional module improves memory, the team explicitly tested hypotheses and removed those that failed.

This makes the project useful not only as a leaderboard entry, but also as a case study in how Agent Memory systems can be iterated.

25. The Broader Question for Agent Memory

ChronoHybridMem ultimately asks a simple question:

When an Agent remembers something, should it be able to show its work?

A useful memory system may need to do more than return:

“I remember this.”

It may need to provide:

“Here is what I found.”

“Here is where it came from.”

“Here is why this evidence was selected.”

“And here is the original record if you want to verify it.”

This changes the role of memory.

Instead of treating memory as a hidden layer that produces an opaque answer, we can treat it as an evidence layer that connects an Agent's current reasoning to its historical context.

That may become increasingly important as Agents move from short-lived interactions toward long-term relationships, persistent tasks, and autonomous decision-making.

Thanks to the ChronoHybridMem Team

We'd like to thank the ChronoHybridMem team for sharing their system design and post-competition research with the AML technical deep dive series.

Their work illustrates an important aspect of Agent Memory research:

Progress does not always mean adding another module. Sometimes it means finding out which modules should not be there.

The goal of the AML solution spotlight series is to make top-performing memory systems easier to understand — not only through leaderboard scores, but through the technical ideas, engineering choices, trade-offs, and failures behind those scores.

More technical deep dives into the first AML leaderboard are coming soon.

ChronoHybridMem

Team: ChronoHybridMem
Team Lead: Haoxuan Meng
GitHub: https://github.com/Tin11Mn/chrono-hybrid-mem

AML

Agent Memory Leaderboard
https://agentmemoryleaderboard.ai/

Leaderboard:
https://huggingface.co/spaces/agent-memory-leaderboard/leaderboard

From “Managing” to “Preserving”: How ActiveMemoryIndex Keeps Memory Simple

Agent Memory Leaderboard — Wed, 26 Aug 2026 03:43:25 +0000

How ActiveMemoryIndex Ranks #3 Without Memory Governance

AI memory systems often try to make memory smarter.

They summarize conversations, extract facts, resolve conflicts, update old values, build hierarchies, and decide which memories should remain active.

But what if a memory system did almost none of that?

ActiveMemoryIndex takes a surprisingly different approach: preserve more, process less, and let the answer model decide.

In the first AML Open Leaderboard, ActiveMemoryIndex ranked #3 on the Open-Source Text track with a score of 44.84.

More interestingly, it ranked #1 on “new value overrides and current state” (54.50) and #1 on “contradiction detection and conflict resolution” (24.79) — despite having no explicit memory governance mechanism.

So how does it work?

1. A Memory System That Doesn't “Manage” Memory

ActiveMemoryIndex has only two storage operations:

CREATE and INSERT OR REPLACE.

There is no UPDATE.

There is no DELETE.

It does not merge memories, invalidate old memories, detect contradictions, or explicitly resolve conflicts.

Every stored record remains available.

Instead of deciding during the write stage which information is still valid, the system preserves the original evidence — including timestamps — and lets the reading model determine what matters.

This leads to a simple principle:

Don't decide too early what the agent will need later.

The approach trades sophisticated memory governance for information preservation.

2. Two Copies: Raw Messages + Atomic Facts

For every Add request, ActiveMemoryIndex stores two versions of the information.

Raw conversation

The original message is preserved verbatim, with:

timestamp
speaker role
original wording

Extracted facts

A second representation is generated by GPT-4o-mini as first-person atomic facts.

Both are embedded and stored in the same SQLite table.

The raw version preserves information fidelity.

The fact version provides a more compact representation that can improve retrieval.

But the system does not assume that the extracted version is always better.

In fact, its own ablation experiments suggest that extraction contributes only modestly.

This reflects a broader design choice:

An imperfect summary should never become the only copy of the memory.

3. Retrieval: Make the Query Speak the Same Language as the Memory

There is another interesting problem.

The stored conversations are written from the user's perspective:

“I want to move to London.”

But the benchmark question might ask:

“Where does John want to move?”

These two expressions may refer to the same information while living in different linguistic frames.

ActiveMemoryIndex therefore rewrites the query into a first-person memory-oriented question.

For example:

“Did I mention where John wanted to move?”

The rewritten query is then combined with the original query for retrieval.

Conceptually:

Original Query
↓
Memory-oriented Query Rewrite
↓
Embedding Retrieval
↓
Combined Ranking

The authors note that this is closely related to HyDE, except that the generated representation is a memory-oriented question rather than a hypothetical answer.

4. Why Return Raw Evidence First?

After retrieval, ActiveMemoryIndex returns up to 100 pieces of evidence.

But the ordering matters.

The system places:

Raw messages first → Extracted facts second

The collection itself is not changed.

Only the ordering changes.

Why?

Because the raw message has not gone through another lossy transformation.

Consider two retrieved pieces:

“I decided to move to London because my company is opening a new office there.”

and:

“John plans to move to London.”

The second is easier to process, but the first contains the actual evidence and context.

If the answer model only receives a compressed representation, information that seemed irrelevant during extraction may already be gone.

So ActiveMemoryIndex takes a conservative approach:

Preserve the evidence. Let the model interpret it later.

5. The Counterintuitive Result: Governance Isn't Always Better

This is perhaps the most interesting finding from the system.

Memory governance sounds inherently useful.

A system could:

detect contradictions;
downgrade stale information;
delete outdated memories;
merge related facts;
maintain a “current state.”

But ActiveMemoryIndex does none of these.

And yet it ranked #1 on the AML state-update and conflict-resolution dimensions.

The authors propose a possible explanation:

Governance only helps if it actually improves what reaches the answer model.

For example, a system might detect that an old value conflicts with a new value and reduce its ranking.

But the old value may still enter the model's context.

The model still has to decide what is current.

By contrast, preserving complete, timestamped evidence may give the reading model enough information to make the decision itself.

This is an important distinction:

Changing the ranking of evidence is not necessarily the same as resolving the conflict.

The authors present this as a hypothesis rather than a universal conclusion.

6. A Surprising Lesson About Context Size

The ActiveMemoryIndex experiments also reveal an interesting benchmark-level issue.

The authors tested returning a larger “parent block” around retrieved messages.

It initially appeared to improve performance significantly.

But when the comparison was controlled for the amount of text returned, the advantage largely disappeared.

In other words:

More context can look like better memory simply because the model was allowed to read more.

This exposes a potential limitation of evaluation setups based primarily on the number of retrieved items.

If one system returns 100 short messages while another returns 100 large blocks, the two systems are technically respecting the same item limit but giving the answer model very different amounts of information.

The authors therefore suggest that future benchmarks could consider reporting:

tokens/query + accuracy

rather than relying only on a fixed number of retrieved items.

This is an important question for memory evaluation more broadly:

Are we measuring memory quality, or how much context the system is allowed to spend?

7. What ActiveMemoryIndex Doesn't Do

The simplicity of the architecture is also its limitation.

ActiveMemoryIndex does not currently use:

knowledge graphs
hierarchical summaries
entity disambiguation
explicit memory governance
sophisticated state management

Instead, it relies heavily on:

preserving raw evidence + retrieval + answer-model reasoning

This makes the approach lightweight, but it also leaves some difficult problems unresolved.

The weakest capability in the current submission was temporal reasoning, where the system scored 18.35.

The authors identify this as a major area for future work.

8. What's Next: From Retrieval to Temporal Memory

The next direction is not simply “more retrieval.”

The team is exploring how to represent temporal information more explicitly.

For example:

“I moved last week.”

A memory system should ideally convert that relative expression into an anchored date.

Similarly, if a user's state changes over time:

“I live in Beijing.”

followed later by:

“I moved to Shanghai.”

the system should understand the relationship between the two states rather than treating them as two unrelated facts.

This suggests a possible next step:

Raw evidence → temporal anchoring → versioned facts → better state reasoning

Interestingly, this moves ActiveMemoryIndex closer to the memory-governance layer it deliberately avoided in its current version.

9. What Can We Learn From ActiveMemoryIndex?

ActiveMemoryIndex offers a different perspective on Agent Memory.

Instead of asking:

How should we organize and govern memory?

it asks:

How much information should we preserve and let the model reason over later?

Its current results suggest that a lightweight system can remain highly competitive when it:

preserves original evidence;
avoids premature information loss;
improves retrieval through query rewriting;
gives the answer model enough context to reason about state and conflicts.

But the approach also exposes an important boundary.

Preserving everything does not automatically mean understanding everything.

As memory tasks become more temporal, stateful, and long-running, systems may eventually need to decide what should be remembered, what should expire, and how different versions of the same fact relate to one another.

That makes the trade-off between preservation and governance an interesting direction for future Agent Memory research.

Conclusion

ActiveMemoryIndex shows that a memory system does not necessarily need a complicated memory-management layer to be competitive.

Sometimes, the better strategy may be surprisingly simple:

Keep the evidence. Retrieve it well. Let the model reason.

Its #3 ranking on the first AML Open Leaderboard makes this approach particularly interesting — not because it proves that “less governance is better,” but because it raises a deeper question:

When should a memory system make decisions for the model, and when should it preserve information and let the model decide?

That is one of the trade-offs we hope to explore through the AML technical deep dive series.

Learn More

ActiveMemoryIndex GitHub:
https://github.com/linxuhao/ActiveMemoryIndex

Underlying research:
An Index, Not a Store: The Model Does Remember — It Just Needs Its Notebook

HyDE:
https://arxiv.org/abs/2212.10496

About AML

The Agent Memory Leaderboard (AML) is an open evaluation platform for AI Agent Memory systems.

Through technical deep dives like this one, we aim to look beyond leaderboard scores and understand the architectures, engineering choices, trade-offs, and limitations behind different memory systems.

One Benchmark. Real Memory.

What If AI Agents Didn’t Need Memory? They Could Just Search Their Past

Agent Memory Leaderboard — Fri, 21 Aug 2026 06:56:06 +0000

From Remembering to Searching: How ReFind Challenges AI Agent Memory

Everyone is building AI memory systems.

But a fundamental question remains:

How should an AI agent actually remember?

As agents move from simple conversations to long-term collaboration, memory becomes a critical capability.

However, building memory is not just about storing more information.

The harder problem is:

When an agent needs something from the past, how can it find the right information?

Many memory systems try to solve this by creating structured memories in advance:

summarizing conversations;
extracting facts;
building knowledge structures;
maintaining memory entries.

But this creates a difficult trade-off:

Before knowing what the user will ask in the future, how can a system know what information will matter?

This leads to another possible direction:

Instead of trying to remember everything beforehand, what if an agent could search its own history when needed?

This is the idea behind ReFind.

In the first Agent Memory Leaderboard (AML) Open Leaderboard, ReFind achieved a score of 44.97, ranking #2 among open-source memory systems.

Rather than pre-building complex memory representations, ReFind keeps raw conversation history and lets agents actively search for relevant evidence.

The Core Idea: Let Agents Search Their Own History

Humans do not perfectly remember every conversation.

When we forget something from an old chat, we usually do not reconstruct the entire conversation from memory.

We search.

We open the conversation.

We try keywords.

We check surrounding messages.

If necessary, we search again.

ReFind takes inspiration from this behavior.

Instead of deciding during the write stage what information will be important, ReFind keeps the original conversation records and allows the agent to explore them when a query arrives.

The goal changes from:

“Build the perfect memory beforehand.”

to:

“Find the right evidence when it is needed.”

How ReFind Works

ReFind combines several mechanisms to simulate human-like search behavior:

1. Multi-turn Search

Traditional retrieval often follows:

Query → Retrieve → Answer

But real information search is usually iterative.

A user may search one keyword, inspect results, refine the query, and search again.

ReFind allows the retrieval agent to decide:

what to search;
whether more searching is needed;
which evidence should be kept.

This turns retrieval from a one-shot operation into an exploration process.

2. Context Browsing

A retrieved message is not always enough.

A single sentence may depend on surrounding conversation:

previous decisions;
explanations;
conditions;
follow-up results.

For example:

“Let’s use the previous approach.”

Without context, the sentence has little meaning.

ReFind therefore allows the agent to inspect surrounding messages instead of treating retrieved memories as isolated fragments.

The key idea:

Useful memory is not always one matching sentence. Sometimes it is the context around it.

3. Moving Computation from Write Time to Query Time

One of ReFind's most important design choices is where computation happens.

Traditional memory systems often process information when it is created:

Conversation
↓
Extract / Summarize
↓
Structured Memory
↓
Retrieve

ReFind takes another approach:

Conversation
↓
Keep Raw Records
↓
Search When Needed
↓
Retrieve Evidence

This creates a clear trade-off.

Pre-built memory can make future retrieval faster, but requires the system to decide early what information should be preserved.

ReFind delays this decision.

Instead of paying the cost for every conversation, it spends more computation only when a query actually requires deeper search.

The Limitation: Search Is Not Always Enough

ReFind shows that active search can be a powerful memory strategy.

But it also reveals a deeper challenge.

A search system still needs a connection between the current query and the memory it should retrieve.

Consider:

Current question:

“I want to adopt a cat.”

Historical memory:

“I have many lilies at home.”

These two statements may have little semantic similarity.

A retrieval system may search for:

cats;
pets;
adoption.

But it may never find the information about lilies.

However, that memory could be critical because lilies can be dangerous for cats.

The problem is not reasoning.

The problem is:

The agent never retrieved the memory it needed.

Beyond Retrieval: The Future of Agent Memory

This challenge is explored by the ReFind team's additional benchmark, InMind.

ReFind and InMind represent two different questions:

ReFind:

How can agents search their history better?

InMind:

What if the important memory cannot be found through similarity alone?

Together, they highlight a broader challenge for Agent Memory:

Future systems may need not only better retrieval, but also better understanding of:

when a past event matters;
which memories should be recalled;
and how hidden connections between memories should be discovered.

Conclusion

ReFind starts from a simple idea:

Agents may not need to remember everything. They need the ability to find what matters.

By preserving raw conversation history and enabling active search, ReFind provides an alternative direction for building AI memory systems.

The first AML results show that search-based memory can be highly competitive.

At the same time, they raise a bigger question:

Should agents only retrieve memories when asked — or should they also know when something from the past matters?

Thanks to the ReFind Team

We would like to thank the ReFind team for sharing their approach and contributing to the AML Technical Deep Dive series.

The goal of this series is to make different memory systems easier to understand, compare, and learn from — not only through leaderboard scores, but also through the ideas and engineering choices behind them.

This is the second technical deep dive in the AML series. More breakdowns of top-performing memory systems are coming soon.

ReFind GitHub Repository
https://github.com/imlrz/ReFind

InMind Benchmark
Keep It InMind: Benchmarking the Implicit-Association Blind Spot in Agent Memory
https://arxiv.org/abs/2607.24368

InMind GitHub Repository
https://github.com/imlrz/InMind

Agent Memory Leaderboard (AML)
https://huggingface.co/spaces/agent-memory-leaderboard/leaderboard

AML Technical Deep Dive #1: From “Similarity” to “Completeness” — How InvMem Retrieves Useful Long-Term Memory

Agent Memory Leaderboard — Thu, 20 Aug 2026 03:33:05 +0000

Retrieving memory for an AI agent sounds simple:

Given the current query, find the most relevant memories.

In practice, however, semantic similarity is not always enough.

A retrieved memory may be highly similar to the query while still missing the surrounding context needed for the agent to make the right decision.

This is the problem InvMem approaches from a different angle:

Memory retrieval should not only find similar information. It should recover enough relevant context to reconstruct a useful memory.

1. Fine-grained Chunking: Breaking Memory into Smaller Units

One part of InvMem's approach is fine-grained chunking.

Instead of treating a long piece of historical information as one indivisible memory unit, InvMem breaks it down into smaller pieces.

This gives the retrieval system more flexibility:

different parts of the same historical interaction can be retrieved independently;
more precise information can be matched against the current query;
irrelevant parts of a long memory are less likely to dominate retrieval.

But finer-grained retrieval also creates a new problem:

What if the most relevant chunk does not contain enough context on its own?

This leads to the next part of the design.

2. Hybrid Retrieval: Dense Retrieval + BM25

InvMem combines dense retrieval with BM25.

These two retrieval approaches capture different types of relevance.

Dense retrieval is useful for semantic similarity. It can identify memories that express similar concepts even when the wording is different.

BM25, on the other hand, is particularly useful for lexical matching and exact terms.

For memory retrieval, combining the two provides a broader retrieval signal than relying on either method alone.

The goal is not simply to find:

“Which memory looks most similar?”

but rather:

“Which memories are most likely to contain useful evidence for the current query?”

3. Weighted RRF: Combining Different Retrieval Signals

After obtaining candidates from different retrieval methods, InvMem uses Weighted Reciprocal Rank Fusion (Weighted RRF) to combine their rankings.

This allows the system to integrate signals from dense retrieval and BM25 rather than choosing one retrieval method over the other.

Conceptually, the pipeline becomes:

Query → Dense Retrieval
Query → BM25
↓
Weighted RRF
↓
Unified candidate ranking

This matters because memory relevance is rarely one-dimensional.

A memory can be semantically relevant while lacking an important keyword, or contain an exact keyword while being less relevant to the overall meaning of the query.

Combining the two signals helps balance these cases.

4. Same-session Adjacency Expansion: Recovering Context

This is one of the more interesting parts of InvMem's approach.

A highly relevant memory chunk does not necessarily contain the entire context needed by the agent.

Historical conversations often contain information that is distributed across neighboring turns.

For example:

Turn A: The user describes a problem.
Turn B: The agent proposes a solution.
Turn C: The user confirms what worked.

If retrieval only returns Turn B, the agent may know the solution but not fully understand why it was proposed or whether it was actually validated.

InvMem therefore performs same-session adjacency expansion after retrieval.

The retrieved chunk can bring in neighboring information from the same session, helping reconstruct a more complete context.

This reflects an important distinction:

The most similar chunk is not necessarily the most useful memory.

Sometimes the useful memory is the retrieved chunk plus the context surrounding it.

5. From Similarity to Completeness

Putting these components together, InvMem's retrieval pipeline can be understood as:

Fine-grained Chunking
↓
Dense Retrieval + BM25
↓
Weighted RRF
↓
Same-session Adjacency Expansion
↓
More complete memory context

The key idea is not to maximize the amount of memory retrieved.

It is to improve the chance that the retrieved information contains the complete evidence needed by the agent.

This leads to a broader question for AI memory systems:

Should memory retrieval be optimized for similarity, or for usefulness?

InvMem's approach suggests that these two objectives are not always the same.

Why This Matters for Agent Memory

Long-term memory is becoming an increasingly important component of agent systems.

But as memory systems become more sophisticated, the retrieval problem becomes more than a standard vector-search problem.

An agent may need:

a specific fact;
a previous decision;
the reasoning behind that decision;
evidence that a solution worked;
or the surrounding context needed to interpret a retrieved memory correctly.

This makes memory completeness an important dimension to consider alongside retrieval relevance.

The InvMem result is therefore interesting not only because it achieved Rank #1 with 45.06 on the first AML Open Leaderboard, but also because it illustrates one possible direction for moving beyond simple similarity-based retrieval.

Thanks to the InvMem Team

We’d like to thank the InvMem team for sharing their approach and contributing to the AML technical deep dive series.

The goal of this series is to make different memory systems easier to understand, compare, and learn from — not just through leaderboard scores, but through the ideas and engineering decisions behind them.

This is the first deep dive in the series. More technical breakdowns are coming soon.

Beyond Retrieval: What We Learned From the First Agent Memory Leaderboard

Agent Memory Leaderboard — Mon, 17 Aug 2026 06:40:15 +0000

AI memory is becoming an increasingly important part of agent infrastructure.

But there is still a basic problem:

How should we actually measure memory?

It is easy to demonstrate that an agent can remember something.

It is much harder to determine whether one memory architecture is genuinely better than another.

Different memory systems can use different datasets, models, prompts, retrieval strategies, and evaluation pipelines.

That makes many existing comparisons difficult to interpret.

The first season of the** Agent Memory Leaderboard (AML)** was designed around this problem.

After evaluating 69 representative memory systems from 136 registered teams, we want to share some of the technical observations from the first season.

1. The Problem With Evaluating Memory

Consider a simple agent:

Conversation History
↓
Memory System
↓
Retrieved Memories
↓
Answer Model
↓
Evaluation

If the agent gives the correct answer, which component deserves the credit?

The memory system may have retrieved excellent evidence.

But the answer model may also be stronger.

The prompt may be better.

The judge may use a different rubric.

Even the underlying dataset may favor one architecture over another.

This creates a fundamental attribution problem.

A memory benchmark therefore needs to control as many variables as possible outside the memory layer.

2. Separating Memory From the Rest of the Agent

AML uses a clearer system boundary.

The participating memory system is responsible for:

Add → Search

The benchmark platform handles:

Answer → Eval

The idea is straightforward.

If different systems are evaluated with the same downstream answering and evaluation pipeline, differences in the final results can be more meaningfully attributed to the memory layer.

This does not make every source of bias disappear.

A benchmark still needs to consider dataset composition, judge calibration, private test sets, system versions, and other factors.

But establishing a clear evaluation contract is an important starting point.

3. Why Retrieval Alone Is Not Enough

One of the most important lessons from the first season is that AI memory is not simply a retrieval problem.

Imagine an agent remembers:

“The project uses PostgreSQL.”

Three months later, the architecture changes to:

“The project migrated to MySQL.”

A useful memory system needs more than semantic similarity.

It needs to understand that the newer information changes the state of the previous memory.

The same problem appears with:

changing user preferences
outdated instructions
conflicting facts
evolving project requirements
previous decisions and rejected alternatives

This is why future memory systems will need stronger mechanisms for:

formation → organization → retrieval → updating → forgetting

Memory management becomes part of the intelligence of the agent.

4. Measuring Multiple Memory Capabilities

AML evaluates memory across multiple dimensions rather than reducing everything to retrieval.

The first evaluation includes dimensions such as:

Fact Recall

Can the system recover information that appeared previously?

Compositional Reasoning

Can the agent connect multiple pieces of historical information?

Temporal Reasoning

Can the system understand changes and relationships across time?

Personalization

Can it maintain an accurate representation of a user's preferences and characteristics?

Memory Governance

Can the system manage memory appropriately?

Rule & Workflow Execution

Can historical information help the agent follow established rules and workflows?

Safety & Privacy

Can memory remain useful without introducing unwanted information or interference?

These dimensions matter because memory systems can have very different capability profiles.

5. What Did the First Leaderboard Show?
Commercial Products

The first Commercial Products — Text Memory ranking was:

Rank System Score
1 MemoraX 58.02
2 MemOS 45.89
3 NTES-MEMORY-SMART 44.21

The gap between the top systems illustrates one of the benefits of having a unified evaluation environment.

Rather than comparing isolated benchmark claims from different projects, the systems are evaluated under the same framework.

But the overall score is only part of the story.

A system's capability profile can reveal much more about its underlying strengths and weaknesses.

6. Open-source Methods Show a Different Pattern

The open-source ranking was considerably tighter:

Rank System Score
1 InvMem 45.10
2 ReFind 45.00
3 ActiveMemoryIndex 44.80

The difference between the top three is relatively small.

This suggests that there is no single dominant approach to memory yet.

Different systems are exploring different combinations of retrieval, indexing, reasoning, memory organization, and query processing.

That diversity is valuable for a young research area.

A benchmark should not only identify winners.

It should help researchers understand which approaches work, under which capabilities, and where the remaining gaps are.

7. Long Context Does Not Equal Long-Term Memory

The rapid expansion of context windows raises another important question.

If a model can process millions of tokens, why do we need a separate memory layer?

Because context and memory solve different problems.

A context window answers:

What information can the model access right now?

Memory asks:

What information should the agent retain and reuse over time?

For a long-running agent, simply keeping more information in context is not necessarily a scalable solution.

The system still needs to determine:

what is important,
what is outdated,
what is relevant to the current task,
and what should influence future behavior.

The memory layer therefore becomes a mechanism for managing persistent state rather than simply extending the prompt.

8. The Next Challenge: Memory for Long-Running Agents

The hardest memory problems are likely to appear when agents operate continuously.

Consider a coding agent working on the same repository for weeks.

It may need to remember:

architectural decisions
coding conventions
previous debugging attempts
dependencies
user preferences
rejected approaches
known failure modes

Without persistent memory, the agent repeatedly rediscovers this information.

With poor memory, it may retrieve outdated or contradictory information.

This makes coding agents an especially interesting environment for studying long-term memory.

The problem becomes:

How can an agent accumulate useful experience without accumulating noise?

9. What a Future Memory Benchmark Should Measure

The first season also raises questions for future evaluation.

A more complete benchmark may need to examine:

Memory quality

Does the system retrieve the right information?

Memory evolution

Can it update outdated information?

Conflict resolution

What happens when memories contradict each other?

Abstention

Can the system recognize when something is not known?

Efficiency

What are the storage, latency, and inference costs?

Long-horizon stability

Does performance degrade as memory accumulates?

Version consistency

Does a commercial memory API behave consistently as the underlying system evolves?

These are difficult problems, but they are increasingly important as memory moves from research prototypes into production agents.

10. The First Season Is Only a Starting Point

The goal of AML is not to declare a permanent winner.

AI memory is still evolving rapidly.

The first season gives us a common starting point for comparing systems and identifying where different approaches succeed or struggle.

Future seasons will need to become more comprehensive as agents become more capable and more autonomous.

We hope AML can provide an open environment where researchers and developers can:

evaluate new memory architectures
compare different approaches
reproduce results
identify weaknesses
and build better long-term agents

The field needs more systems, more experiments, and better evaluation.

The first leaderboard is only the beginning.

Explore the First AML Season

Leaderboard:
Agent Memory Leaderboard

GitHub:
AML GitHub

Hugging Face:
AML on Hugging Face

Building a Fair Benchmark for AI Agent Memory Systems

Agent Memory Leaderboard — Thu, 13 Aug 2026 02:51:12 +0000

Everyone is building AI memory systems.

But how do we know which ones actually work?

As AI agents move from one-off interactions toward long-term collaboration, memory is becoming a core capability. Yet evaluating memory systems fairly is surprisingly difficult.

Different systems often use different datasets, answer models, prompts, and evaluation methods. When the final score changes, it can be hard to tell whether the difference comes from the memory system itself or from the evaluation setup.

That's why we built Agent Memory Leaderboard (AML).

Why Do We Need a Memory Benchmark?

Memory is more than storing conversation history.

A useful memory system needs to retrieve relevant information, connect information across time, handle changing states, and provide useful context for an agent's current task.

But there hasn't been a common evaluation environment where different memory approaches can be compared under the same conditions.

AML was created to provide that common ground.

The first edition was jointly initiated by nearly 30 universities and research institutions and covers two evaluation tracks:

Open-source Methods — Text Memory
Commercial Products — Text Memory

As of August 12, 2026:

136 teams registered for the competition
67 representative memory frameworks successfully completed the first evaluation
The AML website surpassed 200,000 clicks
The website passed 100,000 clicks within the first 10 days

The first leaderboard results are now live.

Making Memory Systems More Comparable

One of the main challenges we wanted to address was evaluation consistency.

In a typical setup, a memory system may be evaluated together with a particular answer model, prompt, or judge.

That makes direct comparison difficult.

A higher score could come from a better memory system — but it could also come from a stronger downstream model or a different evaluation setup.

AML tries to separate these components.

The core interface for participating memory systems is:

Memory System
Add → Search

The memory system receives long-term history through Add, and returns relevant memories through Search.

Then AML handles:

AML Evaluation Platform
Answer → Eval

Answer generation and evaluation are completed by the benchmark platform under the same evaluation process.

This helps reduce the impact of different answer models, prompts, judges, and scoring conventions.

The goal is simple:

Compare memory systems under the same conditions as much as possible.

Memory Is More Than Retrieval

A memory system shouldn't be judged only by whether it can retrieve something that looks similar. Memory quality is not only about retrieving similar information, but about understanding relevance, context, time, and task requirements.

For text memory, AML evaluates multiple capabilities, including:

Factual recall
Relational and multi-hop reasoning
Temporal and event understanding
Memory governance
Personalization and care
Rules and workflow execution
Epistemic safety and privacy

This matters because real-world agent memory is not just a search problem.

An agent may need to understand:

What happened before?
When did it happen?
What changed afterward?
Which previous experience is relevant now?
Which information should be trusted?
How should that memory affect the current task?

A useful memory system needs to handle these questions together.

The First Results

The first AML evaluation has now been completed.

67 representative memory frameworks successfully completed the evaluation across two tracks covering text memory for both open-source methods and commercial products.

The complete rankings, scores, and system versions are available on the leaderboard.

Leaderboard:
[https://agentmemoryleaderboard.ai/leaderboard/industry/textual ]

What Comes Next?

The first leaderboard is not the finish line.

We plan to keep AML running as a long-term evaluation and public leaderboard for agent memory systems.

Going forward, we will publish deeper technical analyses of the first-round results, including:

Technical architectures that performed well
Interesting evaluation patterns
Performance across individual memory capabilities
Differences between text and code memory
What current results tell us about the evolution of agent memory

We also want the benchmark itself to evolve.

If you are working on AI agents, memory systems, evaluation, or related research, we would love to hear what you think a useful memory benchmark should measure next.

Four Evaluation Tracks

To better reflect different types of memory systems, AML organizes evaluation into two tracks:
Open-source Methods

Text Memory

Commercial Products

Text Memory

Each track evaluates systems under the same benchmark framework.

Explore AML

Leaderboard:
[https://agentmemoryleaderboard.ai/leaderboard/industry/textual]

GitHub:
[https://github.com/AML-memory/agent-memory-leaderboard]

Hugging Face Space:
[https://huggingface.co/agent-memory-leaderboard]

X:
[https://x.com/AgentMemoryL/status/2087544165433590240]

Contact:
contactus@agentmemoryleaderboard.ai

The first results are out.

Now the real work begins:

Make memory measurable. Make progress verifiable.

Why AI Agents Need Memory Benchmarks？

Agent Memory Leaderboard — Mon, 10 Aug 2026 08:18:18 +0000

AI agents are getting better at reasoning, coding, and tool use.

But one question remains open:

How do we know if an agent actually remembers and learns from previous interactions?

Most current evaluations focus on immediate task performance. However, real-world agents often need more than that.

They need to remember:

previous decisions and why they were made
failed approaches and lessons learned
project-specific patterns and context

Memory is becoming a key capability for long-running AI agents, but evaluating it fairly is still challenging.

Different systems often use different datasets, models, and evaluation methods, making direct comparisons difficult.

We believe open and reproducible evaluation is an important step toward building better AI agents.

This week, we will share the first results from an open evaluation effort for AI Agent Memory systems.

More updates coming soon.