From Storing Context to Building Experience: What 50+ Teams Tell Us About Agent Memory

More than 50 teams have registered for the second Agent Memory Challenge shortly after the cycle opened.

They come from different backgrounds: universities, open-source projects, independent developers, and commercial teams. They are building different kinds of memory systems, using different engineering approaches, and targeting different agent workflows.

This number does not tell us which system is the best.

It does tell us something important: Agent Memory is becoming a serious research and engineering problem.

Memory is moving beyond chat history

Early discussions around AI memory often focused on a simple question:

Can an agent remember what happened earlier?

That question is still useful, but it is no longer sufficient for long-running agents.

A production agent may need to work across days or weeks. It may interact with multiple users, repositories, documents, tools, and task states. During that time, facts can change, decisions can be revised, and previous solutions can become invalid.

A useful memory system therefore needs to do more than store additional context. It needs to help an agent:

• retrieve evidence from previous interactions;
• connect related facts across multiple sessions;
• understand when information is relevant to the current task;
• recognize when an earlier decision has been superseded;
• avoid relying on stale or conflicting context;
• reuse previous experience without blindly copying it.

In other words, the goal is not simply to make an agent remember more.

The goal is to help an agent use the right experience at the right time.

Why interest in Agent Memory is growing

The rapid growth of agentic systems has changed the role of memory.

For a one-shot question-answering system, the context window may be enough. But for an agent that maintains a project, debugs software, manages a workflow, or collaborates with a user over a long period, the context window is only one part of the problem.

The agent must also decide:

• what should be remembered;
• what should be ignored;
• what should be updated;
• what should be retrieved later;
• which evidence is still valid;
• and how much historical context should be passed into the next task.

This creates a broad design space.

Some systems focus on structured facts. Others build episodic or procedural memory. Some use graphs, timelines, summaries, or topic documents. Others attempt to learn memory policies or evolve their own internal representations.

The diversity of approaches is valuable, but it also creates a comparability problem.

The evaluation problem

Different memory systems are often evaluated with different datasets, answer models, prompts, retrieval methods, and scoring rules.

As a result, two systems may report impressive results while answering fundamentally different questions.

One system may be optimized for factual recall. Another may focus on multi-hop reasoning. A third may target coding history or multimodal evidence. Without a shared evaluation boundary, it is difficult to understand what the scores actually mean.

This is the problem that Agent Memory Leaderboard is trying to address.

AML separates the memory layer from the downstream answer and evaluation process:

• Participants provide an Add interface for writing and updating memory.
• Participants provide a Search interface for retrieving relevant memory evidence.
• AML standardizes the downstream Answer and Eval pipeline.
• Results are reviewed and published by track, division, submitted version, and evaluation cycle.

This does not create one universal definition of memory.

Instead, it creates a common environment where different approaches can be compared more transparently.

What 50+ registrations may indicate

The early response to Cycle 2 should not be interpreted as proof that the field has already converged on a single architecture.

In fact, the opposite may be true.

The number of participating teams suggests that the field is still exploring several important questions:

1. Memory is becoming a system-level capability

Many teams are no longer treating memory as a small feature added to an agent framework. They are designing dedicated systems for storing, organizing, updating, and retrieving experience.

This reflects a shift from prompt-level context management to memory infrastructure.

2. Different agents need different forms of memory

A conversational agent may need user preferences, personal facts, and temporal events.

A coding agent may need repository history, debugging traces, failed approaches, design decisions, and testing evidence.

A multimodal agent may need to connect text with images, captions, visual events, and ordered context.

These requirements cannot always be solved by the same retrieval strategy.

3. Freshness is as important as recall

A memory system can retrieve a fact correctly and still produce the wrong result if that fact is outdated.

For example:

• a project decision may have been changed;
• a user preference may no longer apply;
• a dependency may have been upgraded;
• a previous debugging hypothesis may have been disproven.

This means future memory evaluations need to test not only whether a system can retrieve historical information, but also whether it can identify current and valid evidence.

4. Agent Memory needs reproducible benchmarks

As more teams build memory systems, informal demos become less sufficient.

A demo can show that a system works in one scenario. A benchmark can help reveal how it behaves across different tasks, histories, updates, noise conditions, and evidence requirements.

The field needs both: practical demonstrations and controlled evaluation.

What Cycle 2 evaluates

The second Agent Memory Challenge includes three tracks.

Textual Memory

The Textual Memory track evaluates long conversations and cross-session history, including:

• explicit fact retrieval;
• multi-hop relationships;
• temporal events;
• personalization;
• rule and workflow following;
• memory governance;
• safety and privacy boundaries.

Cycle 2 also includes streaming memory settings, where Add and Search operations occur as events unfold. This tests whether a system can use information available at the correct point in time, rather than relying only on a static archive.

Coding Memory

The Coding Memory track focuses on long-running software engineering tasks.

It evaluates whether an agent can use historical development experience, including:

• repository context;
• implementation decisions;
• debugging trails;
• failed approaches;
• module relationships;
• testing evidence;
• previously validated fixes.

The formal coding evaluation includes 150 software engineering tasks under both relevant-history and noisy-history conditions.

Multimodal Memory

The Multimodal Memory track evaluates how systems write, retrieve, and use memory involving text and images.

Participants may process original images, captions, or both, but must correctly handle ordered multimodal content and disclose which media types their systems use.

What happens next

The early participation in Cycle 2 is encouraging, but the more important work is still ahead.

The field now needs to move toward clearer definitions and stronger evaluation questions:

• How should memory systems represent changing facts?
• How should they handle conflicting evidence?
• How can provenance be preserved?
• When should a memory be updated, decayed, or deleted?
• How should we measure stale retrieval?
• How can we evaluate memory under noisy or adversarial histories?
• What should a coding agent remember from a previous task?
• How should multimodal evidence be linked to later decisions?

These are not only benchmark questions. They are design questions for the next generation of agents.

Join the benchmark

Agent Memory Challenge Cycle 2 is now open to:

• universities and research institutions;
• open-source maintainers;
• independent research teams;
• commercial product teams;
• individual developers;
• cross-organization teams.

Participation is free. The challenge includes separate Open-source Methods and Commercial Products divisions across the three tracks.

The goal is not to declare one universal winner.

The goal is to make long-term Agent Memory more measurable, more comparable, and easier to improve.

More than 50 teams have already registered. We hope to see more researchers and developers contribute systems, feedback, benchmarks, and ideas to this growing area.

Agent Memory Challenge 2026 Cycle 2

Website:

https://agentmemoryleaderboard.ai/

Evaluation:

https://agentmemoryleaderboard.ai/evaluation

Rules:

https://agentmemoryleaderboard.ai/rules

GitHub:

https://github.com/AML-memory/agent-memory-leaderboard

Twitter Official Link:
https://x.com/AgentMemoryL