From “Remembering Code” to “Solving Tasks”: How Coding Memory Helps Agents Reuse Engineering Experie

Compared with textual memory, Coding Memory asks a more direct question:

Can experience accumulated during past software development actually help an Agent complete the next software-engineering task?

This is the question behind the first Agent Memory Leaderboard (AML) Coding Memory evaluation.

A Coding Agent does not work in an empty repository.

It enters a codebase with a history:

• previous feature implementations;
• old bug reports;
• rejected approaches;
• commit messages;
• test failures;
• error traces;
• code-review discussions;
• file paths;
• function names;
• and development sessions that may contain clues about how the project actually works.

Some of this history is useful. Some of it is outdated. Some of it is highly relevant but difficult to find. Some of it may lead an Agent in the wrong direction if retrieved without context.

The purpose of Coding Memory is not simply to store this history.

It is to help an Agent use the right part of it at the right moment.

The first AML Coding Memory benchmark was built from 12 real code repositories. It organized 1,290 annotated historical engineering tasks and evaluated systems on 150 held-out tasks:

• 51 New Feature tasks
• 99 Bug Fix tasks

The memory system writes and retrieves relevant historical information. A standardized Coding Agent then uses the retrieved context to solve the current task.

That design makes the final metric unusually concrete.

The question is not only:

Did the system retrieve relevant historical content?

It is:

Did that historical content help the Agent complete the software-engineering task?

The first leaderboard results show two very different states.

In the industry ranking, a visible gap has already appeared at the top.

In the open-source ranking, leading methods remain tightly clustered.

Together, these results suggest that Coding Memory is beginning to develop distinct technical routes—but no single architecture has yet become the final answer.

1. Coding Memory Is Not Just Textual Memory Applied to Code

Textual memory is often framed as a retrieval problem.

A user asks a question. The system retrieves relevant past conversation. An answer model reasons over the returned context.

Coding Memory has to operate under a more demanding version of this pattern.

The information an Agent needs may not appear in one clean natural-language sentence.

It may be distributed across:

• an issue describing an earlier failure;
• a pull request that introduced a similar feature;
• a commit that explains why a particular workaround exists;
• an error log showing the actual failure mode;
• a test that captures the intended behavior;
• and a later code review explaining why the first solution was rejected.

A coding task may also require multiple forms of reasoning at once.

The Agent may need to understand:

• repository structure;
• dependencies between modules;
• naming conventions;
• historical implementation patterns;
• current test expectations;
• and the difference between a previous fix and the current failure.

This means a Coding Memory system cannot be evaluated only by whether it retrieves similar-looking text.

It must help the Agent make a better engineering decision.

The history must eventually influence the outcome of a real task:

a feature is implemented correctly, a bug is fixed, tests pass, and the patch fits the existing project.

2. What AML Measures in Coding Memory

The AML Coding Memory benchmark separates the memory layer from the Coding Agent.

The memory system is responsible for two broad jobs:

• writing historical development information;
• retrieving relevant information for a new task.

The downstream Coding Agent is responsible for the final implementation work.

This separation matters.

A memory system may retrieve a large amount of accurate history but still fail to help the Agent if the returned context is too broad, too noisy, too old, or not actionable.

Likewise, an Agent may occasionally solve a task without useful historical memory—but that does not prove the memory system is strong.

AML therefore evaluates the result at the task level.

The central metric is Task Solve.

This moves Coding Memory evaluation beyond questions such as:

How many memories were stored?

How high was Recall@K?

How similar was the retrieved content to the query?

Those signals can still be useful for diagnosis. But they are not the final goal.

The final question is whether history improves the Agent’s ability to complete the work.

3. The First Results: Two Different Leaderboard Shapes

The first Coding Memory results reveal a sharp contrast between industry systems and open-source methods.

The industrial ranking already has a clear leading system.

The open-source ranking has a large group of systems separated by less than one percentage point.

This does not mean the industrial systems all use one shared method, or that open-source systems are technically identical.

Instead, it suggests that the two groups are at different stages of differentiation.

Industrial systems are beginning to show more visible product-level separation.

Open-source systems are still exploring a broad design space in which several very different architectures can achieve nearly the same overall result.

That makes the leaderboard more interesting than a simple list of scores.

The central question becomes:

What kinds of memory are different systems trying to build for Coding Agents?

4. Industry Ranking: MemoraX Leads by Ten Points

In the industry ranking, MemoraX achieved the highest overall result.

Its Task Solve rate was 62.00%.

Its task-type results were:

• New Feature: 70.59%
• Bug Fix: 57.58%

The next group—claude-mem, hs, and MemOS—each achieved 52.00% overall Task Solve.

This creates a ten-percentage-point gap between MemoraX and the second tier.

For a benchmark built around real software-engineering tasks, this is a meaningful difference.

It suggests that the way a system selects, updates, and reuses historical experience can matter materially—not only for retrieval quality, but for whether the downstream Agent completes the task.

The public materials behind the leading systems also reveal different product philosophies.

Some systems emphasize preserving an Agent’s development trajectory and restoring it when needed.

Others try to identify which experiences from long-running development work are worth reusing in future tasks.

5. MemoraX: Memory as Reusable Engineering Experience

Public materials indicate that MemoraX Code does not treat Memory as only a “store and retrieve” layer.

It also considers:

• which information is worth writing;
• which memory should be updated;
• when memory should be recalled;
• and how prior engineering work can become useful in a later task.

MemoraX Code combines local repository memory with long-term memory.

Its Memory Model learns mechanisms for memory filtering, updating, and recall.

But the more distinctive part of the approach is what happens on top of historical development trajectories.

The system extracts Procedure Memory.

Procedure Memory is not simply a record of what happened in the past.

It aims to capture reusable engineering experience:

• how a similar problem was analyzed;
• which files or modules were inspected;
• what implementation path was chosen;
• what alternatives failed;
• how the result was verified;
• and which workflow may be useful again.

In one public experiment, MemoraX distilled 15 engineering experiences from 123 historical task segments, then organized them into four categories of Procedure Memory.

This introduces a different question for Coding Memory.

Instead of asking only:

What happened in the past?

the system asks:

Which past engineering experience should influence the next task?

That distinction matters.

A raw historical record can tell an Agent that a bug happened before.

A reusable procedure may help it decide where to begin, what to check, and how to verify the fix.

6. From Event History to Procedure Memory

Consider a simplified example.

A previous task involved an API timeout.

The development trajectory may have included:

• a bug report about transient failures;
• inspection of a network client;
• an unsuccessful first retry implementation;
• a revised retry strategy;
• added error handling;
• and new regression tests.

A conventional memory system may store each event independently.

It can later retrieve the issue, the relevant commit, or the test file.

That can be useful.

But a procedure-oriented memory system attempts to preserve something more general:

When this project encounters transient API failures, inspect the client boundary, distinguish retryable errors from permanent errors, use the established retry pattern, and validate the behavior with the existing test conventions.

The goal is not to replace raw evidence with an unsupported summary.

It is to make past engineering experience easier to reuse when a new task is structurally similar.

This is especially relevant for long-running repositories.

As development history grows, repeatedly rediscovering the same debugging and implementation patterns becomes expensive.

A Coding Agent that can reuse verified procedures may spend less time repeating exploratory work.

7. claude-mem: Preserve the Development Trail

claude-mem emphasizes another important form of Coding Memory.

It continuously records the development activity of a Coding Agent.

This can include tool calls, observations, and other traces created while an Agent works through a task.

Those records are organized into more compact semantic entries and persist across sessions.

When a new session begins, the Agent does not need to load all historical activity into its context window.

Instead, it can:

1. search relevant records;
2. inspect a timeline around a promising result;
3. recover detailed content only when the current task requires it.

This is a layered-retrieval approach to coding history.

It tries to avoid two opposite failure modes.

The first is loading too much past history into context at once.

The second is compressing history so aggressively that the Agent cannot recover the original reasoning trail when it needs more detail.

By preserving a searchable semantic record and a timeline that can expand into fuller context, claude-mem aims to make development sessions resumable.

Its first AML result was:

• Overall Task Solve: 52.00%
• New Feature: 56.86%
• Bug Fix: 49.49%

This illustrates a different Coding Memory thesis:

A useful Agent does not always need a distilled procedure. Sometimes it needs to resume the actual trail of work.

8. Persistent Development History Is Also a Product Capability

Development work often unfolds across multiple sessions.

An Agent may inspect a repository today, identify a likely cause, make a partial change, encounter a failed test, and continue later.

Without memory, the next session may need to rediscover:

• which files were relevant;
• which commands were already run;
• which hypothesis had been rejected;
• what error output was observed;
• and why a particular implementation path was chosen.

This is not only a retrieval problem.

It is a continuity problem.

A timeline-oriented memory system treats the Agent’s work itself as part of the memory.

The history is not limited to the user’s request or the final patch.

It also includes the path through the repository.

This can be useful when the next task is not identical to the previous one but touches the same modules, architectural decisions, or unfinished investigation.

The broader implication is that Coding Memory can preserve more than facts about a codebase.

It can preserve the development process that produced those facts.

9. Industry Systems Are Beginning to Diverge

The first industrial leaderboard suggests that Coding Memory is no longer one generic product category.

Different systems are beginning to emphasize different kinds of value.

One route focuses on continuous capture and recovery of an Agent’s development trajectory.

Another focuses on filtering and distilling long-term work into reusable engineering procedures.

Both approaches address the same underlying problem:

How can an Agent use past development work without repeatedly loading, reading, and rediscovering everything?

But they make different choices about what memory should look like.

Should memory preserve the sequence of work?

Should it extract generalized experience?

Should it do both?

The first leaderboard does not settle these questions.

But it shows that these design choices are starting to appear in final software-engineering outcomes.

10. Open Source: Eight Methods Tied for First

The open-source ranking looks very different.

Eight methods tied for first with an overall 52.67% Task Solve rate:

• AM-Link
• AMC-Memory
• aml-memory-baseline
• aml-memory-mvp
• causal-memory
• Hybrid Episodic Memory
• Memoria
• nano-memory

Immediately behind them, agent-memory, ChronoHybridMem, mem0-BQE, and MemoryBear each achieved 52.00%.

Across more than a dozen leading open-source systems, the difference in final task success is less than one percentage point.

At this stage, ranking systems strictly as first, second, or third is less revealing.

The more interesting question is:

How can different memory architectures produce nearly identical task-level results?

The answer appears to be that Coding Memory still has multiple viable routes.

Some systems preserve raw history.

Some rely on lexical and semantic hybrid retrieval.

Some add code-specific retrieval signals.

Some create more structured memory representations.

The first AML results do not show that one route has already won.

11. causal-memory: Preserve History, Then Retrieve It Well

The causal-memory submission primarily preserves original historical information.

At retrieval time, it combines lexical matching with semantic retrieval. Different signals are fused to rank historical content relevant to the current task.

The overall idea is direct:

Do not impose heavy abstraction on the history before storage.

Preserve the original engineering record, then make retrieval effective.

Its results were:

• Overall Task Solve: 52.67%
• New Feature: 62.75%
• Bug Fix: 47.47%

This is important because it challenges a common assumption.

It is tempting to believe that Coding Memory must first convert development history into increasingly sophisticated summaries, entities, graphs, or abstractions.

But the first results suggest that this is not necessarily required for a competitive system.

Raw history can remain valuable.

If the system can retrieve the right issue, commit, code change, error trace, or discussion at the right time, the original record may be more useful than a more aggressively transformed representation.

12. Why Raw History Can Matter in Code

Software-engineering context is often highly specific.

A file path can matter.

A function name can matter.

A single error string can matter.

A note in a previous pull request can matter.

A summary may preserve the broad idea of a previous change while losing the precise information needed to make the next patch.

For example, a summary may say:

The API client was updated to handle transient failures.

But the original engineering record may reveal:

• which exception types were retryable;
• which function was modified;
• where retries were intentionally capped;
• which test exposed the failure;
• and why another implementation was rejected.

The broad summary is useful for orientation.

The raw record may be necessary for correct implementation.

A Raw-First approach does not eliminate the need for structure.

It simply treats the original historical record as something that should remain available rather than being replaced completely.

13. Memoria: Hybrid Retrieval Optimized for Code

Memoria also uses hybrid retrieval, but adapts it more directly to the details of software engineering.

In addition to semantic retrieval and full-text keyword search, it considers signals such as:

• function names;
• file paths;
• snake_case;
• CamelCase;
• exception messages;
• code tokens;
• and historical messages near retrieved records.

This is a practical response to the fact that code is not ordinary prose.

A developer—or a Coding Agent—may search using an error string, a function name, a package path, or a specific identifier that has little meaning outside one repository.

Natural-language semantic similarity alone may not capture these signals reliably.

Memoria’s code-aware retrieval design gives these concrete engineering details a direct role in search.

Its results were:

• Overall Task Solve: 52.67%
• New Feature: 60.78%
• Bug Fix: 48.48%

The system illustrates a broader point:

Coding Memory should reflect how software projects are actually named, navigated, debugged, and changed.

14. Code Tokens Are Not Noise

In some memory tasks, highly specific tokens may look like clutter.

In coding tasks, they are often the most valuable part of the context.

Consider the following kinds of information:

• fetchWithRetry
• api/client.ts
• ECONNRESET
• NullPointerException
• get_user_profile
• src/services/auth
• Retry-After
• test_rate_limit_timeout

A purely semantic retriever may recognize that a query involves an API failure.

But it may not know which historical record contains the exact function, error signature, or test convention that matters.

Code-aware retrieval can treat these terms as first-class evidence.

It can also use surrounding historical messages to restore context around a matching record.

This is especially useful because software engineering often involves local patterns.

The same repository may use one naming convention, one folder structure, one error-handling pattern, and one test style that differ from another project.

A useful Coding Memory system should help the Agent retrieve repository-specific knowledge rather than relying only on general programming knowledge.

15. agent-memory: A Direct Raw-First Alternative

agent-memory adopts a more direct Raw-First approach.

Historical messages are not first converted into new LLM-generated memory summaries.

They are stored as original records.

At retrieval time, the system combines BM25 and local dense retrieval, then fuses the results.

Its first AML scores were:

• Overall Task Solve: 52.00%
• New Feature: 50.98%
• Bug Fix: 52.53%

This result reinforces the observation from causal-memory.

More complex pre-storage abstraction is not a prerequisite for useful Coding Memory.

A system can preserve raw engineering records, combine lexical and dense retrieval, and still remain competitive in task-level results.

The question is not whether raw history or abstraction is always better.

The question is whether the representation helps the Agent retrieve information that is actionable for the current task.

16. Different Routes, Similar Overall Results

The open-source leaderboard contains several systems with similar overall scores but different design choices.

causal-memory emphasizes original historical information plus hybrid retrieval.

Memoria adds code-specific signals such as paths, identifiers, and exception messages.

agent-memory retains raw messages and fuses BM25 with local dense retrieval.

Their close overall results show that a high-level score can hide meaningful architectural differences.

This is why Coding Memory should not be reduced to one broad label.

Two systems may both claim to “retrieve relevant history,” while one is optimized for exact engineering tokens, another for semantic similarity, and another for preserving the raw development record.

These differences may become more visible as benchmarks expand, repositories become larger, and task types become more diverse.

17. New Features and Bug Fixes May Need Different Memories

The split between New Feature and Bug Fix tasks is particularly revealing.

Consider three representative open-source results:

System	New Feature	Bug Fix
causal-memory	62.75%	47.47%
Memoria	60.78%	48.48%
agent-memory	50.98%	52.53%

These numbers do not prove that one memory architecture is inherently better for one class of task.

The benchmark is not yet large enough to support such a universal conclusion.

But the pattern suggests an important possibility:

New Feature development and Bug Fix work may benefit from different kinds of historical memory.

18. What New Feature Tasks May Need

When implementing a new feature, an Agent may need to understand how the repository prefers to build things.

Useful historical context may include:

• similar feature implementations;
• module boundaries;
• architectural patterns;
• data-flow conventions;
• interface design;
• project-specific abstractions;
• and tests that show how new behavior is normally introduced.

The key question may be:

How has this project solved a structurally similar problem before?

In this setting, a previous implementation pattern can be more useful than an isolated error message.

A system that can retrieve related modules, prior feature work, and established project conventions may provide a stronger starting point for the Agent.

19. What Bug Fix Tasks May Need

Bug Fix work often asks for another kind of history.

The useful evidence may include:

• error messages;
• failing tests;
• stack traces;
• affected files;
• prior failed attempts;
• earlier bug reports;
• previous fixes;
• and the verification process that confirmed a repair.

The question is often not:

How should this repository generally implement a feature?

It is:

Where has this failure appeared before, what did people try, and which path actually worked?

A Bug Fix memory system may therefore benefit from more precise retrieval of diagnostics and local change history.

It may need to find the relevant function, exception signature, test failure, or prior patch with much higher specificity.

This may help explain why systems with similar overall Task Solve rates can differ noticeably when New Feature and Bug Fix tasks are separated.

20. Coding Memory Is Task-Conditioned Memory

This leads to a broader conclusion.

There may not be one universally optimal Coding Memory.

The right retrieval strategy may depend on the task.

A New Feature task may need:

• architectural context;
• reusable patterns;
• related implementation examples;
• and project conventions.

A Bug Fix task may need:

• diagnostic evidence;
• error-specific history;
• local modifications;
• failed attempts;
• and validation traces.

A general-purpose Coding Memory system may eventually need to recognize which kind of task it is facing before deciding:

• what to retrieve;
• how much context to return;
• whether to prioritize broad architectural examples or narrow failure evidence;
• and whether a reusable procedure is more valuable than a raw historical record.

This is a harder problem than simply searching a repository history.

It is a problem of matching the shape of memory to the shape of work.

21. From Retrieval Quality to Engineering Outcomes

The first AML Coding Memory leaderboard moves memory evaluation forward in an important way.

It does not stop at asking whether an Agent can remember the past.

It asks whether the past changes what the Agent can accomplish.

A memory system may look impressive if it stores rich trajectories, builds structured records, or retrieves semantically similar text.

But these capabilities matter only if they improve the next engineering decision.

Can the Agent locate the correct part of the repository more quickly?

Can it avoid repeating a failed attempt?

Can it reuse an implementation pattern that was already validated?

Can it understand why a previous fix worked?

Can it add the right tests?

Can it solve the current task with fewer unnecessary steps?

These are the questions that turn memory from a context store into engineering infrastructure.

22. From “Remembering the Past” to “Reducing Repeated Trial and Error”

The first Coding Memory results do not yet show that one technical route has become the definitive answer.

The leading industry score was 62.00%.

The leading open-source tier reached 52.67%.

Both commercial and open-source systems have substantial room to improve.

But the field has already moved beyond a narrower view of memory.

Memory is no longer only about preserving prior interactions.

It is becoming a way to reduce repeated trial and error.

A useful Coding Memory system should not merely tell an Agent:

Here is something related that happened before.

It should help the Agent understand:

Here is the past experience that matters for this task.

Here is the relevant implementation, failure, or procedure.

Here is the context needed to use it correctly.

And here is the evidence that allows you to verify it.

23. The Second Agent Memory Challenge Opens Soon

The first AML Coding Memory results showed both meaningful progress and major open questions.

Industrial systems have begun to show a visible separation at the top of the ranking.

Open-source methods remain highly competitive, but the leading systems are still closely grouped.

Raw-history retrieval, code-aware hybrid search, session restoration, and reusable procedure memory are all active directions.

The next question is not simply:

Which system can retrieve more history?

It is:

Can Agent Memory become more task-aware, more reusable, and more reliable across real long-term engineering work?

The Second Agent Memory Challenge opens on September 20, 2026.

This cycle includes three tracks:

• Textual Memory — long-horizon conversation retrieval, temporal reasoning, and user-profile alignment
• Coding Memory — development-context retrieval for real software-engineering tasks
• Multimodal Memory — cross-modal memory association across text, images, audio, and video

Each track includes both an Open-Source Methods Ranking and an Industry Systems Ranking.

The challenge is globally open to universities, research institutions, companies, open-source teams, and individual developers. There is no registration fee.

The open-source ranking includes a total prize pool of RMB 150,000, with awards including:

• First Prize: RMB 20,000
• Second Prize: RMB 8,000
• Third Prize: RMB 3,000
• Best Technical Innovation Award: RMB 5,000

Important dates:

• Registration and submission entry opens: September 20, 2026, 00:00
• Rolling evaluation period: September 20 – October 31
• Submission deadline: October 31, 2026, 23:59
• Evaluation queue closes: November 4, 2026, 23:59
• Results planned for release: mid-November 2026

Each participant may initiate up to two full official evaluations.

Once evaluation begins, the submitted version is frozen. The platform then evaluates the system automatically through the standardized Add/Search interface.

Whether you are building a memory layer for conversational Agents, Coding Agents, or multimodal systems, the second AML cycle is an opportunity to test what your architecture can contribute under a shared and reproducible evaluation setting.

Registration opens on September 20. We look forward to seeing what the next generation of Agent Memory systems can do.

More information:

https://mp.weixin.qq.com/s?__biz=MzcwOTM5MjU5OA==&mid=2247483788&idx=1&sn=05b1e0c52ab185fe3330361aacbef351&scene=21#wechat_redirect

AML

Agent Memory Leaderboard

https://agentmemoryleaderboard.ai/

Leaderboard

https://huggingface.co/spaces/agent-memory-leaderboard/leaderboard

Content Notice

This article is based on publicly available system materials and the first-cycle AML Coding Memory leaderboard results. It reflects an analytical interpretation of the systems discussed and does not represent an official technical recommendation from AML.

Scores apply only to the referenced evaluation cycle, track, submitted system version, and official leaderboard release. They should not be interpreted as guarantees of performance in every software-engineering scenario.

Comments

[Giulio D'Erme]

Thanks for the detailed article. I have one question about the Coding Memory challenge protocol.
Can the coding agent call Search multiple times while solving a task, using queries derived from repository inspection, error messages, file paths, or test failures? Or does the platform call Search only once using the original benchmark question?
This distinction is important, because it determines whether participants can experiment with retrieval timing and action-conditioned queries, or only with Add-time memory construction and the response to the initial Search call.