More Memory Didn't Fix Coding-Agent Amnesia—SWE-ContextBench Shows Why

ma wenxin — Tue, 11 Aug 2026 08:29:29 +0000

Yesterday, a coding agent fixed a complex bug. It searched the repository, ruled out several wrong directions, found the root cause, and produced a working patch.

Today, a related issue appears in the same codebase. The agent starts from scratch—repeating much of the same exploration, mistakes, and token expenditure.

Giving the agent access to its history sounds like the obvious solution. Our results suggest otherwise: more history can cost more without improving task resolution.

We have been building MemoraX Code, a long-term memory layer for coding agents. That work led us to a harder question:

Can a coding agent reuse past engineering experience to solve related tasks more accurately and efficiently?

To study it, we built and open-sourced SWE-ContextBench.

What Existing Coding Benchmarks Miss

Most coding benchmarks evaluate whether an agent can understand an issue, locate relevant code, produce a patch, and pass the tests.

But real software-engineering tasks are not independent.

An issue may continue an earlier problem. A pull request may fix several related bugs. Even a failed implementation may reveal an architectural constraint that should not be violated again.

Senior engineers accumulate this context. They remember which approaches failed, which modules had similar problems, and which tests are most likely to expose a regression. Most benchmarks do not measure whether an agent can do the same.

SWE-ContextBench contains:

1,100 experience tasks
376 related evaluation tasks
51 real open-source repositories
9 programming languages

The Lite version includes 300 experience tasks and 99 evaluation tasks, making reproduction less expensive.

The benchmark measures task resolution, runtime, and token cost—not only whether the final patch passes.

More History Is Not Necessarily Better

We tested five ways of providing historical experience to an agent:

Context setting	Task resolution	Average cost
No historical context	26.26%	$0.79
Free retrieval over full trajectories	26.26%	$0.98
Oracle full trajectories	27.27%	$0.77
Free retrieval over distilled summaries	22.22%	$0.91
Oracle experience summaries	34.34%	$0.85

Here, Oracle means that the correct historical experience is supplied using known task relationships. It represents a potential upper bound—not the capability of a production retrieval system.

Three findings stand out.

1. Raw history often adds noise

Full trajectories contain file searches, temporary hypotheses, debugging output, abandoned changes, and failed attempts. Free retrieval over those trajectories left task resolution unchanged at 26.26%, while average cost rose from $0.79 to $0.98—an increase of roughly 24%.

2. Short summaries can still be harmful

Free retrieval over distilled summaries reduced task resolution to 22.22%.

Shorter does not mean more relevant. A concise but unrelated memory may appear confident and authoritative, pushing the agent in the wrong direction.

3. Relevance and quality must work together

The strongest result came from supplying a distilled summary of the correct prior task: task resolution increased to 34.34%.

The implication is simple:

Coding-agent memory is not primarily a storage problem. It is a problem of extracting, selecting, and validating experience.

Memory Systems Make Different Tradeoffs

We also evaluated several memory systems on SWE-ContextBench Lite:

Memory system	Task resolution	Average runtime	Average cost
Mem0	24.24%	4.72 min	$0.62
OpenViking	29.20%	4.20 min	$0.53
Supermemory	30.30%	5.04 min	$0.58

No system led in accuracy, runtime, and cost simultaneously. Under this configuration, Supermemory achieved the higher task-resolution rate, while OpenViking performed better on runtime and cost.

These numbers should not be treated as a permanent product ranking. Models, prompts, embeddings, framework versions, retrieval depth, and integration choices can all change the outcome.

The broader result matters more: retrieving relevant content does not guarantee that an agent will use it correctly.

What This Benchmark Cannot Prove Yet

SWE-ContextBench is a starting point, not a final answer.

The tasks come from public open-source repositories and cannot fully represent private enterprise codebases.
GitHub references indicate relationships between tasks, but not every relationship guarantees transferable engineering experience.
Oracle results depend on knowing the correct task relationship in advance; production retrieval systems do not have that advantage.
Framework comparisons are sensitive to configuration, and the Lite evaluation contains only 99 related tasks.

These limitations are also research questions. How should a memory system distinguish a useful dependency from a superficial reference? How should it handle stale, incomplete, or failed experience?

How This Shaped MemoraX Code

The benchmark moved us away from the idea of preserving every interaction. MemoraX Code now follows two core principles:

Retrieve distilled engineering experience within the correct repository scope. This reduces irrelevant context and cross-project contamination.
Treat memory as evidence, not authority. Historical experience must still be checked against the current implementation, documentation, and tests.

In an additional post-paper evaluation on SWE-ContextBench Lite, MemoraX achieved a 78.79% retrieval match rate and a 45.45% task-resolution rate under the tested configuration.

This result was not part of the paper's original experiments. We are reporting it separately as an additional evaluation, not as a paper result.

SWE-ContextBench Is Open Source

The dataset, Lite version, and evaluation scripts are available on GitHub. For the benchmark methodology and full experimental details, read the paper: SWE Context Bench: A Benchmark for Context Learning in Coding.

One question remains:

Which failure is more dangerous—forgetting a useful lesson, or confidently applying an outdated memory from the wrong project?

We are currently more concerned about the second. But if retrieval becomes too conservative, an agent may miss genuinely useful experience.

Where should a coding-memory system draw that boundary?

This article was originally published on MemoraX Engineering.

DEV Community: ma wenxin