Yesterday, a coding agent fixed a complex bug. It searched the repository, ruled out several wrong directions, found the root cause, and produced a working patch.
Today, a related issue appears in the same codebase. The agent starts from scratch—repeating much of the same exploration, mistakes, and token expenditure.
Giving the agent access to its history sounds like the obvious solution. Our results suggest otherwise: more history can cost more without improving task resolution.
We have been building MemoraX Code, a long-term memory layer for coding agents. That work led us to a harder question:
Can a coding agent reuse past engineering experience to solve related tasks more accurately and efficiently?
To study it, we built and open-sourced SWE-ContextBench.
What Existing Coding Benchmarks Miss
Most coding benchmarks evaluate whether an agent can understand an issue, locate relevant code, produce a patch, and pass the tests.
But real software-engineering tasks are not independent.
An issue may continue an earlier problem. A pull request may fix several related bugs. Even a failed implementation may reveal an architectural constraint that should not be violated again.
Senior engineers accumulate this context. They remember which approaches failed, which modules had similar problems, and which tests are most likely to expose a regression. Most benchmarks do not measure whether an agent can do the same.
SWE-ContextBench contains:
- 1,100 experience tasks
- 376 related evaluation tasks
- 51 real open-source repositories
- 9 programming languages
The Lite version includes 300 experience tasks and 99 evaluation tasks, making reproduction less expensive.
The benchmark measures task resolution, runtime, and token cost—not only whether the final patch passes.
More History Is Not Necessarily Better
We tested five ways of providing historical experience to an agent:
| Context setting | Task resolution | Average cost |
|---|---|---|
| No historical context | 26.26% | $0.79 |
| Free retrieval over full trajectories | 26.26% | $0.98 |
| Oracle full trajectories | 27.27% | $0.77 |
| Free retrieval over distilled summaries | 22.22% | $0.91 |
| Oracle experience summaries | 34.34% | $0.85 |
Here, Oracle means that the correct historical experience is supplied using known task relationships. It represents a potential upper bound—not the capability of a production retrieval system.
Three findings stand out.
1. Raw history often adds noise
Full trajectories contain file searches, temporary hypotheses, debugging output, abandoned changes, and failed attempts. Free retrieval over those trajectories left task resolution unchanged at 26.26%, while average cost rose from $0.79 to $0.98—an increase of roughly 24%.
2. Short summaries can still be harmful
Free retrieval over distilled summaries reduced task resolution to 22.22%.
Shorter does not mean more relevant. A concise but unrelated memory may appear confident and authoritative, pushing the agent in the wrong direction.
3. Relevance and quality must work together
The strongest result came from supplying a distilled summary of the correct prior task: task resolution increased to 34.34%.
The implication is simple:
Coding-agent memory is not primarily a storage problem. It is a problem of extracting, selecting, and validating experience.
Memory Systems Make Different Tradeoffs
We also evaluated several memory systems on SWE-ContextBench Lite:
| Memory system | Task resolution | Average runtime | Average cost |
|---|---|---|---|
| Mem0 | 24.24% | 4.72 min | $0.62 |
| OpenViking | 29.20% | 4.20 min | $0.53 |
| Supermemory | 30.30% | 5.04 min | $0.58 |
No system led in accuracy, runtime, and cost simultaneously. Under this configuration, Supermemory achieved the higher task-resolution rate, while OpenViking performed better on runtime and cost.
These numbers should not be treated as a permanent product ranking. Models, prompts, embeddings, framework versions, retrieval depth, and integration choices can all change the outcome.
The broader result matters more: retrieving relevant content does not guarantee that an agent will use it correctly.
What This Benchmark Cannot Prove Yet
SWE-ContextBench is a starting point, not a final answer.
- The tasks come from public open-source repositories and cannot fully represent private enterprise codebases.
- GitHub references indicate relationships between tasks, but not every relationship guarantees transferable engineering experience.
- Oracle results depend on knowing the correct task relationship in advance; production retrieval systems do not have that advantage.
- Framework comparisons are sensitive to configuration, and the Lite evaluation contains only 99 related tasks.
These limitations are also research questions. How should a memory system distinguish a useful dependency from a superficial reference? How should it handle stale, incomplete, or failed experience?
How This Shaped MemoraX Code
The benchmark moved us away from the idea of preserving every interaction. MemoraX Code now follows two core principles:
- Retrieve distilled engineering experience within the correct repository scope. This reduces irrelevant context and cross-project contamination.
- Treat memory as evidence, not authority. Historical experience must still be checked against the current implementation, documentation, and tests.
In an additional post-paper evaluation on SWE-ContextBench Lite, MemoraX achieved a 78.79% retrieval match rate and a 45.45% task-resolution rate under the tested configuration.
This result was not part of the paper's original experiments. We are reporting it separately as an additional evaluation, not as a paper result.
SWE-ContextBench Is Open Source
The dataset, Lite version, and evaluation scripts are available on GitHub. For the benchmark methodology and full experimental details, read the paper: SWE Context Bench: A Benchmark for Context Learning in Coding.
One question remains:
Which failure is more dangerous—forgetting a useful lesson, or confidently applying an outdated memory from the wrong project?
We are currently more concerned about the second. But if retrieval becomes too conservative, an agent may miss genuinely useful experience.
Where should a coding-memory system draw that boundary?
This article was originally published on MemoraX Engineering.
Top comments (0)