The first experiment measured whether Claude Code knew the right answer. This one measured whether the work actually succeeded.
In my previous experiment, I tested whether a memory layer could stop Claude Code from confidently repeating project-specific mistakes.
It did.
But there was an obvious limitation.
The tasks measured whether the answer was correct. They did not ask the agent to modify a repository, run the result, and pass a deterministic test.
So I ran the harder version.
This time, the agent had to do real work in a temporary repository. A checker decided whether the result was correct. No judge model. No subjective scoring. The task either passed or it did not.
Here is the result from the complete run:
| Configuration | Successful runs |
|---|---|
| Claude Code without memory | 50.0% |
Claude Code with CLAUDE.md
|
36.1% |
| Claude Code with RE-call | 58.3% |
RE-call improved success over the CLAUDE.md baseline by 22.2 percentage points.
The 95% confidence interval was +11.1 to +33.3 points.
The paired McNemar test gave a p-value of 0.000145.
This was not one lucky demonstration. It was 72 paired comparisons across 24 tasks and three seeds, with zero discarded cells.
What changed from the first experiment?
Each task asked Claude Code to create or modify files in a repository and run the result.
The benchmark included tasks involving things like:
- deterministic file ordering
- Unicode normalization
- migration naming
- shell quoting
- money rounding
- stable sorting
- line ending handling
- generated files and golden outputs
The important detail is that every task had a deterministic checker.
The agent could say "done" all it wanted. The checker only cared whether the repository ended up in the correct state.
I ran every task through three configurations:
- Bare Claude Code
- Claude Code with the project's
CLAUDE.md - Claude Code with the same environment plus RE-call memory
The harness also verified that the memory tools were actually available before counting a session. If the memory layer was missing, the entire paired cell was discarded.
That happened zero times in the DeepSeek run.
RE-call helped where project memory mattered
The most useful comparison is RE-call versus the static CLAUDE.md file.
The static file produced a 36.1% success rate.
RE-call produced 58.3%.
That is a 22.2 point improvement.
The paired results were even more revealing:
- RE-call won 17 comparisons that
CLAUDE.mdlost -
CLAUDE.mdwon only 1 comparison that RE-call lost
The effect was strongest on the tasks classified as memory-sensitive. Across those eight tasks, RE-call improved success by 45.8 percentage points.
That is the pattern I was looking for.
Memory did not make every task easier. It helped when the task depended on a project decision, convention, or failure mode that was not fully present in the immediate prompt.
The surprising result was CLAUDE.md
The static CLAUDE.md configuration performed worse than the bare configuration:
| Configuration | Successful runs |
|---|---|
| Bare | 50.0% |
CLAUDE.md |
36.1% |
The bare configuration won by 13.9 percentage points.
This is not what I expected.
It suggests that a static instruction file is not automatically a memory system. It can add useful guidance, but it can also add noise, constraints, or instructions that compete with the task.
RE-call is doing something different.
Instead of putting every historical fact into the permanent prompt, it gives the agent a way to retrieve relevant project history when the current task needs it.
That distinction matters.
Memory did not just improve the final score
The benchmark also measured what happened inside the memory mechanism.
In the DeepSeek run:
- The agent searched memory in 83.3% of eligible sessions.
- When it searched, it reached useful context 85.0% of the time.
- Overall, useful project context was reached in 70.8% of sessions.
So the result is not simply that "the model had more tokens."
The memory layer was being consulted, and in most cases where it was consulted, it reached something useful.
There is still room to improve retrieval. A search is not automatically a good search. The model needs to find the right historical decision, not merely retrieve something related to the words in the prompt.
The cost
The complete DeepSeek run cost an estimated $0.4964 at the captured API prices.
The three arms cost:
| Configuration | Estimated cost |
|---|---|
| Bare | $0.0824 |
CLAUDE.md |
$0.0863 |
| RE-call | $0.3277 |
RE-call used about four times as many total tokens as the static prompt configuration.
That is the tradeoff.
The memory layer costs more on every task, including tasks where memory is unnecessary. In exchange, it can prevent the much more expensive failure mode where an agent confidently follows an outdated convention, repeats a known mistake, or spends several minutes debugging a problem the team already solved.
The next engineering question is not whether memory helps.
It is how to make the retrieval smaller, faster, and more selective.
What about GPT-5.3 Codex?
I also started the same model-freeze run with GPT-5.3 Codex.
That run is not a negative result for GPT.
The provider hit a credit and in-flight request limit while the benchmark was running. The harness correctly discarded the affected paired cells instead of treating missing sessions as failed model attempts.
Only 40 of the 72 paired cells were admissible, so I am not using that run to claim that GPT-5.3 Codex is better or worse.
The descriptive numbers pointed in the same direction:
| Configuration | Successful runs |
|---|---|
| Bare | 47.5% |
CLAUDE.md |
30.0% |
| RE-call | 50.0% |
But the run needs to be repeated with the provider capacity issue fixed before it becomes a valid model comparison.
That is an operational failure, not a model conclusion.
What this proves
This experiment does not show that memory makes an agent generally smarter.
It shows something narrower and more useful:
A production memory layer can improve the probability that an agent completes a real repository task correctly.
In this run, RE-call improved success by 22.2 percentage points over the static project file.
It also improved success over the bare configuration, although that gain was smaller at 8.3 points. That difference is important. The main value is not simply adding more instructions. The value is retrieving the right project-specific context when it matters.
The first experiment showed that memory can make answers more correct.
This experiment showed that the improvement survives contact with the repository.
That is the result I was actually waiting for.
What comes next
The next step is a clean competitor comparison:
- rerun GPT-5.3 Codex with sufficient provider capacity
- compare RE-call against other memory systems
- keep the same tasks, seeds, checkers, and admission gate
- publish discarded cells and operational failures alongside the scores
The goal is not to produce another flattering demo.
The goal is to find out whether a memory layer remains useful when the models, vendors, and retrieval systems change.
That is the benchmark I want to run next.
GiulioDER
/
RE-call
Memory that abstains instead of guessing: agent memory on your own Postgres with a verdict, confidence and provenance on every hit, and a calibrated refusal when nothing clears the threshold.
Memory that abstains instead of guessing.
RE-call is agent memory on your own PostgreSQL with pgvector: every hit carries a verdict
confidence and provenance, a retracted claim comes back marked superseded, and a
question the corpus cannot answer is refused rather than answered from the nearest neighbour
ATM-Bench Recall@10 92.8924 against 79.09 for the best published row (limits) · second of ten on MTRAG correct refusals (limits) · zero memory-layer LLM calls to build memory, where Mem0 pays one per session (limits)
Why RE-call · Quickstart · How it works · Product surface · Documentation · Evidence
Setup guide: install, configure and run RE-call · Validity Frontmatter: the open spec RE-call implements
Why RE-call
Nearest-match retrieval cannot tell the difference between what is true and what merely reads like it. When a corpus keeps its history, and real agent memory does, the retracted…
Top comments (1)
This is the kind of agent benchmark I trust more. The paired tasks and deterministic checker make it harder for a memory layer to win by sounding right while still breaking the repo. The zero discarded cells detail matters too, because that is where a lot of small evals accidentally sand off the ugly cases.