I Gave Claude Code a Brain. Then I Measured What It Costs to Ask for One.

At the end of part 2 I promised a clean competitor comparison: more than one memory system, the same admission gate, enough capacity to finish.

This is that run. 1,453 agent sessions, four arms, five corpus conditions, every outcome graded by executing a checker against the code the agent actually wrote. No LLM judge anywhere.

The corpus

Each condition is a 4,911 document corpus, about 27,000 embedded chunks. Of those documents, 4,902 are adversarial distractors generated from the corpus's own vocabulary to be confusable, and the rest are the planted material for 36 tasks.

That ratio is the point. On the small feed I used earlier, retrieval hit@10 was 1.000: everything was findable, so the benchmark could not tell "the product retrieved badly" from "the agent never searched". With the haystack, BM25 hit@1 drops from 0.485 to 0.182.

The five conditions differ in what is planted, and four of the five are traps:

condition	what the corpus holds	share of cells
present	the governing fact, cleanly	37%
superseded	the fact, beside an outdated version of itself	15%
adjacent	something related and inapplicable	16%
absent	the fact deleted, distractors left behind	17%
contradictory	two rival versions, no authority between them	15%

Only 37 percent of cells contain a clean fact worth retrieving. In the other 63 percent the corpus is outdated, contradictory, inapplicable, or empty, and a memory layer's job there is to not make things worse.

I built it that way deliberately. A benchmark where memory can only help is a demo.

The control that changed the question

Two commenters asked for a placebo. Fair: if you tell an agent "you have memory, consult it before acting" and it then does better, you have not separated the memory from the telling.

So I built an arm carrying the complete memory instruction and no memory at all. Same protocol text, byte for byte, as the two real arms. Nothing behind it.

comparison	wins	losses	net	p
instruction only, vs bare	26	43	-17	0.053

Telling an agent to consult memory, when there is no memory, makes it measurably worse. Seventeen cells of 358, borderline significance, in the direction nobody advertises.

That reframes everything. "Does memory beat no memory" is not directly answerable, because every memory product requires that instruction. The answerable question is: does the product earn back the cost of asking for it?

The comparison

RE-call 0.11.0 is mine, so read accordingly. MemPalace 3.8.0 is the better known of the two. Both received the same instruction and the same gate. Every write tool was withheld from every arm.

comparison	wins	losses	net	p
RE-call vs instruction only	41	21	+20	0.015
MemPalace vs instruction only	25	23	+2	0.885
RE-call vs MemPalace	48	30	+18	0.054
RE-call vs bare	47	44	+3	0.834
MemPalace vs bare	38	53	-15	0.142

McNemar exact on discordant pairs. The only comparison clearing p < 0.05 is RE-call against the instruction-only control.

Note the fourth row before anyone else does. Against a bare agent, RE-call is +3 cells at p = 0.834, which is nothing. The win is real against the instruction the product requires and invisible against an agent you never asked. A sceptic will quote that row and is entitled to.

Arm by arm

Tasks solved out of cells, admitted only where all four arms produced a record. Ordered by how much RE-call gains over the instruction-only arm, which is the comparison that isolates the product from the coaching.

condition	cells	bare	instruction	RE-call	MemPalace	RE-call gain
present	134	52/134 · 0.388	53/134 · 0.396	67/134 · 0.500	56/134 · 0.418	+14
superseded	52	38/52 · 0.731	34/52 · 0.654	41/52 · 0.788	35/52 · 0.673	+7
absent	60	49/60 · 0.817	41/60 · 0.683	44/60 · 0.733	37/60 · 0.617	+3
contradictory	54	44/54 · 0.815	41/54 · 0.759	39/54 · 0.722	42/54 · 0.778	-2
adjacent	58	42/58 · 0.724	39/58 · 0.672	37/58 · 0.638	40/58 · 0.690	-2
all	358	225/358 · 0.628	208/358 · 0.581	228/358 · 0.637	210/358 · 0.587	+20

RE-call is the only arm that beats doing nothing, by 0.9 points overall. MemPalace at 210/358 sits below the no-memory baseline and barely above instruction-only. MemPalace's gains against the same control are +3, +1, -4, +1, +1, totalling +2.

The shape of that table matters more than the total. Memory pays where there is something to recall and costs where there is not. On present RE-call converts 67 of 134 against bare's 52. On the three conditions built to punish retrieval it is flat or slightly negative, because surfacing something inapplicable is worse than surfacing nothing.

This is close to a worst case, and that is the point

Go back to the composition. Sixty-three percent of these cells are conditions built to punish a memory layer, where the corpus is outdated, contradictory, inapplicable, or empty and the best available outcome is to waste nothing. Only 37 percent contain a clean fact worth retrieving.

RE-call is still the only arm that finishes above the no-memory baseline. It does that while spending nearly two thirds of its cells in terrain designed to make retrieval a liability, and while paying the -17 cell entry fee that the instruction alone costs.

That is the number I would defend hardest, because it is the unflattering way to measure my own product and it still comes out positive.

And the composition is a choice I made, not a fact about anyone's repository. Real memory does not sit still at that ratio. How much of it is current, governing and correctly superseded is a property of how the layer writes and maintains notes over time, which this benchmark does not measure at all. It fixes the corpus and varies the product.

So read the table as a floor rather than a forecast. On the condition a well maintained corpus is mostly made of, RE-call is +14 of its +20. What that becomes at a healthier ratio I have not measured, and I am not going to put a number on it.

I would rather someone else did. A composition sweep, holding the product constant and varying the mix, is the obvious next experiment and it is not run.

The number I find most telling

For each product, the solved rate when it searched minus when it did not:

condition	RE-call	MemPalace
present	+0.181	+0.019
adjacent	+0.147	-0.032
contradictory	+0.079	+0.085
absent	+0.031	+0.264
superseded	+0.006	-0.094

When RE-call searches it helps in five conditions of five. When MemPalace searches it hurts in two, while searching more often (0.58 to 0.69 against 0.46 to 0.56).

On present, the condition built so the answer is in memory, searching is worth +0.181 to RE-call and +0.019 to MemPalace. That is the cell both products exist for.

Cost, median beside mean because these distributions are skewed enough that on an earlier run of mine the two disagreed in sign: RE-call 65.9 s and 56,476 input tokens per session, MemPalace 68.6 s and 90,706, bare 44.7 s and 14,085. RE-call is faster and uses 38 percent fewer tokens than MemPalace while delivering ten times the net benefit. Both are materially heavier than no memory, which is the honest cost of the idea.

What I had to publish against myself

The previous run said something different, and I want to be the one who says why.

In it, RE-call looked like the most damaging arm of seven, worst on three of four adversarial conditions. I had written that up as a genuine weakness of my own product.

It was my harness. That run gave RE-call 1,958 bytes of coaching over the shared protocol against MemPalace's 853, and most of the surplus was not about RE-call at all: search before your first write, search by symptom rather than goal. Generic advice that helps any retrieval system, handed to one of them. Worse, the flag selecting the fair variant was hardcoded in my launcher with no override, so no run I had ever done could have been matched, and I had not noticed.

Under the corrected instruction the finding evaporates. RE-call's damage-only net harm falls from +0.073 to +0.0122, identical to MemPalace's, and its superseded damage from 0.179 to 0.036. Its usefulness score also fell, 0.377 to 0.281, while MemPalace's did not move at all, because MemPalace had been on the shared protocol all along. My product's advantage narrowed from 1.8x to 1.35x once I stopped giving it a bigger prompt.

Both corrections point the same way. I had been generous to myself and unfair to the competitor, and it produced one flattering finding and one damning one, both wrong.

What this benchmark cannot see

RE-call has a reasoning-graph layer. Across 2,181 sessions of the earlier run its four graph tools were called zero times. Read alone, damning.

It builds that graph from authored cross-references, the way a wiki links notes. Its own memory store has 44,364 of them. This benchmark's corpus is rendered agent transcripts: conversation turns, no cross-references. The graph gets entities and no edges. It is a node set, not a graph.

So graph reasoning needs an authored, cross-referenced corpus and this benchmark deliberately measures raw transcripts. Different corpus classes, not a weakness. Manufacturing cross-references in to make the feature look good would be assuming the answer. I established that with ten read-only queries rather than the 220 sessions the experiment would have cost, after a reviewer pointed out my draft design would have measured a language model reading trusted evidence and published it as a graph result.

Come and break it

Two systems is a comparison, not a field, and one of them is mine. The harness is public: github.com/GiulioDER/agent-memory-bench.

If you build a memory layer for coding agents, I would rather measure yours than have you take my word for this. Every adapter is hash-pinned and every vendor is invited to review their own config before a measured run, which is what adapters/<name>/VENDOR_REVIEW.md is for. Open an issue and I will help wire it up.

And if you just want to check my arithmetic, the README has a section on verifying a published run with no credentials and no money. The preregistrations are in the repo, including the ones whose predictions I got wrong, and the raw records are published with each run.

If you take one number from this, take that one: 63 percent of the cells were built to make memory a liability, and the memory layer still finished ahead of not having one.

One thing I would most like broken: the 45 percent. On superseded, the largest single loss is that in 45 percent of sessions neither planted document is retrieved at all. That is worth roughly three times more than anything I have measured on the ranking side, it is a query-formulation problem rather than a retrieval-quality one, and it is already preregistered. If someone has a better idea than mine, I would like to lose that one.