DEV Community: Giulio D'Erme

I Rebuilt My AI Memory Benchmark. Here Is What It Can Measure Now.

Giulio D'Erme — Fri, 04 Sep 2026 10:41:16 +0000

A follow up to I Gave Claude Code a Brain. Then I Measured What It Costs to Ask for One

My last article presented a memory benchmark as a comparison between products. The comments correctly pointed out that the design was still mixing several questions together.

Was the agent benefiting from memory, or from being told to search? Was the reported p value a planned test, or simply the most interesting result among several comparisons? Did the control represent a real empty memory store, or only the absence of memory tools?

I rebuilt the benchmark around those questions.

The result is not just a larger leaderboard. It is a more useful instrument. It can now separate several parts of the memory path, measure both help and harm, check whether the treatment was actually present, and let a reader verify the arithmetic from the published session artifacts.
It can also produce a null. In the first official run with the improved design, the placebo scored highest and no arm had an interval that excluded zero.
That is the result. This article is about how to read it.

The measurement question

The benchmark no longer asks only:

Does a memory product improve task success?

That question is too compressed. A memory system can fail because the agent never decides to search, because it asks a poor query, because retrieval returns the wrong passage, because the agent ignores a useful passage, or because the retrieved passage is stale or irrelevant.

The benchmark now asks a sequence of narrower questions:

Does the agent complete more real coding tasks?
Does memory help when the governing fact is present?
Does memory make the agent worse when the corpus is stale, contradictory, unrelated, or empty?
Is any change caused by retrieval, or only by extra instructions?
Is the system paying for a real benefit in tokens and time?
Can another person verify that the reported result follows from the sessions that produced it?

The primary endpoint remains simple. Claude Code edits a repository, and an executable checker decides whether the resulting artifact is correct. There is no LLM judge deciding whether the answer sounds good.

What each arm is for

The arms are not eight competing products. Several are controls or diagnostic tracks. Treating them all as entries in one ranking would throw away the point of the design.

Arm	What it receives	What it controls or tests
`bare`	No memory layer and no memory instruction	The floor. What the agent does without memory or memory coaching
`claude_md`	The task's static project instructions	The designated baseline for the current leaderboard
`placebo`	Neutral project shaped prose, with no task facts or memory tools	Whether extra context alone changes task success
`protocol`	The shared memory instruction, but no memory surface	The cost or benefit of telling the agent to search
`fs_grep`	The transcript corpus as files, searched with ordinary repository tools	Whether a memory product beats a simple searchable file feed
`recall`	The RE call memory layer and its search tools	Natural use: the agent decides whether to search and how to query
`mempalace`	The MemPalace integration through its published interface	A product comparison under the same shared protocol
`recall_prefetch`	RE call retrieval run by the harness using the exact task prompt	Retrieval with the decision to search and query formulation removed

There is also an oracle_memory diagnostic in the repository. It injects the exact relevant evidence before a session and acts as a ceiling control. It is not a product and it is not part of the official leaderboard. Its purpose is to answer a different question: can the agent use correct evidence when retrieval has already been solved?

`bare` is not the same as `protocol`

This distinction is central.

bare receives neither memory tools nor instructions about memory. It measures the agent's ordinary performance.
protocol receives the shared instruction that tells a memory enabled agent how to use memory, but it has no memory store. It measures what happens when the coaching exists without retrieval.
The difference between protocol and bare is therefore the effect of the instruction. The difference between recall and protocol is the value of adding the memory system to that instruction.
That is the comparison the previous article needed to make more explicit.

`placebo` is not an empty memory store

The placebo contains neutral project shaped prose. It does not expose a memory interface and it does not return an empty search result.
An empty store would hold the interface constant, accept the search, return no evidence, and let the agent continue. That is a different control. The current official run does not include it.
The new design therefore separates coaching from retrieval, but it does not yet isolate interface overhead from coaching. That remains a planned experiment, not a result I can claim today.

`recall` and `recall_prefetch` measure different paths

The natural recall arm asks the agent to decide when memory is worth consulting and to formulate a useful query. Its result includes the product and the agent's use of it.
The recall_prefetch arm runs retrieval in the harness using the exact task prompt, then gives the returned evidence to the agent without exposing memory tools. It removes the decision to search and most of the query formulation problem.

The comparison is diagnostic:

If prefetch is better than natural RE call, the agent's search decision or query is part of the bottleneck.
If both are better than the baseline by a similar amount, the retrieved evidence is probably doing most of the work.
If neither is better, better retrieval alone may not solve the task.

Neither arm should be interpreted as a general product ranking without the paired analysis and the other controls.

The improvements to the benchmark

The instruction is now controlled

In the earlier run, the memory arms carried different amounts of generic coaching. That made it possible for an apparent retrieval advantage to be an instruction advantage.
The official run uses one shared protocol document for the memory arms. Its measured size was 3,472 bytes, identical across the arms. Each integration can add a capped result schema for its own tools, and those additions are published in the run metadata.
In the official run, the additional material was 549 bytes for fs_grep, 735 bytes for RE call, and 853 bytes for MemPalace. The earlier imbalance, where RE call carried the largest coaching surplus, was removed. The remaining differences are visible and measurable.

The corpus can now expose harm

The original task suite put the governing fact in the corpus every time. It could measure help, but it was structurally weak at measuring damage.

The current corpus has five conditions:

present: the governing fact is available and current
absent: the fact has been removed
superseded: an old version sits beside a newer version
contradictory: rival versions exist without an authority marker
adjacent: related information exists but does not apply to the task

The same executable checker is used in every condition. A memory system is not rewarded merely for finding text. It is evaluated on whether the completed code is correct.
This changes the interpretation of a positive result. A system that helps on present but damages work on superseded has a different risk profile from one that helps on present and stays neutral elsewhere.

The admission gate is part of the measurement

A cell is admitted only when every arm can demonstrate that its treatment was present and that the session was comparable. The gate checks the available tools, startup signals, sandbox identity, and treatment metadata.
If one arm fails to start, the cell is discarded rather than scored as a task failure. The discard count is published because this rule protects the comparison but can also favour an arm that fails in a way the gate detects.
The official run had eight arms, so it had more ways to lose a cell than the previous four arm run. It produced 2,920 sessions, 317 admitted cells, and 48 discarded cells. The admitted set is not automatically comparable with the earlier run's admitted set.

The artifacts are now independently checkable

The full official run publishes session records, compressed streams, admission decisions, costs, and endpoint results. The verifier recomputes the published values from those artifacts without credentials, a database, or model calls.
That work found defects in the benchmark itself. One older run stored its records beside the run directory under a sibling filename, so the verifier reported missing evidence even though the records existed. Another guard could not inspect compressed streams. A partition invariant assumed that every run used the same set of corpus conditions.
Those cases are now tested explicitly. The repository documents known missing streams as failures with explanations rather than silently turning them into passes. The replication guide explains what an outside runner must report and why a replication gets standing to contradict the official run rather than entering the leaderboard as another incomparable score.

The promotion and verification work is documented in pull request 70.

The official result

The first official run used 26 coding tasks, five corpus conditions, five seeds, eight arms, and the deepseek-v4-flash model. The leaderboard uses claude_md as its designated baseline.

Arm	Success rate	Difference from `claude_md`
placebo	67.2%	+9.5 points
recall	65.9%	+8.2 points
bare	65.9%	+8.2 points
`fs_grep`	63.1%	+5.4 points
MemPalace	60.6%	+2.8 points
`claude_md`	57.7%	baseline

The protocol and recall_prefetch tracks both scored 61.2%. They are reference tracks and are not product rows.

The raw numbers are in official 003 and its published summary.

How to interpret those numbers

The placebo being first does not prove that neutral prose is better than memory. It tells me that this run contains enough variance, task sensitivity, or uncontrolled influence that the ordering of the rates cannot be treated as a ranking.
RE call's 65.9% is above the claude_md baseline's 57.7%, but its interval crosses zero. The correct statement is that this configuration performed better descriptively in this run, while the run did not establish a statistically reliable product benefit.
RE call matching the bare rate is not evidence that memory and no memory are equivalent. It only says that this run did not separate them. The paired contrast and its interval are the evidence to inspect, not the order of the headline rates.
The protocol arm gives the coaching comparison. If protocol had substantially exceeded bare, I would have evidence that the instruction itself changes performance. If recall had then exceeded protocol, I could attribute the additional difference more plausibly to the memory surface. In this run, neither diagnostic track produces a clean positive claim.
The prefetch arm gives the retrieval comparison. A gap between prefetch and natural RE call would show that agent behaviour is limiting the benefit. A gap between prefetch and the baseline would show that the retrieval payload can help when the harness supplies it at the right time. Because the official result is a null at the product level, these tracks should be used to design the next experiment, not to rescue the leaderboard.
The previous article's +20 cell result at p = 0.015 was exploratory. The new official comparison against the protocol arm was +15 cells at p = 0.058. The direction was similar, but the threshold did not clear. That is exactly the kind of change a better control should reveal.

What it can measure now, and what it cannot

The benchmark can now measure task success under useful and hostile memory conditions, negative transfer relative to a bare agent, the contribution of shared memory coaching, the effect of giving an agent a simple searchable feed, natural retrieval behaviour versus prompt driven retrieval, treatment wiring, discard rates, token cost, wall time, and whether the published arithmetic follows from the published sessions.
It still cannot measure the write path. The corpus is bulk ingested before the grid and is not updated by the agent. Nothing here says whether a product extracts a good memory from a session, consolidates repeated information, forgets a correction, or preserves a decision over time.
The task suite also favours discrete retrieval. Most tasks place one governing fact in one transcript. Cross session synthesis tasks now exist, but they have not produced a scored result.
The run uses one model and one realization per task condition and seed. The memory arms are not budget matched, and RE call uses substantially more input tokens than the other arms. The official preregistration was committed after data collection began, so it is weaker evidence than a genuinely prospective test. Those limits are part of the result.
The next registered change adds a paired recall_rerank arm. It keeps the base arm in the same grid and changes the reranking configuration only, so it tests retrieval ordering without comparing two different runs. It is described in pull request 69, and it has no result yet.

The benchmark is now answering better questions

The useful improvement is not a more impressive leaderboard. It is the ability to tell apart:

no memory
memory coaching without memory
extra neutral context
simple file search
natural product use
retrieval supplied by the harness
exact evidence supplied as a ceiling

That makes a null more informative. When no arm wins cleanly, I know the benchmark did not merely lack a memory feature. I know which controls were present, which parts of the causal path were removed, which conditions could expose harm, and which artifacts a reader can inspect.

That is what the benchmark can measure now.

I Gave Claude Code a Brain. Then I Measured What It Costs to Ask for One.

Giulio D'Erme — Tue, 01 Sep 2026 15:57:06 +0000

At the end of part 2 I promised a clean competitor comparison: more than one memory system, the same admission gate, enough capacity to finish.

This is that run. 1,453 agent sessions, four arms, five corpus conditions, every outcome graded by executing a checker against the code the agent actually wrote. No LLM judge anywhere.

The corpus

Each condition is a 4,911 document corpus, about 27,000 embedded chunks. Of those documents, 4,902 are adversarial distractors generated from the corpus's own vocabulary to be confusable, and the rest are the planted material for 36 tasks.

That ratio is the point. On the small feed I used earlier, retrieval hit@10 was 1.000: everything was findable, so the benchmark could not tell "the product retrieved badly" from "the agent never searched". With the haystack, BM25 hit@1 drops from 0.485 to 0.182.

The five conditions differ in what is planted, and four of the five are traps:

condition	what the corpus holds	share of cells
`present`	the governing fact, cleanly	37%
`superseded`	the fact, beside an outdated version of itself	15%
`adjacent`	something related and inapplicable	16%
`absent`	the fact deleted, distractors left behind	17%
`contradictory`	two rival versions, no authority between them	15%

Only 37 percent of cells contain a clean fact worth retrieving. In the other 63 percent the corpus is outdated, contradictory, inapplicable, or empty, and a memory layer's job there is to not make things worse.

I built it that way deliberately. A benchmark where memory can only help is a demo.

The control that changed the question

Two commenters asked for a placebo. Fair: if you tell an agent "you have memory, consult it before acting" and it then does better, you have not separated the memory from the telling.

So I built an arm carrying the complete memory instruction and no memory at all. Same protocol text, byte for byte, as the two real arms. Nothing behind it.

comparison	wins	losses	net	p
instruction only, vs bare	26	43	-17	0.053

Telling an agent to consult memory, when there is no memory, makes it measurably worse. Seventeen cells of 358, borderline significance, in the direction nobody advertises.

That reframes everything. "Does memory beat no memory" is not directly answerable, because every memory product requires that instruction. The answerable question is: does the product earn back the cost of asking for it?

The comparison

RE-call 0.11.0 is mine, so read accordingly. MemPalace 3.8.0 is the better known of the two. Both received the same instruction and the same gate. Every write tool was withheld from every arm.

comparison	wins	losses	net	p
RE-call vs instruction only	41	21	+20	0.015
MemPalace vs instruction only	25	23	+2	0.885
RE-call vs MemPalace	48	30	+18	0.054
RE-call vs bare	47	44	+3	0.834
MemPalace vs bare	38	53	-15	0.142

McNemar exact on discordant pairs. The only comparison clearing p < 0.05 is RE-call against the instruction-only control.

Note the fourth row before anyone else does. Against a bare agent, RE-call is +3 cells at p = 0.834, which is nothing. The win is real against the instruction the product requires and invisible against an agent you never asked. A sceptic will quote that row and is entitled to.

Arm by arm

Tasks solved out of cells, admitted only where all four arms produced a record. Ordered by how much RE-call gains over the instruction-only arm, which is the comparison that isolates the product from the coaching.

condition	cells	bare	instruction	RE-call	MemPalace	RE-call gain
present	134	52/134 · 0.388	53/134 · 0.396	67/134 · 0.500	56/134 · 0.418	+14
superseded	52	38/52 · 0.731	34/52 · 0.654	41/52 · 0.788	35/52 · 0.673	+7
absent	60	49/60 · 0.817	41/60 · 0.683	44/60 · 0.733	37/60 · 0.617	+3
contradictory	54	44/54 · 0.815	41/54 · 0.759	39/54 · 0.722	42/54 · 0.778	-2
adjacent	58	42/58 · 0.724	39/58 · 0.672	37/58 · 0.638	40/58 · 0.690	-2
all	358	225/358 · 0.628	208/358 · 0.581	228/358 · 0.637	210/358 · 0.587	+20

RE-call is the only arm that beats doing nothing, by 0.9 points overall. MemPalace at 210/358 sits below the no-memory baseline and barely above instruction-only. MemPalace's gains against the same control are +3, +1, -4, +1, +1, totalling +2.

The shape of that table matters more than the total. Memory pays where there is something to recall and costs where there is not. On present RE-call converts 67 of 134 against bare's 52. On the three conditions built to punish retrieval it is flat or slightly negative, because surfacing something inapplicable is worse than surfacing nothing.

This is close to a worst case, and that is the point

Go back to the composition. Sixty-three percent of these cells are conditions built to punish a memory layer, where the corpus is outdated, contradictory, inapplicable, or empty and the best available outcome is to waste nothing. Only 37 percent contain a clean fact worth retrieving.

RE-call is still the only arm that finishes above the no-memory baseline. It does that while spending nearly two thirds of its cells in terrain designed to make retrieval a liability, and while paying the -17 cell entry fee that the instruction alone costs.

That is the number I would defend hardest, because it is the unflattering way to measure my own product and it still comes out positive.

And the composition is a choice I made, not a fact about anyone's repository. Real memory does not sit still at that ratio. How much of it is current, governing and correctly superseded is a property of how the layer writes and maintains notes over time, which this benchmark does not measure at all. It fixes the corpus and varies the product.

So read the table as a floor rather than a forecast. On the condition a well maintained corpus is mostly made of, RE-call is +14 of its +20. What that becomes at a healthier ratio I have not measured, and I am not going to put a number on it.

I would rather someone else did. A composition sweep, holding the product constant and varying the mix, is the obvious next experiment and it is not run.

The number I find most telling

For each product, the solved rate when it searched minus when it did not:

condition	RE-call	MemPalace
present	+0.181	+0.019
adjacent	+0.147	-0.032
contradictory	+0.079	+0.085
absent	+0.031	+0.264
superseded	+0.006	-0.094

When RE-call searches it helps in five conditions of five. When MemPalace searches it hurts in two, while searching more often (0.58 to 0.69 against 0.46 to 0.56).

On present, the condition built so the answer is in memory, searching is worth +0.181 to RE-call and +0.019 to MemPalace. That is the cell both products exist for.

Cost, median beside mean because these distributions are skewed enough that on an earlier run of mine the two disagreed in sign: RE-call 65.9 s and 56,476 input tokens per session, MemPalace 68.6 s and 90,706, bare 44.7 s and 14,085. RE-call is faster and uses 38 percent fewer tokens than MemPalace while delivering ten times the net benefit. Both are materially heavier than no memory, which is the honest cost of the idea.

What I had to publish against myself

The previous run said something different, and I want to be the one who says why.

In it, RE-call looked like the most damaging arm of seven, worst on three of four adversarial conditions. I had written that up as a genuine weakness of my own product.

It was my harness. That run gave RE-call 1,958 bytes of coaching over the shared protocol against MemPalace's 853, and most of the surplus was not about RE-call at all: search before your first write, search by symptom rather than goal. Generic advice that helps any retrieval system, handed to one of them. Worse, the flag selecting the fair variant was hardcoded in my launcher with no override, so no run I had ever done could have been matched, and I had not noticed.

Under the corrected instruction the finding evaporates. RE-call's damage-only net harm falls from +0.073 to +0.0122, identical to MemPalace's, and its superseded damage from 0.179 to 0.036. Its usefulness score also fell, 0.377 to 0.281, while MemPalace's did not move at all, because MemPalace had been on the shared protocol all along. My product's advantage narrowed from 1.8x to 1.35x once I stopped giving it a bigger prompt.

Both corrections point the same way. I had been generous to myself and unfair to the competitor, and it produced one flattering finding and one damning one, both wrong.

What this benchmark cannot see

RE-call has a reasoning-graph layer. Across 2,181 sessions of the earlier run its four graph tools were called zero times. Read alone, damning.

It builds that graph from authored cross-references, the way a wiki links notes. Its own memory store has 44,364 of them. This benchmark's corpus is rendered agent transcripts: conversation turns, no cross-references. The graph gets entities and no edges. It is a node set, not a graph.

So graph reasoning needs an authored, cross-referenced corpus and this benchmark deliberately measures raw transcripts. Different corpus classes, not a weakness. Manufacturing cross-references in to make the feature look good would be assuming the answer. I established that with ten read-only queries rather than the 220 sessions the experiment would have cost, after a reviewer pointed out my draft design would have measured a language model reading trusted evidence and published it as a graph result.

Come and break it

Two systems is a comparison, not a field, and one of them is mine. The harness is public: github.com/GiulioDER/agent-memory-bench.

If you build a memory layer for coding agents, I would rather measure yours than have you take my word for this. Every adapter is hash-pinned and every vendor is invited to review their own config before a measured run, which is what adapters/<name>/VENDOR_REVIEW.md is for. Open an issue and I will help wire it up.

And if you just want to check my arithmetic, the README has a section on verifying a published run with no credentials and no money. The preregistrations are in the repo, including the ones whose predictions I got wrong, and the raw records are published with each run.

If you take one number from this, take that one: 63 percent of the cells were built to make memory a liability, and the memory layer still finished ahead of not having one.

One thing I would most like broken: the 45 percent. On superseded, the largest single loss is that in 45 percent of sessions neither planted document is retrieved at all. That is worth roughly three times more than anything I have measured on the ranking side, it is a query-formulation problem rather than a retrieval-quality one, and it is already preregistered. If someone has a better idea than mine, I would like to lose that one.

The method, the leaderboard and the submission rules now live at giulioder.github.io/agent-memory-bench

ghttps://huggingface.co/spaces/Gde05/agent-memory-bench.

The #1 row on this AI memory leaderboard is not a measurement

Giulio D'Erme — Sat, 29 Aug 2026 15:00:01 +0000

Bench'd (benchd.ai) calls itself the neutral benchmark authority for AI memory, and sells vendors a verification badge from $299 to $3,999.99 a month.

I ran my memory system through their harness. Then I checked their board. Every number below is from their own site and repository, read on 2026-08-29, and every one takes seconds to verify.

The numbers are fake

Their three track leaders, recomputed against the harness's own aggregation rule:

Track	#1	Score	What its own dimensions say
Knowledge Brain	gbrain	100.0	perfect recall, reliability score 4.0
Agent Memory	Letta	80.0	own dimensions average 26.7
Conversational Memory	LangMem	60.0	dimensions are all zero

A 100.0 beside a reliability score of 4.0 is the signature of an adapter echoing expected answers. An 80.0 built from dimensions averaging 26.7 is an aggregation bug. A 60.0 from all-zero dimensions is not a number.

None of the three has a manifest anyone can download. And per Bench'd's own methodology page, rows marked Community-Verified are "independently run" by Bench'd. All three are. So these are not vendor claims. Letta did not submit an 80.0. gbrain did not submit a 100.0. The vendors named in those rows are being misrepresented by the harness, not caught by it.

Filed as [issue #5 (https://github.com/benchdai/harness/issues/5) on 2026 08-23. Still on the board.

The trust is fake

Their methodology page has a section headed Verify a Receipt Yourself: "You don't need to trust us. Every receipt can be independently verified." It gives you this:

```bash curl -sL https://benchd.dev/api/receipt/run_abc123.json -o receipt.json




`benchd.dev` does not exist. NXDOMAIN against 1.1.1.1, against 8.8.8.8, and against my local resolver. `benchd.ai` resolves fine.



```console $ nslookup benchd.dev 1.1.1.1 *** one.one.one.one can't find benchd.dev: Non-existent domain

The entire independent-verification story terminates at a domain that has never existed. Nobody has ever verified a Bench'd receipt using Bench'd's published method, because there has never been a host to fetch one from. That check took four seconds.

Two more, quickly. Their trust page defines authenticity as a signature from "a key published on this page"; the key signing their own shipped manifests matches neither published fingerprint. And their trust page promises they take no payment from vendors on the leaderboard, while their pricing page sells those vendors a leaderboard badge. An organisation that ranks vendors cannot also invoice them.

Nobody is home

The harness repo, the entire basis of the "independent, reproducible, open" claim: no commit since 2026-06-06. Eighty-four days.
Five open issues, from two different vendors. Every comment on all five is from the person who filed it. Dustin-a11y has been waiting since 14 July.
Their submission endpoint caps uploads at ~4.5 MB, so no full trace manifest fits. Not even their own 16 MB manifests fit.

Submit real traces, get rejected by an unmaintained upload cap. Submit nothing, be Community-Verified at 100.0.

Meanwhile the board publishes receipts dated two days ago, and the pricing page keeps selling.

The control finished third

Bench'd runs a control: GPT-4o-mini with no memory attached. Conversational Memory track, 2026-08-29:

#	System	Score
1	LangMem	60.0
2	LlamaIndex Memory	59.0
3	no memory at all	57.6
4	LangChain Memory	34.0
5	Mem0	32.4

A benchmark whose control finishes third is not measuring the thing it is named after.

This is the industry, not one website

Bench'd is downstream of a worse problem: the numbers it aggregates come from READMEs, and the READMEs do not survive checking.

Over the past months I have checked the benchmark claims of several memory projects and have not yet found one that held up. A published 96.6% recall that becomes 87.0% when rescored with the benchmark's own scorer, because the field reports recall_any where the benchmark specifies recall_all. A "clean held-out" set built after the tuning it was meant to hold out. LoCoMo ground truth with roughly 99 wrong or misattributed answers, per @dial481's audit, and a judge accepting ~63% of deliberately wrong answers. Session-level LoCoMo recall quoted without mentioning that a random ranker scores 0.371 at the depths everyone reports.

None of that is fraud. It is a field where the number goes in the README, the arithmetic goes nowhere, and nobody is expected to check. Every project that publishes a soft metric under a hard name makes it more rational for the next one to do the same. The honest ones lose ground for being honest, which is the part that actually disrupts the market.

What I measured, briefly

RE-call, through their harness, unmodified scoring path, on 2026-08-23: 69.0 LongMemEval, 71.6 LoCoMo, $6.64 total spend. The no-memory control is 57.6, so the number that means anything is +11.4 over having no memory. I am giving you the control so you can compute that instead of trusting a ranking.

Two disclosures, since I am demanding them from everyone else. I set RE-call's abstention threshold to zero: every question is answerable by construction and the judge scores "insufficient information" as wrong, so honest abstention forfeits 26.7 points, and refusing when nothing clears the threshold is the whole premise of my product. And I compare only within the Conversational Memory track.

pip install benchd-harness
benchd verify longmemeval-v1.manifest.signed.json

Signed manifests, run records and SHA256SUMS: benchd official-2026-08-23.
Pre-registered before measurement, wrong predictions left standing.

Over to you

Every check above took minutes. The DNS one took four seconds. If the checks are that cheap and keep coming back positive, the problem is not difficulty, it is that nobody is expected to look.

Three questions, and I will answer every reply:

Is this just normal now? Have you ever gone to verify a benchmark number in a README and found it did not hold? I want to know whether my hit rate is bad luck or the base rate, because from where I am standing it looks like the base rate.
What do you do when it is your project on the losing end? You ship a real number. Someone ships a bigger one that nobody checks, and takes the users, the stars and the funding with it. Do you match them and publish your softest metric, spend your time auditing theirs, or eat it and keep building? I have picked one of those three and I am not sure it was right.
What actually gets an unreal number corrected? This is the one I most want answers to. A filed issue gets ignored. A polite email gets ignored. A public writeup costs you a week and makes you look like a competitor with an axe. What has worked for you: reproduction repos, pressure from the benchmark's own authors, a shared audit registry, something I have not thought of? Not how to attack anyone, how to make the correction actually land, because everything I have tried so far has mostly not.

If any line here is wrong, name it and I will correct it publicly, same prominence, same day. I put that offer in their issue tracker before publishing rather than after.

I Gave Claude Code a Brain. This Time I Let the Tests Judge It.

Giulio D'Erme — Mon, 24 Aug 2026 16:06:44 +0000

The first experiment measured whether Claude Code knew the right answer. This one measured whether the work actually succeeded.

In my previous experiment, I tested whether a memory layer could stop Claude Code from confidently repeating project-specific mistakes.

It did.

But there was an obvious limitation.

The tasks measured whether the answer was correct. They did not ask the agent to modify a repository, run the result, and pass a deterministic test.

So I ran the harder version.

This time, the agent had to do real work in a temporary repository. A checker decided whether the result was correct. No judge model. No subjective scoring. The task either passed or it did not.

Here is the result from the complete run:

Configuration	Successful runs
Claude Code without memory	50.0%
Claude Code with `CLAUDE.md`	36.1%
Claude Code with RE-call	58.3%

RE-call improved success over the CLAUDE.md baseline by 22.2 percentage points.

The 95% confidence interval was +11.1 to +33.3 points.

The paired McNemar test gave a p-value of 0.000145.

This was not one lucky demonstration. It was 72 paired comparisons across 24 tasks and three seeds, with zero discarded cells.

What changed from the first experiment?

Each task asked Claude Code to create or modify files in a repository and run the result.

The benchmark included tasks involving things like:

deterministic file ordering
Unicode normalization
migration naming
shell quoting
money rounding
stable sorting
line ending handling
generated files and golden outputs

The important detail is that every task had a deterministic checker.

The agent could say "done" all it wanted. The checker only cared whether the repository ended up in the correct state.

I ran every task through three configurations:

Bare Claude Code
Claude Code with the project's CLAUDE.md
Claude Code with the same environment plus RE-call memory

The harness also verified that the memory tools were actually available before counting a session. If the memory layer was missing, the entire paired cell was discarded.

That happened zero times in the DeepSeek run.

RE-call helped where project memory mattered

The most useful comparison is RE-call versus the static CLAUDE.md file.

The static file produced a 36.1% success rate.

RE-call produced 58.3%.

That is a 22.2 point improvement.

The paired results were even more revealing:

RE-call won 17 comparisons that CLAUDE.md lost
CLAUDE.md won only 1 comparison that RE-call lost

The effect was strongest on the tasks classified as memory-sensitive. Across those eight tasks, RE-call improved success by 45.8 percentage points.

That is the pattern I was looking for.

Memory did not make every task easier. It helped when the task depended on a project decision, convention, or failure mode that was not fully present in the immediate prompt.

The surprising result was `CLAUDE.md`

The static CLAUDE.md configuration performed worse than the bare configuration:

Configuration	Successful runs
Bare	50.0%
`CLAUDE.md`	36.1%

The bare configuration won by 13.9 percentage points.

This is not what I expected.

It suggests that a static instruction file is not automatically a memory system. It can add useful guidance, but it can also add noise, constraints, or instructions that compete with the task.

RE-call is doing something different.

Instead of putting every historical fact into the permanent prompt, it gives the agent a way to retrieve relevant project history when the current task needs it.

That distinction matters.

Memory did not just improve the final score

The benchmark also measured what happened inside the memory mechanism.

In the DeepSeek run:

The agent searched memory in 83.3% of eligible sessions.
When it searched, it reached useful context 85.0% of the time.
Overall, useful project context was reached in 70.8% of sessions.

So the result is not simply that "the model had more tokens."

The memory layer was being consulted, and in most cases where it was consulted, it reached something useful.

There is still room to improve retrieval. A search is not automatically a good search. The model needs to find the right historical decision, not merely retrieve something related to the words in the prompt.

The cost

The complete DeepSeek run cost an estimated $0.4964 at the captured API prices.

The three arms cost:

Configuration	Estimated cost
Bare	$0.0824
`CLAUDE.md`	$0.0863
RE-call	$0.3277

RE-call used about four times as many total tokens as the static prompt configuration.

That is the tradeoff.

The memory layer costs more on every task, including tasks where memory is unnecessary. In exchange, it can prevent the much more expensive failure mode where an agent confidently follows an outdated convention, repeats a known mistake, or spends several minutes debugging a problem the team already solved.

The next engineering question is not whether memory helps.

It is how to make the retrieval smaller, faster, and more selective.

What about GPT-5.3 Codex?

I also started the same model-freeze run with GPT-5.3 Codex.

That run is not a negative result for GPT.

The provider hit a credit and in-flight request limit while the benchmark was running. The harness correctly discarded the affected paired cells instead of treating missing sessions as failed model attempts.

Only 40 of the 72 paired cells were admissible, so I am not using that run to claim that GPT-5.3 Codex is better or worse.

The descriptive numbers pointed in the same direction:

Configuration	Successful runs
Bare	47.5%
`CLAUDE.md`	30.0%
RE-call	50.0%

But the run needs to be repeated with the provider capacity issue fixed before it becomes a valid model comparison.

That is an operational failure, not a model conclusion.

What this proves

This experiment does not show that memory makes an agent generally smarter.

It shows something narrower and more useful:

A production memory layer can improve the probability that an agent completes a real repository task correctly.

In this run, RE-call improved success by 22.2 percentage points over the static project file.

It also improved success over the bare configuration, although that gain was smaller at 8.3 points. That difference is important. The main value is not simply adding more instructions. The value is retrieving the right project-specific context when it matters.

The first experiment showed that memory can make answers more correct.

This experiment showed that the improvement survives contact with the repository.

That is the result I was actually waiting for.

What comes next

The next step is a clean competitor comparison:

rerun GPT-5.3 Codex with sufficient provider capacity
compare RE-call against other memory systems
keep the same tasks, seeds, checkers, and admission gate
publish discarded cells and operational failures alongside the scores

The goal is not to produce another flattering demo.

The goal is to find out whether a memory layer remains useful when the models, vendors, and retrieval systems change.

That is the benchmark I want to run next.

GiulioDER / RE-call

Memory that abstains instead of guessing: agent memory on your own Postgres with a verdict, confidence and provenance on every hit, and a calibrated refusal when nothing clears the threshold.

Memory that abstains instead of guessing.
RE-call is agent memory on your own PostgreSQL with pgvector: every hit carries a verdict confidence and provenance, a retracted claim comes back marked superseded, and a question the corpus cannot answer is refused rather than answered from the nearest neighbour

ATM-Bench Recall@10 92.8924 against 79.09 for the best published row (limits) · second of ten on MTRAG correct refusals (limits) · zero memory-layer LLM calls to build memory, where Mem0 pays one per session (limits)

Why RE-call · Quickstart · How it works · Product surface · Documentation · Evidence

Setup guide: install, configure and run RE-call · Validity Frontmatter: the open spec RE-call implements

Why RE-call

Nearest-match retrieval cannot tell the difference between what is true and what merely reads like it. When a corpus keeps its history, and real agent memory does, the retracted…

View on GitHub

I Gave Claude Code a Brain. Its Mistake Rate Went to Zero.

Giulio D'Erme — Fri, 21 Aug 2026 20:43:21 +0000

40 tasks. Without memory it got them wrong 52.5% of the time. With memory: 0%.

Not "fewer mistakes". Not "improved by 30%". Zero. Forty out of forty.

Here is the entire result before I explain anything:

	Claude Code	Claude Code + RE-call
Got it wrong	52.5% of runs	0%
Answer correctness	0.144	0.358
Factual correctness	0.288	0.559
Times memory made things worse		0 out of 40

Correctness doubled. Factual accuracy doubled. And across 40 head to head runs, the version with memory never once lost a round the other one won.

The p values are 0.0003 and below. In plain language: this is not luck.

What that actually buys you

Numbers are nice. Here is what they mean on a Tuesday afternoon.

Your agent stops confidently recommending commands that do not work. Not commands that error out, because you would catch those. Commands that succeed, print something reassuring, and do nothing at all. Those are the expensive ones, precisely because you believe them.

Your agent knows the decisions your team already made. Why that library version is pinned. Why that directory is gitignored. Which flag broke production in March. The knowledge that lives in people's heads, old pull requests, and nowhere the model can see.

Answers about your repository get twice as correct. Not longer, not vaguer. More right.

It never makes anything worse. Zero regressions across 40 paired runs. That is the number I would want to see before letting anything near my workflow.

Why I did not just publish another benchmark

I have run RE-call through plenty of them. Retrieval benchmarks, long context benchmarks, answer quality benchmarks. They are genuinely useful and I keep running them, because a benchmark tells you where your system is weak, which knob actually matters, and whether last week's idea was clever or merely expensive.

But a retrieval benchmark proves your retriever retrieves. It does not prove anyone is better off.

You can top a leaderboard and change nothing about a single developer's day. That gap bothered me enough to go and measure the day instead.

How the test was built

The whole design fits in one sentence: run the same task through two Claude Code sessions that differ by exactly one thing, and count what happens.

The two sessions. Both ran the same model. Both got the same tasks. Both loaded the same CLAUDE.md project file, byte for byte, and I made the harness verify that at the start of every run rather than take my word for it. Both ran with hooks, plugins and auto loaded context switched off, so nothing could sneak in a hidden advantage. The only difference: one session could also search a curated memory of the project, and the other could not.

The tasks. Ten tasks, each hiding a hazard this project ran into once, paid for, and wrote down. A CPU cap that reports success and caps nothing. An image renderer that does not exist on this machine. A library install that quietly breaks the environment. Ordinary requests, each with a booby trap that only bites if you do not know the history.

The scoring for hazards. Each hazard has a deterministic checker that reads the session transcript and decides whether the agent walked into it. A pattern, not an opinion. No model gets a vote on this part.

The repetitions. Each hazard task ran ten times per arm, because agents are not deterministic and one run of anything proves nothing. Sixty paired runs in total.

The honesty gate. Before any pair counts, the harness checks that the memory tools were actually present in the memory session and actually absent in the other one. If either is wrong, the pair is thrown away rather than scored.

The predictions. I wrote down what I expected before running anything, and committed it with a timestamp. That is the only reason I can tell you honestly which results surprised me, rather than building a story around whatever came out.

The statistics. Every comparison is paired, same task through both arms, so the test is on the difference within each pair rather than two independent piles of numbers. Hazards get an exact McNemar test, which looks only at pairs where the two arms disagreed. Costs get a Wilcoxon signed rank test, which does not assume the data is nicely shaped, because agent token counts absolutely are not.

How I measured answer quality, with Ragas

Counting hazards is the easy half. "Was the answer actually any good" is where most demos quietly stop and switch to screenshots.

For that I used Ragas, an open source evaluation toolkit for LLM applications. It exists to turn "that looks about right" into a number, which is exactly the problem I had.

The way it works is refreshingly boring. You write down the correct answer in advance. Ragas then breaks the agent's actual response into individual claims and checks each one against your reference, so you find out how much of what it said was right, and how much was confidently wrong.

Two metrics from their library did the work:

AnswerCorrectness, which scores how much of the reference answer the response actually got right. I configured it with weights=[1.0, 0.0], which turns off the embedding similarity half. That half rewards an answer for sounding like the reference, and I did not want to reward paraphrasing. Only factual overlap counts.
FactualCorrectness, which compares the claims in the response against the claims in the reference and scores the agreement.

Three things kept it honest:

The judge is a different model family from the agent under test. Claude Code did the work; a GPT model did the grading. Nothing marked its own homework.
Every reference answer was written before the runs. Nobody got to decide what "correct" meant after seeing the output.
Every failed judge call is recorded, not dropped. That matters more than it sounds. On my first attempt, five of six scores silently failed because the judge hit a token limit while decomposing long answers. Because failures were recorded rather than skipped, I caught it. Had they been skipped, I would have published a quality number computed from one sixth of the run with a perfectly plausible figure on top.

Final tally: 120 responses scored, zero failures.

The control that makes the rest believable

This is the part product demos skip, and the part I would look for first as a reader.

I also ran tasks where the answer was already sitting in CLAUDE.md. Both agents had that file. So memory should add precisely nothing.

Result: 0.083 against 0.083. A difference of exactly zero. p = 1.0.

If memory had "won" there too, my experiment would be broken and this article would not exist. The gain appears exactly where memory holds something the file does not, and vanishes everywhere else. That is what a real effect looks like, as opposed to a flattering one.

The finding I did not expect

The first time I ran this, I did the clever thing. I gave the agent RE call instead of CLAUDE.md. Retrieval replaces your documentation. That is the entire pitch of RAG.Answer quality improved by 0.044, with a p value of 0.43. Translation: nothing happened.Then I ran it the way an actual developer would set it up. Keep your CLAUDE.md. Add memory on top.

Quality nearly doubled.

Replacing your documentation with retrieval does nothing. Adding retrieval to it changes everything.

I had written down the opposite prediction before running a single session, which is the only reason I can prove I was surprised rather than inventing the narrative afterwards.

What it costs

About 15,000 extra input tokens per session, and a couple of seconds. On tasks where memory cannot possibly help, it runs slightly slower, because it goes and looks something up and finds nothing useful.

That is the trade: pay a little on every task, stop paying enormously on the ones that would have gone wrong. Roughly once in twenty runs, the agent without memory disappeared down a hole for seven minutes chasing something that had been written down all along.

What this does not prove yet

I would rather say this myself than have someone say it for me.

This shows memory eliminates a specific, expensive class of mistake and makes answers about your project substantially more correct. It does not show memory makes an agent generally smarter. Where memory has nothing to add, it adds nothing, and I have the control numbers proving both halves.

It also does not yet show that the work succeeds more often. Every task here asks for a recommendation and scores whether that recommendation is right. None of them writes code, runs a test suite, or ships anything.

Part 2 coming soon

The next round is already running, and it asks the harder question: does the work actually succeed?

Real tasks, in a real repository, where a test suite decides the outcome. No judge, no reference answer, no interpretation. It passes or it does not. Alongside it, more measurements on where the gains come from: how often memory gets consulted, how often it returns the fact that matters, and which kinds of task benefit most.

That is the number I actually want, and I will publish it whichever way it falls.

Part 2 coming soon.

The RAG benchmark that pays you for saying "I don't know", and what it exposed

Giulio D'Erme — Sun, 09 Aug 2026 11:33:16 +0000

Nine published systems. One of them refuses 2.3× more often than gpt-4o and still finishes 4th on answer quality. It calls no LLM.

Part 5 of The Answerability Problem. Parts 1 to 4 argued the field measures accuracy on questions that have answers. This one is about the benchmark that doesn't. Code: RE-call (Apache 2.0).

MTRAG (IBM, TACL 2025) is 842 human-written multi-turn tasks with an official gpt-4o-mini judge that pays a full 1.0 on every metric for correctly saying "I do not have that information."

Almost no other benchmark scores a refusal as anything but a miss. So this is the one board where the thing my system is built for is actually worth points.

Here is what nine published systems do with that.

Nobody is collecting the free money

Correct refusals on the 55 unanswerable tasks. Same judge, same tasks, every system:

#	system	correct refusals
1	llama-3.1-8b	18/55	32.7%
2	🔹 `gpt-4o` + RE-call	16/55	29.1%
3	llama-3.1-70b	16/55	29.1%
4	gpt-4o-mini	13/55	23.6%
6	gpt-4o (benchmark's own retrieval)	7/55	12.7%
7	llama-3.1-405b	3/55	5.5%
9	qwen-2.5-72b	1/55	1.8%
10	mixtral-8x22b	0/55	0.0%

Two rows in that table read the same document set with the same model. Row 6 is gpt-4o on the benchmark's own retrieval. Row 2 is gpt-4o on RE-call's. Same reader, same prompt, same judge, only the contexts differ, and the refusal rate goes from 7 to 16 out of 55.

Then read the rank order. It is close to inverted against model size. The 8B model refuses best. The 405B model refuses 5.5% of the time. One model refuses nothing at all. Scale does not buy this, and on this evidence it costs it.

RE-call is second of ten, tied exactly with llama-3.1-70b at 16 of 55, and it refuses 2.3× more often than gpt-4o and 16× more often than qwen-2.5-72b.

The usual objection, answered

Abstention is cheap if you don't care about answering. So here is what it cost, on the same 842 tasks:

#	system	answer quality
1	llama-3.1-405b	0.5691
2	qwen-2.5-72b	0.5625
3	gpt-4o (benchmark's own retrieval)	0.5591
4	🔹 `gpt-4o` + RE-call	0.5527
5	c4ai-command-r-plus	0.5502

RE-call is the only system in that top four that is also top two on abstention. The three above it refuse 5.5%, 1.8% and 12.7%. The gap to gpt-4o is 0.0064.

That is the whole claim, and it is a narrow one: not "best", but the only one on the board that is near the top of both columns at once. Everyone else picked a column.

Same two rows again, 3 and 4, same reader. Swapping the benchmark's own retrieval for RE-call's costs 0.0064 on answer quality and more than doubles what the system correctly refuses. That is the trade, priced.

And against the benchmark's own retrieval directly, identical generator, prompt and judge, only the contexts swapped, RE-call is ahead: 0.5527 against 0.5516.

Small. Positive under both prompts I tested. I am not going to inflate it.

(Every row above is scored by MTRAG's official gpt-4o-mini-2024-07-18 judge, hard-coded by the benchmark and applied identically to my rows and theirs. The reader on my rows is gpt-4o, the same model as the row I am compared against.)

One engine, whatever you can afford

The other half of this. The same pipeline runs at both ends of the price range, and every rung is a named flag with a measured price, not a tier you buy.

configuration	nDCG@5	cost
🔹 SPLADE learned sparse (the free default)	0.3573	local, $0
+ Voyage rerank (one flag)	0.4342	paid API

A 48% relative span between two flags, measured on 777 judged queries.

The reranker is worth +0.0769 nDCG@5 and is worse on 162 of the 777. That is exactly why it is off by default. I would rather ship a flag with two published numbers than a default with a marketing claim.

Everything dashed and amber below is opt-in and switched on by name. The solid path is what runs if you change nothing, and it costs $0 at any scale, because there is no LLM anywhere in it:

Writing a memory is a local embedding. No LLM call, so the marginal cost of a memory is $0 and stays $0 at any scale, on any model, forever. There is no upstream pricing change that can alter that.

Air-gapped and fully local, or cloud embedder and reranker when a jargon-heavy corpus makes it worth paying. Same engine, same guarantee at every setting: superseded or expired memories are demoted rather than served, and the confidence threshold is fitted per embedder and corpus at install time, never a shipped constant.

The number I nearly published

One thing before you trust any of the above.

An earlier version of this run had RE-call beating every baseline, including llama-3.1-405b. It was an artifact and it was one edit away from going out.

The official scorer reads the answerability label in lower case. The release files ship it capitalised. So the conditioning that pays 1.0 for a correct refusal silently never ran on my data: 2,526 lookups, zero label matches. My metrics were raw, the published baselines' were conditioned, and I was comparing two different quantities in exactly the place where my system is strongest.

Both sides were individually correct. The error lived entirely in the join.

Two signals should have caught it earlier and I explained both away. I had even written the memo describing the two-key problem, that same day, and never connected it. Having the fact is not the same as applying it.

Corrected, the honest reading is: RE-call at 0.5527 against gpt-4o's 0.5591, so −0.0064. Fourth, not first. The abstention result survives intact, because it was measured by the official judge rather than by the string-matching detector that had earlier inflated it from 29% to 43.6%.

The anomaly you can explain is still an anomaly, and the one that flatters you is the one to chase hardest.

So which rung is yours

Take the free default if the marginal cost of a memory has to be zero, your data cannot leave, or you need it to work offline. 0.3573, local, $0, no API key.

Add the reranker when a human is waiting. One second is invisible next to the generator call that follows it, and it is the largest single gain in the project. Check it on your corpus first: it lost on 162 of 777 queries here.

Use something else if abstention is worth nothing to you and you only need the top of one column. Three systems beat me on answer quality. All three of them will confidently answer a question whose answer is not in the corpus, between 87% and 98% of the time.

That is the trade. A system that invents a plausible answer and one that says nothing are not two points on one scale.

Every number, the six generation runs behind them, the correction in full, and the scoring bug I reported upstream: docs/MTRAG_BENCHMARK.md. RE-call is Apache 2.0, pip install recall-rag.

Make your AI assistant predict the result before it runs it

Giulio D'Erme — Sun, 02 Aug 2026 14:20:27 +0000

Less technical than my usual posts. Nothing to install, nothing to pay for, and it works the same whether you're using an AI assistant for code, spreadsheets, research or writing.

Here is the shape of the problem I want to describe. See if you recognise it.

You ask your assistant to do something that produces a result. It does. A number, a summary, a status, an answer comes back. It's plausible. You ask why, and you get a clear, confident explanation that makes complete sense of it. So you believe it, you write it down, and you build your next three decisions on top of it.

A week later you find out the whole thing was wrong. Not wrong in an obvious way. Wrong because the tool was measuring something you didn't ask about, or reading a file that was empty, or answering a slightly different question than the one in your head.

Notice what didn't happen there. Nothing was invented. No fake citation, no made-up fact, none of the failures people mean when they say "hallucination". The failure was structural: nobody committed to what the answer should be while it was still possible to be wrong.

The habit

Before your assistant runs anything that produces a result, make it answer two questions.

1. What do you expect the result to be, and why?

2. How would you know if this were broken?

That's the whole technique. Two sentences before the work instead of a paragraph of explanation after it. It works on the free tier. It works on any model. It costs a few seconds.

Why the first question works

This isn't about "thinking harder", and it isn't a motivational trick. It changes the kind of problem your assistant is solving.

Explaining a result you've already seen is an open-ended task. There are a huge number of stories that fit any given outcome, and a language model is exceptionally good at finding one of them. That's not a flaw, it's the thing it's built to do. The problem is that a fluent explanation tells you nothing about whether the result is correct. It reads identical either way.

Predicting a result you haven't seen is a closed task. Everything the assistant believes about your data, your files and your request has to collapse into one statement that can turn out to be wrong. And when the real output disagrees with the prediction, you get a signal that could not have been manufactured after the fact, because the prediction is already sitting there on the screen.

That gap between expected and actual is the entire product. It's cheap to produce and impossible to fake in hindsight.

There's a bonus effect, and it's the reason results feel steadier once you do this. Predicting first pins down the standard of success before the data arrives. Most of the wobble between one run and the next isn't the model changing its mind, it's the standard quietly moving to fit whatever showed up.

Science has a name for this and an entire apparatus around it: pre-registration. Nobody needed to invent it for AI. We just forgot to apply it.

Why the second question matters more

I learned this one the expensive way.

I had a system meant to answer time-sensitive questions: which of these documents was true as of a particular date. It scored badly. I had predicted it would score badly, because the task is genuinely hard. Prediction matched result. I nodded, wrote the number down, and moved on.

The prediction was right and the measurement was worthless.

None of the documents had dates attached to them. Not one, out of four hundred and twenty. The system was being asked a question about time using material that had no time in it. When I fixed that, with exactly the same documents, the score went from 0.19 to 0.98.

I want to be careful with that number, because it's the kind of number people quote badly, including the person who produced it. That is not an improvement. Nothing got smarter. That's what it looks like from the outside when a broken instrument gets repaired, and if I'd published it as a win I'd have been telling a lie with correct arithmetic in it.

Here's the general rule, and it's the part almost everyone skips:

Predicting the answer will not tell you the question was never asked. If the setup is broken, your prediction and your result can be wrong in the same direction, agree with each other perfectly, and confirm nothing.

Which is exactly what the second question is for. "How would I know if this were broken?" forces something different out of the assistant: not a guess at the answer, but a statement about the machinery, one that has to hold true whatever the answer turns out to be.

Expectation: "the score will be low, this is a hard problem."
Broken-check: "if the documents don't have dates on them, the question isn't being asked at all."

The first one was right and taught me nothing. The second one would have gone off on day one.

Done is not the same as proved

That second question generalises into something I now say out loud several times a week.

When a task reports success, what you have learned is that a task reported success. You have not learned that a file was written, that the file has anything in it, that the numbers cover the dates you asked for, or that the check you requested was actually performed.

My favourite example is small and stupid. I once wrote a rule-checker to catch a specific kind of mistake in my own work. It ran clean for weeks. It ran clean because of a settings mismatch that meant the rule I cared about was never switched on. The checker was structurally incapable of finding anything. Nothing was broken, nothing errored, everything was green, and the green meant nothing at all.

The first test I wrote to fix that read the settings file to confirm the settings file. It passed. Two separate reviews walked straight past it. The thing that finally caught anything was running the checker against a deliberately bad example and watching it complain.

So: ask for the output, not the status. Show me the rows. Show me the file. Show me it catching one.

Anything that's supposed to warn you, make it warn you once

Same idea, pointed somewhere useful.

A safety net you have never seen catch anything is not a safety net. It's a guess in the shape of one. This applies to far more than code: the alert that's supposed to email you when a number goes out of range, the filter that's supposed to catch the bad rows, the check that's supposed to stop the bad export.

So test it deliberately. Feed it the bad thing on purpose, once, and watch it fire.

I ran that exercise across a system I'd been maintaining for months. It found twelve warnings that could never have gone off under any circumstances. One of them was the reason a cost had been quietly running for about two months before anyone noticed. Every one of those twelve had been read and approved by someone, including me. Reading them was never going to work, because they read correctly. That's precisely what makes them dangerous.

Two things I got wrong before I got them right, and they generalise:

Breaking it on purpose only proves whatever you actually broke. I once "confirmed" that two checks happened in the right order by deleting one of them entirely. All that proved was that a check existed. The ordering, which was the actual thing I cared about, stayed untested.

A check that can't pass is as useless as one that can't fail. I hit five in a row where the thing also refused to go green on a known-good input, and each one had been introduced while fixing the previous one. Identical behaviour on the good case and the bad case means it's measuring nothing, in either direction.

Look it up, don't remember it

One more, cheaper than everything above. When the assistant needs a fact it could go and check, make it go and check. A confident wrong value is worse than an admitted blank, because a blank is honest and gets dealt with, while a wrong value slides silently into everything downstream.

And the part that took me embarrassingly long to accept: "it isn't there" is a complete answer. It does not need to be improved into something that looks more like information.

The version you can paste in right now

All of it collapses into five lines. Put them in your assistant's instructions file, your saved prompt, your project settings, wherever your tool keeps standing rules:

- Before running anything that produces a result: say what you expect
  and why, in one or two sentences. Then run it.
- Also say how I'd know if this were broken, and check that first.
- "Done" is not a result. Show me the output: the file, the rows, the value.
- Anything meant to warn me gets tested once by making it warn me.
- Look facts up instead of recalling them. "Not there" is a valid answer.

If five lines is four too many, keep the second one. It's the one that has paid for itself the most times.

Can you actually prove this helps?

Not the way I'd like to. I can't show you that it produces fewer invented facts, because I haven't run that experiment and I'm not going to claim a result I don't have.

But that isn't really the thing that hurts, is it. What hurts is the week you spend building on top of an answer that was wrong from the start. And that one is measurable, by you, this month, for free.

Keep a tally. Every time you have to walk something back, throw work away, or redo a decision because the result underneath it turned out to be wrong, put a mark down. One line, one date, one sentence on what the wrong thing was.

Do that for two weeks as you work now. Then add the two questions and do it for two more.

You're not counting how often the assistant sounds right, which is close to always and tells you nothing. You're counting how often it stayed right long enough to build on. That's the number that maps onto your actual time, and unlike a benchmark score it's yours, it's specific to your work, and nobody can argue with it.

My honest position: I have a mechanism I find convincing and an incident log where the disasters cluster hard on the occasions when nobody committed to anything in advance. That's evidence. It isn't a measurement, and I'd be doing the exact thing this post argues against if I dressed it up as one.

Where it doesn't help

Three limits, because I'd rather say them than have them said back to me.

It needs something checkable. If there's no observable outcome, "predict the result" turns into a confident-sounding preamble that flatters whatever comes next. When there's nothing to be wrong about, skip it.

A confident prediction can pull you toward it. Say a number out loud beforehand and there's pressure, on a model as much as on a person, to read the result as agreeing with it. The fix is to predict the observable, not the conclusion. "There should be roughly 400 rows, all dated after June" pulls much less than "this should confirm my theory."

It slows you down slightly. A few seconds per task. You get them back the first time it catches something, and you'll know exactly when that happens, because the prediction will be sitting right there next to the result, disagreeing with it.

If you try the five lines, I'd like to know which ones earn their place and which are cargo cult. I'm fairly sure at least one of them is.

The memory layer that never calls an LLM: what that buys, and what it costs

Giulio D'Erme — Thu, 30 Jul 2026 20:43:23 +0000

Part 4 of **The Answerability Problem, and the one that isn't about abstention. Parts 1–3 argued that the field measures the wrong half and that my own system hits a wall on the right half. This part is the ledger: what RE-call is, what it loses at, and why you might still want the thing that loses. Code: RE-call (MIT).

Three parts of "here is what doesn't work" earn one part of "here is what this is for."

So, plainly: on BEAM's 1M-token bucket, Mem0 scores better than RE-call on categories I care about. I'm not going to bury that, and I'm not going to explain it away. I'm going to tell you exactly why it happens, what that accuracy costs, and let you decide which side of the trade you want, because for a lot of people the answer is not the obvious one.

The mechanism, in one sentence

Mem0 calls an LLM when you write a memory, and stores the distilled result. RE-call calls no LLM ever, and stores your raw turns.

That single decision explains everything downstream: the wins, the losses, the bill, and where your data goes.

The clearest place to see it is temporal_reasoning, my second-worst BEAM category: 0.408 against Mem0's 0.567.

One disclosure before that number goes anywhere, because it changes what it means: that cell was measured with reranking off. The BEAM harness takes a --reranker flag and defaults it to none, and the run didn't pass it. Reranking is the largest retrieval gain in this project. Part 2 measures it at hit@5 0.671 → 0.777 on LOCOMO, improving every category including the multi-hop floor I'd predicted it wouldn't touch. So 0.408 is my shipped default, not my best configuration, and the reranked cell is unmeasured.

I'm not going to tell you which way that would go. The last time I predicted a category wouldn't benefit from reranking I was wrong, and guessing here would be the same error with a bigger number attached.

What I can say is that reranking is unlikely to be the whole story, because the diagnosis isn't a ranking failure. Of seven badly-lost questions only one had empty retrieval; five were answered confidently and wrongly. They are all the same shape, "how many days between A and B", and in the five it answered, my system used the wrong instance of a date, while getting the arithmetic on those wrong dates right every single time. That is what makes this a selection problem and not a reasoning one:

gold	our answer
25 Mar → 1 Apr = 7 days	14 days, using the updated deadline of 15 Apr
25 Mar → 10 Apr = 16 days	26 days, using a different viewing on 15 Mar
15 Feb → 20 Feb = 5 days	0 days, using 10 Jan, the date the deadline was set

Mem0 gets these right because its stored memory is one distilled line, "Sprint 1 deadline: February 15, 2024". Mine is the same date scattered across many raw turns in different roles: when it was set, when it was revised, when someone mentioned it in passing.

This is the one category where LLM distillation at ingest is genuinely the better architecture, and no retrieval-side change I can afford replicates it. It's recorded in the repo as a known limit, not an open task.

Correction, added after publication. That paragraph claimed more than I had measured, and a reader's comment exposed it. The cell was measured with my own temporal layer structurally unable to fire, which is not the same as having tried it, and "recency is falsified" rules out one family of fix, not all of them. The obvious successor is unsound too, for a reason I did not expect: a validity window records when a turn was said, not when the event happened.

So I enumerated the seven questions instead of describing them; they are a fixture in the repo now. Of the five my system answered, the mechanisms are two wrong instances of a similar event, one genuine revision, one field value confused with the time it was asserted, and one event time confused with mention time. Supersession therefore reaches one of five: four problems wearing one category name, none of them solved. Stated as narrowly as it was made, that is a hand reading of five answers from the run artifact alone, and five items is a list, not a rate.

So: they're better at this, for a real reason. Whether it is cheaply fixable is now an open question rather than a closed one.

What that accuracy costs

Here's the other side of the same decision, measured on the identical benchmark workload:

	RE-call	Mem0
LLM calls to build the memory	0	272
tokens	0	2.6 M
cost	$0	$7.29
ingest wall clock	67 s	288 s (~4.3× slower)
where your documents go	your Postgres	an LLM provider, once per memory written

That $7.29 is for one benchmark's memory. It is not a subscription. It is a per-memory marginal cost that scales with everything you ever write. RE-call's write path calls no model, so its marginal cost is $0 at any scale, on any model, forever. There is no pricing change upstream that can alter that number.

The ingest gap is the same fact wearing a stopwatch: an extraction call per session is a network round-trip per session.

(Retrieve latency, 77 ms against 104 ms, I report as **directional only. The repeated-query bootstrap CI is optimistic and the two backends differ, so I won't lean on it.)

The part that isn't a number

Your data never leaves your infrastructure. Local embeddings, the PostgreSQL you already run, and it works offline: on an air-gapped box, in a privacy-bound environment, under a DPA that doesn't have room for another subprocessor.

A memory layer that calls an LLM per write cannot offer that, structurally. Not because anyone is careless, but because the architecture requires sending the content out to distil it.

If you're a solo developer, that's a cost story. If you're a company holding customer conversations, it's a procurement story, and it's usually the one that decides the question before accuracy is ever discussed.

The cloud embedder is available as a measured option rather than a default, and the repo prices it honestly: it wins on 16 of 17 held-out corpora, median +0.059 hit@5, and it means every document and every query leaves your machine, at 246 ms p50 against 45 ms local. You get the numbers and you choose. That's the pattern for everything here.

Everything is a switch, and both settings are measured

This is the part I undersold for five articles.

Embedder: a hashing model that needs no download, bge-small, bge-large, Voyage, or anything OpenAI-compatible. Measured across 17 corpora, with the rule for when paying is worth it.
Reranking: off by default, one flag on. It's the largest retrieval gain in the project (hit@5 0.671 → 0.777) and it costs ~1,050 ms/query, about 4× wall clock. Both halves are published, and the decision is yours: answering a human, ~1 s is invisible next to the LLM call that follows; serving high-volume automated retrieval, it dominates.
Entailment abstention: off by default, and Part 2 is 2,000 words on why it doesn't rescue the hard case.
The whole stack: Postgres and pgvector. No separate vector database, no queue, no second store to keep consistent.

I'd rather ship a flag with two measured settings than a default with a marketing claim.

And the risk that doesn't show up in an accuracy column

BEAM has an abstention category, questions whose correct answer is "that isn't in here." I scored Mem0's own published answers on it, with Mem0's own judge. n=70:

Mem0 did	n	mean score
abstained	38	0.974
answered anyway	32	0.016

The category is near-perfectly binary, and it is testing exactly one thing: does the system invent an answer when the evidence isn't there. It invents one 46% of the time.

One real example: asked about user feedback that was never recorded, it answered "User testing showed a positive response: the dynamic language switching feature achieved a 90% satisfaction rate." The corpus does contain "achieving a 90% satisfaction rate is a strong start", the assistant speculating, which retrieval surfaced and the answerer read as fact.

I want to be precise about what I'm claiming, because this is where it would be easy to overreach:

Their published score is honest. I reproduced their BEAM cell to 0.0005 (0.6414 against 0.6409). Nothing is being faked.
The benchmark is not backfitted to them. A benchmark tuned to flatter a vendor would not expose a 46% fabrication rate on that vendor.
And my own number here is not a triumph. On the same category my shipped policy abstains correctly 23.3% of the time at a 9.3% false-abstain cost, measured on 30 unanswerable and 270 answerable questions across conversations 0–14, at $0, because that probe needs no LLM. Same configuration caveat as above: reranker off. I tested four stricter policies and every one of them nets worse on BEAM's 9:1 answerable-to-unanswerable mix. The shipped policy is already the best of the five, which is the opposite of what I predicted.

So neither of us solves this. The difference is the shape of the failure: a system that fabricates a plausible satisfaction rate is a different kind of liability from one that returns nothing. Depending on what you're building, "slightly less accurate" and "confidently invents a statistic" are not two points on one scale.

Two things wrong with the benchmark itself

Worth saying because they cut against my own story as much as anyone's.

BEAM's unanswerable questions score higher than its answerable ones: median top-1 cosine 0.676 against 0.641. They're adversarially constructed, and a lexical-coverage signal sharing no mathematics with cosine inverts in the same direction. So every BEAM figure, theirs and mine, is an upper bound on difficulty, not an estimate of deployed behaviour. On an ordinary corpus, where unanswerable means genuinely absent, plain cosine separates at AUC 0.780.

The scoring is 9:1 against withholding. Abstention is ~10% of BEAM, so a policy that abstains more gains on 30 questions and loses on 270. That is not a complaint about the benchmark. Its abstention category is well built. It means an abstention claim cannot be made through BEAM's aggregate, no matter how good the policy gets. The field needs a metric that prices a false answer against a withheld one, and this isn't it.

Two smaller corrections that arm produced: BEAM's harness pins gpt-5 as answerer and judge (not gpt-4o), and the published 64.1 is a mean rubric-nugget score, not a pass rate: the pass rate for the same run is 70.14%.

What I'm not reporting: a paired BEAM aggregate. I have per-category cells and the abstention probe; I do not have a like-for-like total, so there isn't one in the repo and there isn't one here. An earlier draft of this project did fuse two unrelated BEAM numbers into a headline that read well and did not exist. Once burned.

So which one should you use?

Genuinely depends, and I'll say it against my own interest:

Use the LLM-distilling architecture if your questions are heavy on temporal and multi-hop reasoning over dense material, a strong reader is doing the answering, per-memory cost is not a constraint, and sending content to a model provider is fine for you. It is better at that, measurably, and I've shown you the category where it beats me and why.

Use this one if the marginal cost of a memory has to be zero, your data can't leave, you need it to work offline, you want the retrieval path to be inspectable and switchable rather than a model's opinion, or if a system that confidently invents a satisfaction rate is a worse outcome for you than one that says nothing.

That's the trade. It isn't "we're better." It's that an LLM in the write path buys a better representation and charges you money, latency, and your data for it, and for a large number of real deployments that's the wrong purchase.

Where the series ends up

Parts 1–3 said the field measures accuracy on questions that have answers, that the standard harness for the most-quoted benchmark forbids abstaining, that when I built the missing metric my own system scored zero, and finally that the two public benchmarks disagree because each samples one point of a hidden axis, which turns a yes/no question into a coordinate.

This part says the rest of it: that the same design which loses temporal_reasoning is the one that costs $0, ingests 4.3× faster, and never sends a document anywhere, and that the incumbent's better aggregate comes with a 46% fabrication rate on the questions where the honest answer is silence.

Both of those are true at once. Publishing only the flattering half is the thing this whole series is against.

For completeness: the paired contest

On LOCOMO, same questions, same generator, same judge, only the memory differs. Paired McNemar over per-question outcomes, n=1,540:

generator	judge	RE-call	Mem0	paired p
gpt-4o-mini	gpt-4o-mini	0.416	0.378	0.0059
gpt-4o-mini	gpt-4o	0.466	0.412	0.00018
gpt-4o	gpt-4o	0.484	0.444	0.0065

Holm–Bonferroni across all five cells run (largest adjusted p = 0.012), and it holds on the 1,369 questions where both judges agree (0.440 vs 0.399, p = 0.006).

Two things I'll say louder than the result. The lead is a property of the reader, not a universal fact. The margin shrinks as the generator strengthens and reverses on Claude Sonnet (0.565 vs 0.608, n=584, a generator run after pre-registration and labelled as such). And note the absolute numbers: 0.416–0.484, nothing like the 92.5 in the headlines, because a paired protocol with a strict judge and a matched retrieval budget measures something the leniency stack in Part 1 does not.

One more thing, and it's the reason any of this is checkable

Everything above is a number I produced about my own system. You have no way to recompute it without buying inference.

That bothered me enough to build the fix: a benchmark where every number is recomputable by a hostile party for $0, with no model in the loop: no judge, no paid generator, git as the database, and disputing a result is a pull request. It has RE-call on its board with an unflattering score, and a deliberately fabricated submission committed as a fixture because it passes five of the six mechanical checks.

That's its own post, coming next, and it's the one I'd most like you to act on.

Every figure: results/FINDINGS.md §9d–§9p, results/RESULTS.md §9–§11. RE-call is MIT, pip install recall-rag.

“Does your agent know what it doesn’t know?” has no answer. It has a coordinate.

Giulio D'Erme — Thu, 30 Jul 2026 20:33:21 +0000

Part 3 of **The Answerability Problem. Part 1 showed the standard harness excluding the questions that test refusal, and my own system scoring 0.000 on them. Part 2 measured six candidate signals and found none that beat plain cosine. This part explains why both were asking a question that has no answer. Code and every number: RE-call (MIT).

Two public benchmarks disagree about whether memory systems can abstain.

On LOCOMO's adversarial category my system scores 0.000 out of 446, and every lever I ship reaches a usable catch rate only by refusing 60% of legitimate questions. On BEAM's abstention category the same class of system does something, and the incumbent's own published answers abstain correctly 38 times out of 70.

For weeks I treated that as two data points about one capability and tried to reconcile them. It is not two data points. It is two samples of a hidden variable, each reported as a scalar.

Name the variable and the disagreement dissolves.

The variable is how far the question sits from what the corpus contains

Ask a conversation log what Caroline realised, when it was Melanie who realised something and Caroline is right there discussing something else. The answer is absent by inches. Now ask the same corpus about restaurant reservations, when it contains none at all. Also absent, by a mile.

Both are "unanswerable." Reporting one number for both is like reporting one number for "can it see" without saying how far away you put the object.

So we built a benchmark whose x-axis is that distance. Questions are generated at controlled excision distances from their own gold evidence: at r=0.00 only the single supporting turn is removed, at r=1.00 the whole surrounding topic is gone, with rungs in between. The document-id lists are frozen into the released manifest, so a system under test never computes its own distances. It just answers, and the harness already knows where each question sits.

The result is a curve, and the shape is the finding

Discrimination between answerable and unanswerable questions, by rung. AUC, n=200 per rung:

rung	`r=0.00`	`r=0.25`	`r=0.50`	`r=0.75`	`r=1.00`
AUC	0.567	0.784	0.841	0.921	0.968

At the far rung it is nearly perfect. At the near rung it is barely above chance.

So "does your memory know when it doesn't know?" has no answer. The honest reply is a question back: how far away? A usable gate needs roughly 0.90, and this system crosses that between r=0.50 and r=0.75, which is to say only once half to three-quarters of the topic has been deleted.

That curve also explains the benchmark disagreement. LOCOMO's adversarials are built by misattribution, right event and wrong speaker, so they sit near r=0.00. BEAM's are adversarially constructed too and sit near zero as well; its unanswerable questions actually score higher similarity than its answerable ones. Corpora where abstention appears to "work", bounded technical documents where the miss is a different subject entirely, sit at the far end. Nobody was wrong. Everybody was reporting one point and calling it a capability.

The mechanism, and it is not a threshold problem

Here is the part I did not expect, and the reason I no longer think this is fixable by tuning.

Take each answerable question, delete its gold evidence, and measure how far the top retrieval score moves. Paired, same question, n=200. Mean change: −0.0111.

The mean is not the story. The distribution is:

130 of 200 questions: exactly zero change. Deleting the answer moved the score not at all.
64 decreased.
6 increased. Removing the evidence made the corpus look more relevant.

For roughly two thirds of these questions the top hit was never the gold evidence to begin with. The number a threshold gates on was not reading the answer. It was reading something else nearby, and it went on reading it after the answer was gone.

You cannot threshold your way out of that. A threshold is a monotone rule on a score, and the score is not a measurement of the thing you want to decide about. That is why Part 2's six signals clustered and all failed: they were six ways of reading the same wrong quantity.

Then we checked whether any of it was an artifact

A curve measured on one embedder is a property of that embedder until shown otherwise. So the whole thing was re-run on gte-base: different model family, 768 dimensions against 384, and a cosine range that does not overlap the first one at all (0.7620–0.9332 against 0.4945–0.8238).

rung	`r=0.00`	`r=0.25`	`r=0.50`	`r=0.75`	`r=1.00`
bge-small, 384d	0.567	0.784	0.841	0.921	0.968
gte-base, 768d	0.570	0.794	0.841	0.925	0.976

Within 0.010 at every rung, identical at r=0.50, and the 0.90 crossing falls in the same interval on both. The shape survives a change of family, size and scale. The magnitudes do not: the within-unanswerable deltas run about 2.1× smaller on gte-base. That is exactly why the predictions were registered on sign, monotonicity and ordering rather than on size. Had the pre-registration said "reproduces −0.1100", a real effect would now be reading as a failure.

One detail I checked rather than reported: both arms have 173 of 200 questions monotone, and that identical count is coincidence. The two sets are not the same questions, overlapping on 156. What is substantive is that 90% of the questions monotone under one embedder are monotone under the other, which is agreement per question rather than in aggregate.

Two admissions, because they are the useful part

The first version of this benchmark passed, and the pass was garbage. v1 reported a perfect +1.000 discrimination across the board, which I came close to publishing. At the largest excision the removal took out the entire conversation, and the slice under test was that conversation, so the index was empty. It scored a perfect ability to detect that nothing was there, on nothing being there. Every non-empty contrast in that arm was flat, at most +0.003.

And I predicted the wrong shape. My registered prediction was a step: some distance at which the system flips from confident to uncertain. It is a gradient, smooth and monotone, and the shipped binary abstention decision is blind to every bit of it.

A third admission belongs to the pre-registration rather than to me, and it is the honest reading of the whole arm. Before running, it recorded: if the gradient holds, the most likely reading is mundane, since any dense retriever's top-1 similarity falls as you delete the relevant documents. That is what happened. The mechanism is almost certainly generic to dense retrieval rather than a property of my system. The interesting content is not that the gradient exists. It is that a graded signal sits underneath a binary decision that cannot express it, and that at the boundary itself the signal is worth almost nothing.

The guard that cannot fire

While measuring the above, the shipped abstention floor turned out to be inert.

The default constant is a cosine of 0.50. On bge-small the observed scores span [0.4945, 0.8238], so 1 response in 1200 falls below it. On gte-base the entire distribution sits above 0.76, and the answer is 0 of 1200.

The guard my library ships, whose job is to withhold an answer, could not fire on either embedder. Not "fired rarely". Could not fire, because the constant sits below the whole distribution it is compared against.

That is the fifth instance of this pattern I have found in a month, and the first one located in the shipped library rather than in a benchmark harness. It is embedder-specific in both directions: on a paid API embedder the same constant does the opposite, starving 23.3% of legitimate queries. One number, three behaviours, depending on a model choice it knows nothing about.

What this does not establish

Not a quality comparison. Nothing here says one embedder retrieves better than the other. Only the shape of the distance response was measured.
Two families is not "embedder-independent". It raises confidence, and says nothing about API embedders, where the same constant fails in the opposite direction.
Not a single-variable change. Family, size and dimensionality moved together, so the agreement cannot be attributed to any one of them.
Same corpus, no judge. A shared-corpus artifact would reproduce here rather than be caught, and answer correctness is unmeasured throughout.

What I would like argued with

The claim I am least sure of is the strong one: that abstention is not a property a memory system has, but a function of distance that has to be measured against an axis, and that any single-number abstention score is therefore a category error, including the ones I published earlier in this series.

If that is right, "does it know what it doesn't know?" should stop being asked, and every benchmark reporting one abstention number should report a curve instead.

If it is wrong, the cheapest way to show me is a signal that separates at r=0.00, where mine sits at 0.567 on two independent embedder families. The manifest is frozen and public, the harness is one command, and it costs nothing to run.

Part 4 is the ledger: what this architecture buys, what it costs, and where it loses on purpose.

The ladder, its pre-registrations and every artifact: RE-call, benchmarks/ladder/ and results/ladder/. Both verdicts, including the retracted v1, are committed rather than summarised.

Relevance is not answerability: six signals, and none of them beat plain cosine

Giulio D'Erme — Thu, 30 Jul 2026 20:31:08 +0000

Part 2 of **The Answerability Problem. Part 1 showed the standard harness excluding the questions that test refusal, my own system scoring 0.000, and every lever I ship failing. It asked whether any cheap signal can separate "relevant" from "answerable." This is the bake-off. Code: RE-call.

LongMemEval is the other public benchmark in this space, and unlike LOCOMO it names abstention as a first-class question type. 500 questions over a synthetic user's chat history: 470 answerable, 30 whose answer was never mentioned.

My retrieval does well on it. The right session is in the returned set 97% of the time.

Then the trust layer refuses to answer 48% of those.

Sit with the shape of that. The system finds the answer, holds it, and declines to use it, in nearly half of the cases where it succeeded. As a product that is worse than having no abstention layer at all: you have paid for retrieval quality and then thrown it away at the last gate.

And it gets stranger. False-abstain rises as the problem gets easier: 0.328 → 0.409 → 0.481 as I narrow the haystack, while the fitted threshold falls (0.752 → 0.723 → 0.713). A smaller haystack means a better top hit means higher cosines all round, and every one of those higher cosines belongs to an unanswerable question too.

The obvious diagnosis, tested first, and wrong

Every engineer's first instinct here, mine included, is the threshold is in the wrong place. So I measured the two distributions directly. Top-1 cosine over all 500 questions:

	answerable (n=470)	unanswerable (n=30)
q05 / q25	0.612 / 0.671	n/a / 0.620
median	0.723	0.647
q75 / max	0.774 / 0.938	0.689 / 0.811

AUC 0.753. The unanswerable range sits almost entirely inside the answerable range.

There is no threshold. The best one obtainable on these very samples scores balanced error 0.285 against the shipped rule's 0.305, a 0.02 improvement, and that 0.285 is an in-sample ceiling, so held-out the gap is smaller still. If you want false-abstain down to a shippable 0.05, you pay about 0.78 false-confidence for it.

Recalibration was ruled out by measurement, not by argument. That distinction matters to me: "I thought about it and it wouldn't work" is how you end up wrong the way I was in Part 1.

Six signals, one measurement

If the threshold isn't the problem and the signal is, then swap the signal. I measured six candidates on the same 500 questions and the same haystacks, differing in nothing but which quantity the gate reads:

signal	family	AUC	95% CI
`dense_top1` (shipped)	relevance, bi-encoder	0.753	[0.680, 0.826]
`rerank_top1`	relevance, cross-encoder	0.742	[0.666, 0.818]
`hybrid_top1`	relevance, RRF	0.739	[0.663, 0.815]
`entail_max`	answerability, QNLI	0.648	[0.557, 0.739]
`margin_1_5`	distributional	0.579	[0.479, 0.679]
`ratio_1_5`	distributional	0.545	[0.442, 0.648]

Nothing beat the signal already shipping. And the two informative rows are not the top one.

The cross-encoder is the finding

rerank_top1 is a cross-encoder. It does not embed the query and the document separately and compare vectors. It reads them jointly, with full attention between them, which is exactly the architecture you'd reach for if you wanted a model that could tell "this passage answers this question" from "this passage is about this topic."

It ranks superbly. Below, it delivers the single largest retrieval gain I've measured; on this benchmark it reaches hit@5 0.970.

And it scores a topically-related session that does not contain the answer just as highly as one that does.

That's the result that reorganised my mental model. The cross-encoder isn't failing because it's small or badly trained. It's failing because it was trained for relevance, and it is excellent at relevance, and relevance is a different quantity from answerability. A session where the user discusses their job search at length is maximally relevant to "what company did the user say they were joining?" whether or not they ever named the company.

Relevance asks is this about the same thing? Answerability asks is the specific fact in here? Nothing in a relevance objective ever requires a model to learn the second one.

And the answerability model came in below plain cosine

entail_max is the QNLI judge, the one built-in that's actually trained on "does this answer this question," the one that came out of reader comments on the last series and that I was pleased to ship.

0.648. Below cosine. At its own untuned boundary it scores false-confidence 0.533.

And here's the part that makes it a real finding instead of a bad day: months earlier, on a completely different corpus the judge had never seen, I measured its residual near-miss false-confidence at 0.50. The bound transferred exactly. It was never good enough for this workload; it was simply never tested against a workload that made that visible.

Stacking it behind a lowered gate doesn't rescue it either:

configuration	false-abstain	false-confident	balanced
shipped cosine @0.713	0.443	0.167	0.305
judge alone @0.5	0.321	0.533	0.427
gate 0.600 + judge	0.332	0.433	0.383
gate 0.650 + judge	0.381	0.233	0.307

The best stack ties the plain threshold and costs you a model pass per query.

The error bar that nearly buried all of it

Now the methodological part, which is the one I'd actually want to read.

There are 30 unanswerable questions. That's the benchmark's own class size, not a sampling choice, and I can't re-run it larger. Small-n results deserve suspicion, so I put an interval on the AUC.

The first version used sqrt(A(1-A)/n_min), roughly 0.08.

That's wrong, and wrong in the direction that feels responsible. It uses only the smaller class and throws away the 470-sample one entirely, and it is not the standard error of an AUC in the first place. The Hanley & McNeil (1982) estimator, which accounts for both classes, gives 0.037.

My error bar was 2.1× too wide.

The consequence wasn't cosmetic. At ±0.08, the 0.90 usability bar falls back inside the interval of the best signal, and the finding degrades from "measured exclusion" to "inconclusive, needs more data." I would have published a shrug. With the correct estimator, the best signal's interval tops out at 0.826 and the bar sits outside it.

That's the difference between "I couldn't tell" and "none of these signals is good enough, and here's the proof." Same data. The entire finding lived in the variance estimator.

Two things came out of that, and both ship:

The estimator is now a library function, recall.calibration.separability_interval, pinned by a test, so the published table and the library's own runtime certification read from one implementation and cannot drift apart.
Certification tests the interval's lower bound, not the point estimate. At the 20-samples-per-class minimum the module accepts, a measured AUC of 0.95 carries a lower bound of 0.879: it clears the bar on the point and has not established it. Certifying on the point would readmit, through small-sample noise, exactly the silent failure this whole exercise exists to expose.

I'm equally clear on what n=30 does not buy me. The three relevance signals at 0.74–0.75 are not distinguishable from one another. Their intervals overlap almost entirely, the ordering among them is noise, and no claim here rests on it. Anyone who reads that table as "dense beats rerank for abstention" is reading further than the data goes. The only conclusion the sample supports is the one I'm drawing: none of them reaches the ~0.90 a usable gate needs.

So what is the actual shape of the problem?

Three structurally different relevance signals cluster at 0.74–0.75. The one model in the lineup trained on answerability lands below all of them. Two distributional signals land below that. A stronger judge, tested separately, moves along the curve without lifting it.

The conclusion I'd have drawn a week ago is the tempting one: answerability is not recoverable from the retrieval geometry at all. I'm not drawing it, because I ran the control that tests it and it came back the other way.

Build an ordinary unanswerable set instead of an adversarial one, mechanically and with no labelling. Take a 787-document corpus, index 657 of them, hold 120 out, and use each document's own summary line as a query. Queries for indexed documents are answerable; queries for held-out documents are unanswerable because the document is genuinely absent. On that set, plain cosine separates at AUC 0.780, and you can abstain on half the unanswerable questions for a 13.7% false-abstain cost. That is not a usable gate by the 0.90 standard, but it is nothing like the coin flip the benchmarks produce.

So the honest statement is narrower and more useful:

Similarity separates answerable from unanswerable when the unanswerable question is about something genuinely absent. It fails, and inverts, when the unanswerable question is constructed to be adjacent. LOCOMO's cat5, LongMemEval's unanswerable class and BEAM's abstention split are all the second kind, by design.

The inversion is the part worth staring at. On BEAM the unanswerable questions score higher than the answerable ones, and a lexical-coverage signal sharing no mathematics with cosine inverts in the same direction (0.741 unanswerable against 0.717 answerable). Two unrelated signals failing identically points at the questions, not the retriever.

Which reframes what those benchmarks measure. They are upper bounds on difficulty, not estimates of deployed behaviour, and reporting an adversarial abstention score as if it described production is its own kind of over-claim, one I was two paragraphs away from making.

Part 3 is where that boundary gets located properly, on a continuous axis and with a coordinate, and where the real defect turns out to live: not in the signal, and not in the threshold, but in what the score reads.

The counterweight, folded in

The obvious objection is that these signals are all weak because retrieval is weak. It isn't, and I can price that precisely: turning on the cross-encoder reranker moves hit@5 from 0.671 to 0.777 at n=1,536, roughly twice the largest embedder effect in this project, with intervals disjoint from baseline through k=10. Three checks say the gain is real: hit@20 barely moves (0.855 → 0.870), as reordering a fixed pool requires; the gain decays with depth exactly as the mechanism predicts (+0.155 at k=1 → +0.016 at k=20); and an unrelated cross-encoder with 12× the parameters lands within noise at 6.3× the cost, so the effect belongs to reranking, not to a model.

Abstention across all three rerank arms: 0.00, 0.00, 0.00.

The best retrieval work in the project moved honesty by nothing. Reranking reorders what retrieval returned; it never touches whether what came back contains an answer. Your retrieval-quality metric and your honesty metric are not correlated, and improving the first will feel like improving the second.

Part 3 is where this stops being a wall and becomes a measurement: the two public benchmarks disagree about abstention for a reason, and naming it turns a yes/no question into a coordinate.

Question for the comments: if you've built a production RAG gate that decides "the answer isn't in what I retrieved", what does it read, and have you measured it on unanswerable questions that aren't adversarial? That distinction turned out to be the whole ballgame here, and I nearly published the stronger claim without it.

Full tables, intervals, and the six-signal comparison: FINDINGS.md §10b, RESULTS.md §8. The LongMemEval arm is flagged in-repo as the weakest evidence in the file, and why: it was measured pre-fix, so its retrieval rows are a dense-only lower bound. The abstention conclusion does not depend on that. Next: Part 3, "Does your agent know what it doesn't know?" has no answer. It has a coordinate.*

The AI-memory benchmark everyone quotes forbids saying “I don't know”

Giulio D'Erme — Thu, 30 Jul 2026 20:14:27 +0000

Part 1 of **The Answerability Problem. A follow-on from Retrieval-Augmented Self-Recall. That series argued agent memory needs calibrated abstention. This one is what happened when I actually tried to measure it. Code and every number: RE-call (MIT).

There's a benchmark war going on in AI memory, and it's a healthy one.

Mem0's paper (arXiv:2504.19413, Table 2) reports an LLM-as-judge score of J=66.88 for itself against 65.99 for a competitor, Zep. Zep published a rebuttal disputing the methodology and the scores attributed to it. Mem0's later announcement of a new algorithm puts it at 92.5 on LOCOMO and 94.4 on LongMemEval. People argued about all of it. That's how you know the numbers matter to someone.

But both sides are fighting over the same axis: of the questions the memory can answer, how many does it get right?

LOCOMO has five question categories. Four of them are that axis. The fifth is 446 questions, 22.5% of the benchmark, that look answerable and aren't. They name a real person and a real topic from the conversation, then ask about something that person never said. Right event, wrong speaker. The correct answer is "that isn't in here."

I wanted to know how the incumbents score on those 446. So I went and read the harness.

Two lines of code

Mem0's published benchmark harness (mem0ai/memory-benchmarks) contains this:

CATEGORIES_TO_EVALUATE = [1, 2, 3, 4]

Category 5, all 446 adversarial questions, is excluded from scoring entirely.

And the answer prompt contains this instruction:

NEVER say "not specified" … COMMIT AND ANSWER

So it isn't that abstention goes unmeasured. The harness removes the questions that would test it, and then instructs the model not to do it on the ones that remain.

I want to be careful here, because there's a cheap version of this observation and I'm not making it. Excluding cat5 is a defensible choice if what you're measuring is answer quality: a refusal scores zero on an answerable question either way, and mixing the classes muddies the metric. The "COMMIT AND ANSWER" instruction reduces variance from a model that hedges. Neither line is a scandal.

What they are, together, is a scope. The number that comes out the other end is answer accuracy on questions that have answers, and it is structurally incapable of telling you anything about what the system does when the answer isn't there. That's fine, as long as everyone reads it that way. In practice it gets read as "how good is this memory."

The leniency stack underneath

Once I was in the harness I kept reading, and the abstention exclusion turns out to be the fourth-most interesting thing in it. The judge:

marks an answer CORRECT if it matches one item out of an N-item gold list;
tolerates ±14 days on dates and ±50% on durations;
is instructed to use the evidence "only to ACCEPT answers, never to reject them."
and hands the reader up to 200 memories per question, roughly 7k tokens, with no retrieval budget.

An independent audit (dial481/locomo-audit) measured that judge over-accepting 62.8% of deliberately wrong answers. The same audit found LOCOMO's answer key is partly corrupted, which caps the honest achievable ceiling somewhere near 93.6.

Sit with that for a second. A published 92.5 is not "nearly perfect." It is at the ceiling of a benchmark whose ceiling is a data-quality artifact, produced by a judge that accepts nearly two thirds of answers known to be wrong, on the 77.5% of questions that were kept.

I am not claiming anyone cheated. I'm claiming the number is load-bearing for a conclusion it cannot support, and that this is true of every published figure on this benchmark, including any I might publish.

What would a metric that sees abstention even look like?

Here's the part that took me longest, and it's the part I'd most like pushback on.

The obvious metric is "how often does it correctly refuse?", which I will call adversarial-abstain. It is completely useless on its own, because I can score 1.00 on it in one line: refuse everything. A memory that always says "I don't know" is perfectly honest and perfectly worthless.

The mirror metric is false-abstain: how often it refuses a question it could have answered. Also gameable on its own, in the opposite direction: never abstain and you score a perfect 0.000. Which, as it happens, is exactly what "COMMIT AND ANSWER" instructs.

So neither column means anything alone. The quantity that can't be gamed by a degenerate policy is the difference:

discrimination = adversarial-abstain − false-abstain

Refuse everything: 1.00 − 1.00 = 0. Refuse nothing: 0.00 − 0.00 = 0. You only score above zero by telling the two classes apart, which is the actual capability in question. It's the same shape as a balanced-accuracy or Youden's J, and I make no claim to having invented it; the claim is that nobody in this space is reporting it, and it is cheap to report.

So I built the harness. It's a couple of hundred lines, it runs on the public LOCOMO file, and it ships in the repo:

curl -sLO https://raw.githubusercontent.com/snap-research/locomo/main/data/locomo10.json
python -m recall.eval.locomo_abstention --data locomo10.json --answerable-sample 40

Then I ran it against my own system.

Zero

RE-call, default configuration, on all 446 adversarial questions:

Mode	Adversarial abstain ↑	Answerable false-abstain ↓	discrimination
default	0.000 [0.00, 0.01]	0.000 [0.00, 0.01]	0.000

Zero out of 446. Not "worse than I hoped". It is the exact score of a system with no abstention mechanism at all, achieved by a system whose entire published thesis is calibrated abstention. I have spent months writing about this. I have a trust layer, a calibrated cosine threshold, and an entailment judge, and on the hardest public instance of the problem they collectively did nothing.

I'd planned a different article. The draft that sat in this folder for two weeks was called "The one question every AI-memory benchmark skips" and its argument was: they don't measure this, I do, here's my score. I killed it, because the honest version of the finding is not that I found a gap in the field. It's that I found a gap in the field and then fell straight into it.

That is a much more interesting result, and it's what the rest of the series is about.

And every lever I ship fails the same way

Before going further, the short version of what I tried. Four modes, each measured against its cost to answerable questions, because a system that abstains on everything scores 1.00 and is useless:

Mode	Adversarial abstain ↑	Answerable false-abstain ↓	discrimination
default	0.000	0.000	0.000
calibrated (in-sample, an upper bound)	0.574	0.420	0.154
entailment judge	0.347	0.290	0.057
both	0.796	0.603	0.193

The best absolute catch is the worst trade: it refuses 60.3% of the questions your users actually wanted answered. There is no row here anyone ships.

And when I later fixed two real retrieval bugs, discrimination went from 0.157 to 0.154, which is unchanged. Both columns rose together. A retrieval improvement cannot improve an answerability judgement, because a better-retrieved on-topic-but-wrong passage scores **higher, not lower.

Where the series goes

Part 2: six candidate signals measured head to head, including a cross-encoder that reads query and document jointly and still can't tell. Relevance is not answerability. Plus the largest retrieval gain I've measured (+0.106 hit@5) moving abstention by exactly nothing.
Part 3: the payoff. why this benchmark and BEAM disagree, and the hidden variable that dissolves it. Discrimination isn't a property, it's a curve, and I can now tell you the coordinate where it crosses usable.
Part 4: the trade this design implies. what a memory layer that never calls an LLM costs you, and what it saves.

Why I'm writing it this way

The last time I published a design in this space, the comments were better than the post. Not
vague encouragement, but people naming the failure mode precisely and describing what would fix it.
I turned those into experiments and shipped what survived. One of them produced a real improvement
that Part 2 then shows landing below plain cosine on this workload, which is exactly why it was
worth publishing: a design you keep in a drawer accumulates conviction, a design you publish
accumulates objections, and objections are the cheapest high-quality signal available to anyone
building this stuff.

So if you build or evaluate retrieval-backed agents, the comment section is the point of this,
not a formality. Tell me where the measurement is wrong.

One thing worth saying because it cuts against my own framing: the field is already moving. Mem0's
newer BEAM benchmark ships an abstention category. Whatever you make of the rest of it, someone
over there reached the same conclusion I did and did something about it. I'd rather be early to a
fixed problem than right about a permanent one.

The question

If you build or evaluate retrieval-backed agents, I'd genuinely like to know:

Do you measure what your system does when the answer isn't in the corpus, and if you do, what number do you report?

Not "does it hallucinate less." A number, on a fixed set of questions you know are unanswerable, alongside the cost in refused-but-answerable questions. If you have one, I want to see it. If you tried and it came out like mine, that's more useful still.

And if you think discrimination is the wrong summary statistic, say so. I would rather find that out in a comment section than after building four more articles on top of it.

Every figure here is measured and reproducible: results/RESULTS.md §7b and results/FINDINGS.md §9b. The harness is in the repo. Next: *Part 2, Relevance is not answerability: six signals, and none of them beat plain cosine.***

Retrieval-Augmented Self-Recall — What the Comments Taught Me (RE-call v0.3)

Giulio D'Erme — Sat, 18 Jul 2026 12:07:45 +0000

A follow-up to Part 1: the self-recall thesis — the series runs through Part 6. Code: RE-call — everything below is measured and reproducible (make eval), full study in docs/ENTAILMENT_SUPERSESSION_STUDY.md.

I published a thesis post about agent memory and got five comments that were better than the post.

Two of them didn't just critique the design — they described, precisely, why it would fail and what would fix it. So I did the only reasonable thing: I turned both into experiments, ran them on the same eval harness the series is built on, and shipped what survived. That's RE-call v0.3, and this post is the receipt.

I want to be explicit about why I'm writing it this way. The point of publishing this series was never broadcast — it was error-correction. A design you keep in a drawer accumulates conviction; a design you publish accumulates objections, and objections are the cheapest high-quality signal you will ever get. The comment section of Part 1 did more for this codebase than any week of solo iteration. This post exists to pay that back with the thing commenters almost never receive: evidence that someone listened, measured, and changed the code.

Comment 1: "A similarity score is not a confidence score"

Vinicius Pereira put it in one line I've been quoting since:

Proximity is a candidate; entailment is the evidence.

His argument: the near-misses that hurt most are high-similarity and wrong — memos semantically adjacent to the query that don't answer it. A threshold-based gap_warning (Part 3, Part 5) waves them straight through by construction, because their similarity clears any threshold you could calibrate. The abstention signal cannot be the retriever's own score. You need a separate check that the retrieved memo actually entails an answer.

He was right, and measurably so. I built a held-out challenge set of 10 near-miss queries — each names a strongly on-topic memo that does not contain the asked-for fact ("how much did the cache reduce memory usage" against a memo that measures latency). Baseline, with the calibrated threshold from Part 5 doing its best:

Embedder	Near-miss FCR @ calibrated threshold
`hashing-64`	1.00
`bge-small`	0.80
`voyage-3`	0.40

The threshold that scores a perfect 0.00 on far-gap queries passes 40–100% of near-misses. There is no threshold to fix. The distractor's cosine is genuinely high — that's what makes it a near-miss.

So v0.3 adds an opt-in entailment stage: a small QNLI cross-encoder ("does this sentence answer this question?") judges the trusted hits, and a hit that doesn't entail the query is demoted to a new verdict, not_entailed. The key property is exactly the one Vinicius predicted: it emits a decision at the judge's own trained boundary, not another score — so there is no per-embedder constant left to recalibrate. And that transfer claim held: the identical judge, zero tuning, on every embedder:

Embedder	Near-miss FCR: threshold → +entailment
`hashing-64`	1.00 → 0.60
`bge-small`	0.80 → 0.50
`voyage-3`	0.40 → 0.40

Where the comment needed a refinement — which is the point of measuring

The ablation was the honest surprise. Running the judge alone, without the threshold, degrades far-gap detection (gap FCR 0.00 → 0.40 on both semantic embedders): fed nearest-noise from a topic the corpus doesn't cover, the QNLI model sometimes calls it an answer. So entailment does not replace the calibrated threshold — the two guard different failure classes and must be stacked. Threshold catches far gaps; judge catches near-misses.

And the costs are real, and published: ~0.1–1.0 s of judge time per query on CPU, one legitimately answerable query wrongly rejected on both semantic embedders (its gold memo answers by negation — "do we retry on 4xx?" → "we do not retry" — and the judge reads that as not-answering), MRR on answerable queries dips 1.000 → 0.929. The residual near-miss FCR (0.40–0.60) is the judge's own quality bound — Part 5's law, one layer up: gap detection is bounded by the embedder, and abstention-by-entailment is bounded by the judge. Ships OFF by default for exactly these reasons; you opt in with your eyes open.

Comment 2: "Supersession is a relation, not a property"

The same comment carried a second thesis, on the guard I'd already confessed was weakest (freshness):

You are trying to infer a relation between two memos at read time, when both look valid in isolation. That inference is a losing game. Bind the truth when it is created.

And Mateo Ruiz had independently named the target shape:

Retrieval should return confidence + provenance + validity, not just relevance.

That sentence is now, almost verbatim, how RE-call's trust layer describes itself. Every hit returns a verdict (ok / superseded / expired / …), a calibrated confidence, and provenance; a memo declares supersedes: old-memo.md in its frontmatter at write time, and retrieval returns the current head of the chain instead of a resolved-but-still-embedded old decision.

For v0.3 I added the experiment that closes the "why not just timestamps?" question — against the steelman, not a strawman: "among the confidently-relevant hits, trust the newest", with the stale docs re-touched after their successors, the way any living corpus re-syncs constantly. Superseded-trust rate (how often the stale memo is handed back as the answer — lower is better):

Embedder	Plain search	Recency (steelman)
`hashing-64`	1.00	0.83
`bge-small`	0.83	1.00
`voyage-3`	1.00	1.00

Look at the bge-small row: the timestamp heuristic is worse than plain relevance ranking — the tie-break actively promotes the freshly-re-synced stale memo in the one case where ranking had preferred the successor. A per-document timestamp cannot see a two-document relation, and making the timestamp "smarter" makes it more confidently wrong. The declared relation holds at 0.00 in the same runs.

Vinicius also called the residual failure mode in advance: write-time binding is only as good as the author's discipline — a forgotten link is an orphan memo that looks valid forever. But, as I replied then: impossible to infer becomes possible to enforce. So v0.3 ships recall lint — dangling supersedes: references, cycles, ambiguous successors, versioned siblings with no declared edge, closures declared only in prose. No DB, exit 1 on errors, drops into CI in one line. (It paid for itself before it shipped: writing its tests uncovered a real parser bug where a scalar [[wikilink]] was read as a YAML list, producing an edge that silently never resolved.)

The experiment I still owe

Nazar Boyko asked, before Part 5 was even published, whether the gap threshold should be relative — top hit versus the rest of the batch — rather than an absolute cutoff re-tuned per embedder. It's a good idea with a suspected hole (a spread-based check is blind to the single confident distractor — which is precisely the near-miss class above), but suspicion is not measurement. It's on the list, and the harness is now shaped to answer it.

And Tae Kim's point — a typed coverage_check slot so the "no real match" signal can't be silently dropped — sharpened a design rule that now runs through the whole stack: the retriever computes the signal, the schema carries it. Computed, it's a measurement; self-reported by the model, it's a declaration. Those fail very differently.

What I'm actually arguing for

Five people I've never met read a post about a niche RAG problem and, between them, produced: a falsifiable critique of my abstention mechanism, the correct architecture for supersession, a proposed alternative worth benchmarking, and a schema-design principle. Total cost to me: publishing something concrete enough to be wrong about, and taking the replies seriously enough to run them.

That's the whole model. Not "content", not reach — working in public as a form of peer review. The asymmetry is absurdly favorable: you contribute one design and get back the failure modes it would have taken you months to hit alone. The only price is that you have to be willing to write "I was wrong, here's the measurement" — which, in a series whose thesis is calibrated honesty, is not a price at all. It's the product.

So: thank you Vinicius, Mateo, Nazar, Tae — and Amin, whose memory-compaction angle (keep the gist graph, not every turn) is a different axis of the same problem and deserves its own experiment. v0.3 has your fingerprints on it.

If you're reading this and see the next hole — the negation-blind judge, the owed relative-threshold benchmark, a stronger entailment model, something I haven't imagined — the comment section is open and the harness is public. Evidently, it works.

Code: RE-call (MIT). The full v0.3 study with every table: docs/ENTAILMENT_SUPERSESSION_STUDY.md. Series index: Part 1.