DEV Community: Giulio D'Erme

The RAG benchmark that pays you for saying "I don't know", and what it exposed

Giulio D'Erme — Sun, 09 Aug 2026 11:33:16 +0000

Nine published systems. One of them refuses 2.3× more often than gpt-4o and still finishes 4th on answer quality. It calls no LLM.

Part 5 of The Answerability Problem. Parts 1 to 4 argued the field measures accuracy on questions that have answers. This one is about the benchmark that doesn't. Code: RE-call (Apache 2.0).

MTRAG (IBM, TACL 2025) is 842 human-written multi-turn tasks with an official gpt-4o-mini judge that pays a full 1.0 on every metric for correctly saying "I do not have that information."

Almost no other benchmark scores a refusal as anything but a miss. So this is the one board where the thing my system is built for is actually worth points.

Here is what nine published systems do with that.

Nobody is collecting the free money

Correct refusals on the 55 unanswerable tasks. Same judge, same tasks, every system:

#	system	correct refusals
1	llama-3.1-8b	18/55	32.7%
2	🔹 `gpt-4o` + RE-call	16/55	29.1%
3	llama-3.1-70b	16/55	29.1%
4	gpt-4o-mini	13/55	23.6%
6	gpt-4o (benchmark's own retrieval)	7/55	12.7%
7	llama-3.1-405b	3/55	5.5%
9	qwen-2.5-72b	1/55	1.8%
10	mixtral-8x22b	0/55	0.0%

Two rows in that table read the same document set with the same model. Row 6 is gpt-4o on the benchmark's own retrieval. Row 2 is gpt-4o on RE-call's. Same reader, same prompt, same judge, only the contexts differ, and the refusal rate goes from 7 to 16 out of 55.

Then read the rank order. It is close to inverted against model size. The 8B model refuses best. The 405B model refuses 5.5% of the time. One model refuses nothing at all. Scale does not buy this, and on this evidence it costs it.

RE-call is second of ten, tied exactly with llama-3.1-70b at 16 of 55, and it refuses 2.3× more often than gpt-4o and 16× more often than qwen-2.5-72b.

The usual objection, answered

Abstention is cheap if you don't care about answering. So here is what it cost, on the same 842 tasks:

#	system	answer quality
1	llama-3.1-405b	0.5691
2	qwen-2.5-72b	0.5625
3	gpt-4o (benchmark's own retrieval)	0.5591
4	🔹 `gpt-4o` + RE-call	0.5527
5	c4ai-command-r-plus	0.5502

RE-call is the only system in that top four that is also top two on abstention. The three above it refuse 5.5%, 1.8% and 12.7%. The gap to gpt-4o is 0.0064.

That is the whole claim, and it is a narrow one: not "best", but the only one on the board that is near the top of both columns at once. Everyone else picked a column.

Same two rows again, 3 and 4, same reader. Swapping the benchmark's own retrieval for RE-call's costs 0.0064 on answer quality and more than doubles what the system correctly refuses. That is the trade, priced.

And against the benchmark's own retrieval directly, identical generator, prompt and judge, only the contexts swapped, RE-call is ahead: 0.5527 against 0.5516.

Small. Positive under both prompts I tested. I am not going to inflate it.

(Every row above is scored by MTRAG's official gpt-4o-mini-2024-07-18 judge, hard-coded by the benchmark and applied identically to my rows and theirs. The reader on my rows is gpt-4o, the same model as the row I am compared against.)

One engine, whatever you can afford

The other half of this. The same pipeline runs at both ends of the price range, and every rung is a named flag with a measured price, not a tier you buy.

configuration	nDCG@5	cost
🔹 SPLADE learned sparse (the free default)	0.3573	local, $0
+ Voyage rerank (one flag)	0.4342	paid API

A 48% relative span between two flags, measured on 777 judged queries.

The reranker is worth +0.0769 nDCG@5 and is worse on 162 of the 777. That is exactly why it is off by default. I would rather ship a flag with two published numbers than a default with a marketing claim.

Everything dashed and amber below is opt-in and switched on by name. The solid path is what runs if you change nothing, and it costs $0 at any scale, because there is no LLM anywhere in it:

Writing a memory is a local embedding. No LLM call, so the marginal cost of a memory is $0 and stays $0 at any scale, on any model, forever. There is no upstream pricing change that can alter that.

Air-gapped and fully local, or cloud embedder and reranker when a jargon-heavy corpus makes it worth paying. Same engine, same guarantee at every setting: superseded or expired memories are demoted rather than served, and the confidence threshold is fitted per embedder and corpus at install time, never a shipped constant.

The number I nearly published

One thing before you trust any of the above.

An earlier version of this run had RE-call beating every baseline, including llama-3.1-405b. It was an artifact and it was one edit away from going out.

The official scorer reads the answerability label in lower case. The release files ship it capitalised. So the conditioning that pays 1.0 for a correct refusal silently never ran on my data: 2,526 lookups, zero label matches. My metrics were raw, the published baselines' were conditioned, and I was comparing two different quantities in exactly the place where my system is strongest.

Both sides were individually correct. The error lived entirely in the join.

Two signals should have caught it earlier and I explained both away. I had even written the memo describing the two-key problem, that same day, and never connected it. Having the fact is not the same as applying it.

Corrected, the honest reading is: RE-call at 0.5527 against gpt-4o's 0.5591, so −0.0064. Fourth, not first. The abstention result survives intact, because it was measured by the official judge rather than by the string-matching detector that had earlier inflated it from 29% to 43.6%.

The anomaly you can explain is still an anomaly, and the one that flatters you is the one to chase hardest.

So which rung is yours

Take the free default if the marginal cost of a memory has to be zero, your data cannot leave, or you need it to work offline. 0.3573, local, $0, no API key.

Add the reranker when a human is waiting. One second is invisible next to the generator call that follows it, and it is the largest single gain in the project. Check it on your corpus first: it lost on 162 of 777 queries here.

Use something else if abstention is worth nothing to you and you only need the top of one column. Three systems beat me on answer quality. All three of them will confidently answer a question whose answer is not in the corpus, between 87% and 98% of the time.

That is the trade. A system that invents a plausible answer and one that says nothing are not two points on one scale.

Every number, the six generation runs behind them, the correction in full, and the scoring bug I reported upstream: docs/MTRAG_BENCHMARK.md. RE-call is Apache 2.0, pip install recall-rag.

Make your AI assistant predict the result before it runs it

Giulio D'Erme — Sun, 02 Aug 2026 14:20:27 +0000

Less technical than my usual posts. Nothing to install, nothing to pay for, and it works the same whether you're using an AI assistant for code, spreadsheets, research or writing.

Here is the shape of the problem I want to describe. See if you recognise it.

You ask your assistant to do something that produces a result. It does. A number, a summary, a status, an answer comes back. It's plausible. You ask why, and you get a clear, confident explanation that makes complete sense of it. So you believe it, you write it down, and you build your next three decisions on top of it.

A week later you find out the whole thing was wrong. Not wrong in an obvious way. Wrong because the tool was measuring something you didn't ask about, or reading a file that was empty, or answering a slightly different question than the one in your head.

Notice what didn't happen there. Nothing was invented. No fake citation, no made-up fact, none of the failures people mean when they say "hallucination". The failure was structural: nobody committed to what the answer should be while it was still possible to be wrong.

The habit

Before your assistant runs anything that produces a result, make it answer two questions.

1. What do you expect the result to be, and why?

2. How would you know if this were broken?

That's the whole technique. Two sentences before the work instead of a paragraph of explanation after it. It works on the free tier. It works on any model. It costs a few seconds.

Why the first question works

This isn't about "thinking harder", and it isn't a motivational trick. It changes the kind of problem your assistant is solving.

Explaining a result you've already seen is an open-ended task. There are a huge number of stories that fit any given outcome, and a language model is exceptionally good at finding one of them. That's not a flaw, it's the thing it's built to do. The problem is that a fluent explanation tells you nothing about whether the result is correct. It reads identical either way.

Predicting a result you haven't seen is a closed task. Everything the assistant believes about your data, your files and your request has to collapse into one statement that can turn out to be wrong. And when the real output disagrees with the prediction, you get a signal that could not have been manufactured after the fact, because the prediction is already sitting there on the screen.

That gap between expected and actual is the entire product. It's cheap to produce and impossible to fake in hindsight.

There's a bonus effect, and it's the reason results feel steadier once you do this. Predicting first pins down the standard of success before the data arrives. Most of the wobble between one run and the next isn't the model changing its mind, it's the standard quietly moving to fit whatever showed up.

Science has a name for this and an entire apparatus around it: pre-registration. Nobody needed to invent it for AI. We just forgot to apply it.

Why the second question matters more

I learned this one the expensive way.

I had a system meant to answer time-sensitive questions: which of these documents was true as of a particular date. It scored badly. I had predicted it would score badly, because the task is genuinely hard. Prediction matched result. I nodded, wrote the number down, and moved on.

The prediction was right and the measurement was worthless.

None of the documents had dates attached to them. Not one, out of four hundred and twenty. The system was being asked a question about time using material that had no time in it. When I fixed that, with exactly the same documents, the score went from 0.19 to 0.98.

I want to be careful with that number, because it's the kind of number people quote badly, including the person who produced it. That is not an improvement. Nothing got smarter. That's what it looks like from the outside when a broken instrument gets repaired, and if I'd published it as a win I'd have been telling a lie with correct arithmetic in it.

Here's the general rule, and it's the part almost everyone skips:

Predicting the answer will not tell you the question was never asked. If the setup is broken, your prediction and your result can be wrong in the same direction, agree with each other perfectly, and confirm nothing.

Which is exactly what the second question is for. "How would I know if this were broken?" forces something different out of the assistant: not a guess at the answer, but a statement about the machinery, one that has to hold true whatever the answer turns out to be.

Expectation: "the score will be low, this is a hard problem."
Broken-check: "if the documents don't have dates on them, the question isn't being asked at all."

The first one was right and taught me nothing. The second one would have gone off on day one.

Done is not the same as proved

That second question generalises into something I now say out loud several times a week.

When a task reports success, what you have learned is that a task reported success. You have not learned that a file was written, that the file has anything in it, that the numbers cover the dates you asked for, or that the check you requested was actually performed.

My favourite example is small and stupid. I once wrote a rule-checker to catch a specific kind of mistake in my own work. It ran clean for weeks. It ran clean because of a settings mismatch that meant the rule I cared about was never switched on. The checker was structurally incapable of finding anything. Nothing was broken, nothing errored, everything was green, and the green meant nothing at all.

The first test I wrote to fix that read the settings file to confirm the settings file. It passed. Two separate reviews walked straight past it. The thing that finally caught anything was running the checker against a deliberately bad example and watching it complain.

So: ask for the output, not the status. Show me the rows. Show me the file. Show me it catching one.

Anything that's supposed to warn you, make it warn you once

Same idea, pointed somewhere useful.

A safety net you have never seen catch anything is not a safety net. It's a guess in the shape of one. This applies to far more than code: the alert that's supposed to email you when a number goes out of range, the filter that's supposed to catch the bad rows, the check that's supposed to stop the bad export.

So test it deliberately. Feed it the bad thing on purpose, once, and watch it fire.

I ran that exercise across a system I'd been maintaining for months. It found twelve warnings that could never have gone off under any circumstances. One of them was the reason a cost had been quietly running for about two months before anyone noticed. Every one of those twelve had been read and approved by someone, including me. Reading them was never going to work, because they read correctly. That's precisely what makes them dangerous.

Two things I got wrong before I got them right, and they generalise:

Breaking it on purpose only proves whatever you actually broke. I once "confirmed" that two checks happened in the right order by deleting one of them entirely. All that proved was that a check existed. The ordering, which was the actual thing I cared about, stayed untested.

A check that can't pass is as useless as one that can't fail. I hit five in a row where the thing also refused to go green on a known-good input, and each one had been introduced while fixing the previous one. Identical behaviour on the good case and the bad case means it's measuring nothing, in either direction.

Look it up, don't remember it

One more, cheaper than everything above. When the assistant needs a fact it could go and check, make it go and check. A confident wrong value is worse than an admitted blank, because a blank is honest and gets dealt with, while a wrong value slides silently into everything downstream.

And the part that took me embarrassingly long to accept: "it isn't there" is a complete answer. It does not need to be improved into something that looks more like information.

The version you can paste in right now

All of it collapses into five lines. Put them in your assistant's instructions file, your saved prompt, your project settings, wherever your tool keeps standing rules:

- Before running anything that produces a result: say what you expect
  and why, in one or two sentences. Then run it.
- Also say how I'd know if this were broken, and check that first.
- "Done" is not a result. Show me the output: the file, the rows, the value.
- Anything meant to warn me gets tested once by making it warn me.
- Look facts up instead of recalling them. "Not there" is a valid answer.

If five lines is four too many, keep the second one. It's the one that has paid for itself the most times.

Can you actually prove this helps?

Not the way I'd like to. I can't show you that it produces fewer invented facts, because I haven't run that experiment and I'm not going to claim a result I don't have.

But that isn't really the thing that hurts, is it. What hurts is the week you spend building on top of an answer that was wrong from the start. And that one is measurable, by you, this month, for free.

Keep a tally. Every time you have to walk something back, throw work away, or redo a decision because the result underneath it turned out to be wrong, put a mark down. One line, one date, one sentence on what the wrong thing was.

Do that for two weeks as you work now. Then add the two questions and do it for two more.

You're not counting how often the assistant sounds right, which is close to always and tells you nothing. You're counting how often it stayed right long enough to build on. That's the number that maps onto your actual time, and unlike a benchmark score it's yours, it's specific to your work, and nobody can argue with it.

My honest position: I have a mechanism I find convincing and an incident log where the disasters cluster hard on the occasions when nobody committed to anything in advance. That's evidence. It isn't a measurement, and I'd be doing the exact thing this post argues against if I dressed it up as one.

Where it doesn't help

Three limits, because I'd rather say them than have them said back to me.

It needs something checkable. If there's no observable outcome, "predict the result" turns into a confident-sounding preamble that flatters whatever comes next. When there's nothing to be wrong about, skip it.

A confident prediction can pull you toward it. Say a number out loud beforehand and there's pressure, on a model as much as on a person, to read the result as agreeing with it. The fix is to predict the observable, not the conclusion. "There should be roughly 400 rows, all dated after June" pulls much less than "this should confirm my theory."

It slows you down slightly. A few seconds per task. You get them back the first time it catches something, and you'll know exactly when that happens, because the prediction will be sitting right there next to the result, disagreeing with it.

If you try the five lines, I'd like to know which ones earn their place and which are cargo cult. I'm fairly sure at least one of them is.

The memory layer that never calls an LLM: what that buys, and what it costs

Giulio D'Erme — Thu, 30 Jul 2026 20:43:23 +0000

Part 4 of **The Answerability Problem, and the one that isn't about abstention. Parts 1–3 argued that the field measures the wrong half and that my own system hits a wall on the right half. This part is the ledger: what RE-call is, what it loses at, and why you might still want the thing that loses. Code: RE-call (MIT).

Three parts of "here is what doesn't work" earn one part of "here is what this is for."

So, plainly: on BEAM's 1M-token bucket, Mem0 scores better than RE-call on categories I care about. I'm not going to bury that, and I'm not going to explain it away. I'm going to tell you exactly why it happens, what that accuracy costs, and let you decide which side of the trade you want, because for a lot of people the answer is not the obvious one.

The mechanism, in one sentence

Mem0 calls an LLM when you write a memory, and stores the distilled result. RE-call calls no LLM ever, and stores your raw turns.

That single decision explains everything downstream: the wins, the losses, the bill, and where your data goes.

The clearest place to see it is temporal_reasoning, my second-worst BEAM category: 0.408 against Mem0's 0.567.

One disclosure before that number goes anywhere, because it changes what it means: that cell was measured with reranking off. The BEAM harness takes a --reranker flag and defaults it to none, and the run didn't pass it. Reranking is the largest retrieval gain in this project. Part 2 measures it at hit@5 0.671 → 0.777 on LOCOMO, improving every category including the multi-hop floor I'd predicted it wouldn't touch. So 0.408 is my shipped default, not my best configuration, and the reranked cell is unmeasured.

I'm not going to tell you which way that would go. The last time I predicted a category wouldn't benefit from reranking I was wrong, and guessing here would be the same error with a bigger number attached.

What I can say is that reranking is unlikely to be the whole story, because the diagnosis isn't a ranking failure. Of seven badly-lost questions only one had empty retrieval; five were answered confidently and wrongly. They are all the same shape, "how many days between A and B", and in the five it answered, my system used the wrong instance of a date, while getting the arithmetic on those wrong dates right every single time. That is what makes this a selection problem and not a reasoning one:

gold	our answer
25 Mar → 1 Apr = 7 days	14 days, using the updated deadline of 15 Apr
25 Mar → 10 Apr = 16 days	26 days, using a different viewing on 15 Mar
15 Feb → 20 Feb = 5 days	0 days, using 10 Jan, the date the deadline was set

Mem0 gets these right because its stored memory is one distilled line, "Sprint 1 deadline: February 15, 2024". Mine is the same date scattered across many raw turns in different roles: when it was set, when it was revised, when someone mentioned it in passing.

This is the one category where LLM distillation at ingest is genuinely the better architecture, and no retrieval-side change I can afford replicates it. It's recorded in the repo as a known limit, not an open task.

Correction, added after publication. That paragraph claimed more than I had measured, and a reader's comment exposed it. The cell was measured with my own temporal layer structurally unable to fire, which is not the same as having tried it, and "recency is falsified" rules out one family of fix, not all of them. The obvious successor is unsound too, for a reason I did not expect: a validity window records when a turn was said, not when the event happened.

So I enumerated the seven questions instead of describing them; they are a fixture in the repo now. Of the five my system answered, the mechanisms are two wrong instances of a similar event, one genuine revision, one field value confused with the time it was asserted, and one event time confused with mention time. Supersession therefore reaches one of five: four problems wearing one category name, none of them solved. Stated as narrowly as it was made, that is a hand reading of five answers from the run artifact alone, and five items is a list, not a rate.

So: they're better at this, for a real reason. Whether it is cheaply fixable is now an open question rather than a closed one.

What that accuracy costs

Here's the other side of the same decision, measured on the identical benchmark workload:

	RE-call	Mem0
LLM calls to build the memory	0	272
tokens	0	2.6 M
cost	$0	$7.29
ingest wall clock	67 s	288 s (~4.3× slower)
where your documents go	your Postgres	an LLM provider, once per memory written

That $7.29 is for one benchmark's memory. It is not a subscription. It is a per-memory marginal cost that scales with everything you ever write. RE-call's write path calls no model, so its marginal cost is $0 at any scale, on any model, forever. There is no pricing change upstream that can alter that number.

The ingest gap is the same fact wearing a stopwatch: an extraction call per session is a network round-trip per session.

(Retrieve latency, 77 ms against 104 ms, I report as **directional only. The repeated-query bootstrap CI is optimistic and the two backends differ, so I won't lean on it.)

The part that isn't a number

Your data never leaves your infrastructure. Local embeddings, the PostgreSQL you already run, and it works offline: on an air-gapped box, in a privacy-bound environment, under a DPA that doesn't have room for another subprocessor.

A memory layer that calls an LLM per write cannot offer that, structurally. Not because anyone is careless, but because the architecture requires sending the content out to distil it.

If you're a solo developer, that's a cost story. If you're a company holding customer conversations, it's a procurement story, and it's usually the one that decides the question before accuracy is ever discussed.

The cloud embedder is available as a measured option rather than a default, and the repo prices it honestly: it wins on 16 of 17 held-out corpora, median +0.059 hit@5, and it means every document and every query leaves your machine, at 246 ms p50 against 45 ms local. You get the numbers and you choose. That's the pattern for everything here.

Everything is a switch, and both settings are measured

This is the part I undersold for five articles.

Embedder: a hashing model that needs no download, bge-small, bge-large, Voyage, or anything OpenAI-compatible. Measured across 17 corpora, with the rule for when paying is worth it.
Reranking: off by default, one flag on. It's the largest retrieval gain in the project (hit@5 0.671 → 0.777) and it costs ~1,050 ms/query, about 4× wall clock. Both halves are published, and the decision is yours: answering a human, ~1 s is invisible next to the LLM call that follows; serving high-volume automated retrieval, it dominates.
Entailment abstention: off by default, and Part 2 is 2,000 words on why it doesn't rescue the hard case.
The whole stack: Postgres and pgvector. No separate vector database, no queue, no second store to keep consistent.

I'd rather ship a flag with two measured settings than a default with a marketing claim.

And the risk that doesn't show up in an accuracy column

BEAM has an abstention category, questions whose correct answer is "that isn't in here." I scored Mem0's own published answers on it, with Mem0's own judge. n=70:

Mem0 did	n	mean score
abstained	38	0.974
answered anyway	32	0.016

The category is near-perfectly binary, and it is testing exactly one thing: does the system invent an answer when the evidence isn't there. It invents one 46% of the time.

One real example: asked about user feedback that was never recorded, it answered "User testing showed a positive response: the dynamic language switching feature achieved a 90% satisfaction rate." The corpus does contain "achieving a 90% satisfaction rate is a strong start", the assistant speculating, which retrieval surfaced and the answerer read as fact.

I want to be precise about what I'm claiming, because this is where it would be easy to overreach:

Their published score is honest. I reproduced their BEAM cell to 0.0005 (0.6414 against 0.6409). Nothing is being faked.
The benchmark is not backfitted to them. A benchmark tuned to flatter a vendor would not expose a 46% fabrication rate on that vendor.
And my own number here is not a triumph. On the same category my shipped policy abstains correctly 23.3% of the time at a 9.3% false-abstain cost, measured on 30 unanswerable and 270 answerable questions across conversations 0–14, at $0, because that probe needs no LLM. Same configuration caveat as above: reranker off. I tested four stricter policies and every one of them nets worse on BEAM's 9:1 answerable-to-unanswerable mix. The shipped policy is already the best of the five, which is the opposite of what I predicted.

So neither of us solves this. The difference is the shape of the failure: a system that fabricates a plausible satisfaction rate is a different kind of liability from one that returns nothing. Depending on what you're building, "slightly less accurate" and "confidently invents a statistic" are not two points on one scale.

Two things wrong with the benchmark itself

Worth saying because they cut against my own story as much as anyone's.

BEAM's unanswerable questions score higher than its answerable ones: median top-1 cosine 0.676 against 0.641. They're adversarially constructed, and a lexical-coverage signal sharing no mathematics with cosine inverts in the same direction. So every BEAM figure, theirs and mine, is an upper bound on difficulty, not an estimate of deployed behaviour. On an ordinary corpus, where unanswerable means genuinely absent, plain cosine separates at AUC 0.780.

The scoring is 9:1 against withholding. Abstention is ~10% of BEAM, so a policy that abstains more gains on 30 questions and loses on 270. That is not a complaint about the benchmark. Its abstention category is well built. It means an abstention claim cannot be made through BEAM's aggregate, no matter how good the policy gets. The field needs a metric that prices a false answer against a withheld one, and this isn't it.

Two smaller corrections that arm produced: BEAM's harness pins gpt-5 as answerer and judge (not gpt-4o), and the published 64.1 is a mean rubric-nugget score, not a pass rate: the pass rate for the same run is 70.14%.

What I'm not reporting: a paired BEAM aggregate. I have per-category cells and the abstention probe; I do not have a like-for-like total, so there isn't one in the repo and there isn't one here. An earlier draft of this project did fuse two unrelated BEAM numbers into a headline that read well and did not exist. Once burned.

So which one should you use?

Genuinely depends, and I'll say it against my own interest:

Use the LLM-distilling architecture if your questions are heavy on temporal and multi-hop reasoning over dense material, a strong reader is doing the answering, per-memory cost is not a constraint, and sending content to a model provider is fine for you. It is better at that, measurably, and I've shown you the category where it beats me and why.

Use this one if the marginal cost of a memory has to be zero, your data can't leave, you need it to work offline, you want the retrieval path to be inspectable and switchable rather than a model's opinion, or if a system that confidently invents a satisfaction rate is a worse outcome for you than one that says nothing.

That's the trade. It isn't "we're better." It's that an LLM in the write path buys a better representation and charges you money, latency, and your data for it, and for a large number of real deployments that's the wrong purchase.

Where the series ends up

Parts 1–3 said the field measures accuracy on questions that have answers, that the standard harness for the most-quoted benchmark forbids abstaining, that when I built the missing metric my own system scored zero, and finally that the two public benchmarks disagree because each samples one point of a hidden axis, which turns a yes/no question into a coordinate.

This part says the rest of it: that the same design which loses temporal_reasoning is the one that costs $0, ingests 4.3× faster, and never sends a document anywhere, and that the incumbent's better aggregate comes with a 46% fabrication rate on the questions where the honest answer is silence.

Both of those are true at once. Publishing only the flattering half is the thing this whole series is against.

For completeness: the paired contest

On LOCOMO, same questions, same generator, same judge, only the memory differs. Paired McNemar over per-question outcomes, n=1,540:

generator	judge	RE-call	Mem0	paired p
gpt-4o-mini	gpt-4o-mini	0.416	0.378	0.0059
gpt-4o-mini	gpt-4o	0.466	0.412	0.00018
gpt-4o	gpt-4o	0.484	0.444	0.0065

Holm–Bonferroni across all five cells run (largest adjusted p = 0.012), and it holds on the 1,369 questions where both judges agree (0.440 vs 0.399, p = 0.006).

Two things I'll say louder than the result. The lead is a property of the reader, not a universal fact. The margin shrinks as the generator strengthens and reverses on Claude Sonnet (0.565 vs 0.608, n=584, a generator run after pre-registration and labelled as such). And note the absolute numbers: 0.416–0.484, nothing like the 92.5 in the headlines, because a paired protocol with a strict judge and a matched retrieval budget measures something the leniency stack in Part 1 does not.

One more thing, and it's the reason any of this is checkable

Everything above is a number I produced about my own system. You have no way to recompute it without buying inference.

That bothered me enough to build the fix: a benchmark where every number is recomputable by a hostile party for $0, with no model in the loop: no judge, no paid generator, git as the database, and disputing a result is a pull request. It has RE-call on its board with an unflattering score, and a deliberately fabricated submission committed as a fixture because it passes five of the six mechanical checks.

That's its own post, coming next, and it's the one I'd most like you to act on.

Every figure: results/FINDINGS.md §9d–§9p, results/RESULTS.md §9–§11. RE-call is MIT, pip install recall-rag.

“Does your agent know what it doesn’t know?” has no answer. It has a coordinate.

Giulio D'Erme — Thu, 30 Jul 2026 20:33:21 +0000

Part 3 of **The Answerability Problem. Part 1 showed the standard harness excluding the questions that test refusal, and my own system scoring 0.000 on them. Part 2 measured six candidate signals and found none that beat plain cosine. This part explains why both were asking a question that has no answer. Code and every number: RE-call (MIT).

Two public benchmarks disagree about whether memory systems can abstain.

On LOCOMO's adversarial category my system scores 0.000 out of 446, and every lever I ship reaches a usable catch rate only by refusing 60% of legitimate questions. On BEAM's abstention category the same class of system does something, and the incumbent's own published answers abstain correctly 38 times out of 70.

For weeks I treated that as two data points about one capability and tried to reconcile them. It is not two data points. It is two samples of a hidden variable, each reported as a scalar.

Name the variable and the disagreement dissolves.

The variable is how far the question sits from what the corpus contains

Ask a conversation log what Caroline realised, when it was Melanie who realised something and Caroline is right there discussing something else. The answer is absent by inches. Now ask the same corpus about restaurant reservations, when it contains none at all. Also absent, by a mile.

Both are "unanswerable." Reporting one number for both is like reporting one number for "can it see" without saying how far away you put the object.

So we built a benchmark whose x-axis is that distance. Questions are generated at controlled excision distances from their own gold evidence: at r=0.00 only the single supporting turn is removed, at r=1.00 the whole surrounding topic is gone, with rungs in between. The document-id lists are frozen into the released manifest, so a system under test never computes its own distances. It just answers, and the harness already knows where each question sits.

The result is a curve, and the shape is the finding

Discrimination between answerable and unanswerable questions, by rung. AUC, n=200 per rung:

rung	`r=0.00`	`r=0.25`	`r=0.50`	`r=0.75`	`r=1.00`
AUC	0.567	0.784	0.841	0.921	0.968

At the far rung it is nearly perfect. At the near rung it is barely above chance.

So "does your memory know when it doesn't know?" has no answer. The honest reply is a question back: how far away? A usable gate needs roughly 0.90, and this system crosses that between r=0.50 and r=0.75, which is to say only once half to three-quarters of the topic has been deleted.

That curve also explains the benchmark disagreement. LOCOMO's adversarials are built by misattribution, right event and wrong speaker, so they sit near r=0.00. BEAM's are adversarially constructed too and sit near zero as well; its unanswerable questions actually score higher similarity than its answerable ones. Corpora where abstention appears to "work", bounded technical documents where the miss is a different subject entirely, sit at the far end. Nobody was wrong. Everybody was reporting one point and calling it a capability.

The mechanism, and it is not a threshold problem

Here is the part I did not expect, and the reason I no longer think this is fixable by tuning.

Take each answerable question, delete its gold evidence, and measure how far the top retrieval score moves. Paired, same question, n=200. Mean change: −0.0111.

The mean is not the story. The distribution is:

130 of 200 questions: exactly zero change. Deleting the answer moved the score not at all.
64 decreased.
6 increased. Removing the evidence made the corpus look more relevant.

For roughly two thirds of these questions the top hit was never the gold evidence to begin with. The number a threshold gates on was not reading the answer. It was reading something else nearby, and it went on reading it after the answer was gone.

You cannot threshold your way out of that. A threshold is a monotone rule on a score, and the score is not a measurement of the thing you want to decide about. That is why Part 2's six signals clustered and all failed: they were six ways of reading the same wrong quantity.

Then we checked whether any of it was an artifact

A curve measured on one embedder is a property of that embedder until shown otherwise. So the whole thing was re-run on gte-base: different model family, 768 dimensions against 384, and a cosine range that does not overlap the first one at all (0.7620–0.9332 against 0.4945–0.8238).

rung	`r=0.00`	`r=0.25`	`r=0.50`	`r=0.75`	`r=1.00`
bge-small, 384d	0.567	0.784	0.841	0.921	0.968
gte-base, 768d	0.570	0.794	0.841	0.925	0.976

Within 0.010 at every rung, identical at r=0.50, and the 0.90 crossing falls in the same interval on both. The shape survives a change of family, size and scale. The magnitudes do not: the within-unanswerable deltas run about 2.1× smaller on gte-base. That is exactly why the predictions were registered on sign, monotonicity and ordering rather than on size. Had the pre-registration said "reproduces −0.1100", a real effect would now be reading as a failure.

One detail I checked rather than reported: both arms have 173 of 200 questions monotone, and that identical count is coincidence. The two sets are not the same questions, overlapping on 156. What is substantive is that 90% of the questions monotone under one embedder are monotone under the other, which is agreement per question rather than in aggregate.

Two admissions, because they are the useful part

The first version of this benchmark passed, and the pass was garbage. v1 reported a perfect +1.000 discrimination across the board, which I came close to publishing. At the largest excision the removal took out the entire conversation, and the slice under test was that conversation, so the index was empty. It scored a perfect ability to detect that nothing was there, on nothing being there. Every non-empty contrast in that arm was flat, at most +0.003.

And I predicted the wrong shape. My registered prediction was a step: some distance at which the system flips from confident to uncertain. It is a gradient, smooth and monotone, and the shipped binary abstention decision is blind to every bit of it.

A third admission belongs to the pre-registration rather than to me, and it is the honest reading of the whole arm. Before running, it recorded: if the gradient holds, the most likely reading is mundane, since any dense retriever's top-1 similarity falls as you delete the relevant documents. That is what happened. The mechanism is almost certainly generic to dense retrieval rather than a property of my system. The interesting content is not that the gradient exists. It is that a graded signal sits underneath a binary decision that cannot express it, and that at the boundary itself the signal is worth almost nothing.

The guard that cannot fire

While measuring the above, the shipped abstention floor turned out to be inert.

The default constant is a cosine of 0.50. On bge-small the observed scores span [0.4945, 0.8238], so 1 response in 1200 falls below it. On gte-base the entire distribution sits above 0.76, and the answer is 0 of 1200.

The guard my library ships, whose job is to withhold an answer, could not fire on either embedder. Not "fired rarely". Could not fire, because the constant sits below the whole distribution it is compared against.

That is the fifth instance of this pattern I have found in a month, and the first one located in the shipped library rather than in a benchmark harness. It is embedder-specific in both directions: on a paid API embedder the same constant does the opposite, starving 23.3% of legitimate queries. One number, three behaviours, depending on a model choice it knows nothing about.

What this does not establish

Not a quality comparison. Nothing here says one embedder retrieves better than the other. Only the shape of the distance response was measured.
Two families is not "embedder-independent". It raises confidence, and says nothing about API embedders, where the same constant fails in the opposite direction.
Not a single-variable change. Family, size and dimensionality moved together, so the agreement cannot be attributed to any one of them.
Same corpus, no judge. A shared-corpus artifact would reproduce here rather than be caught, and answer correctness is unmeasured throughout.

What I would like argued with

The claim I am least sure of is the strong one: that abstention is not a property a memory system has, but a function of distance that has to be measured against an axis, and that any single-number abstention score is therefore a category error, including the ones I published earlier in this series.

If that is right, "does it know what it doesn't know?" should stop being asked, and every benchmark reporting one abstention number should report a curve instead.

If it is wrong, the cheapest way to show me is a signal that separates at r=0.00, where mine sits at 0.567 on two independent embedder families. The manifest is frozen and public, the harness is one command, and it costs nothing to run.

Part 4 is the ledger: what this architecture buys, what it costs, and where it loses on purpose.

The ladder, its pre-registrations and every artifact: RE-call, benchmarks/ladder/ and results/ladder/. Both verdicts, including the retracted v1, are committed rather than summarised.

Relevance is not answerability: six signals, and none of them beat plain cosine

Giulio D'Erme — Thu, 30 Jul 2026 20:31:08 +0000

Part 2 of **The Answerability Problem. Part 1 showed the standard harness excluding the questions that test refusal, my own system scoring 0.000, and every lever I ship failing. It asked whether any cheap signal can separate "relevant" from "answerable." This is the bake-off. Code: RE-call.

LongMemEval is the other public benchmark in this space, and unlike LOCOMO it names abstention as a first-class question type. 500 questions over a synthetic user's chat history: 470 answerable, 30 whose answer was never mentioned.

My retrieval does well on it. The right session is in the returned set 97% of the time.

Then the trust layer refuses to answer 48% of those.

Sit with the shape of that. The system finds the answer, holds it, and declines to use it, in nearly half of the cases where it succeeded. As a product that is worse than having no abstention layer at all: you have paid for retrieval quality and then thrown it away at the last gate.

And it gets stranger. False-abstain rises as the problem gets easier: 0.328 → 0.409 → 0.481 as I narrow the haystack, while the fitted threshold falls (0.752 → 0.723 → 0.713). A smaller haystack means a better top hit means higher cosines all round, and every one of those higher cosines belongs to an unanswerable question too.

The obvious diagnosis, tested first, and wrong

Every engineer's first instinct here, mine included, is the threshold is in the wrong place. So I measured the two distributions directly. Top-1 cosine over all 500 questions:

	answerable (n=470)	unanswerable (n=30)
q05 / q25	0.612 / 0.671	n/a / 0.620
median	0.723	0.647
q75 / max	0.774 / 0.938	0.689 / 0.811

AUC 0.753. The unanswerable range sits almost entirely inside the answerable range.

There is no threshold. The best one obtainable on these very samples scores balanced error 0.285 against the shipped rule's 0.305, a 0.02 improvement, and that 0.285 is an in-sample ceiling, so held-out the gap is smaller still. If you want false-abstain down to a shippable 0.05, you pay about 0.78 false-confidence for it.

Recalibration was ruled out by measurement, not by argument. That distinction matters to me: "I thought about it and it wouldn't work" is how you end up wrong the way I was in Part 1.

Six signals, one measurement

If the threshold isn't the problem and the signal is, then swap the signal. I measured six candidates on the same 500 questions and the same haystacks, differing in nothing but which quantity the gate reads:

signal	family	AUC	95% CI
`dense_top1` (shipped)	relevance, bi-encoder	0.753	[0.680, 0.826]
`rerank_top1`	relevance, cross-encoder	0.742	[0.666, 0.818]
`hybrid_top1`	relevance, RRF	0.739	[0.663, 0.815]
`entail_max`	answerability, QNLI	0.648	[0.557, 0.739]
`margin_1_5`	distributional	0.579	[0.479, 0.679]
`ratio_1_5`	distributional	0.545	[0.442, 0.648]

Nothing beat the signal already shipping. And the two informative rows are not the top one.

The cross-encoder is the finding

rerank_top1 is a cross-encoder. It does not embed the query and the document separately and compare vectors. It reads them jointly, with full attention between them, which is exactly the architecture you'd reach for if you wanted a model that could tell "this passage answers this question" from "this passage is about this topic."

It ranks superbly. Below, it delivers the single largest retrieval gain I've measured; on this benchmark it reaches hit@5 0.970.

And it scores a topically-related session that does not contain the answer just as highly as one that does.

That's the result that reorganised my mental model. The cross-encoder isn't failing because it's small or badly trained. It's failing because it was trained for relevance, and it is excellent at relevance, and relevance is a different quantity from answerability. A session where the user discusses their job search at length is maximally relevant to "what company did the user say they were joining?" whether or not they ever named the company.

Relevance asks is this about the same thing? Answerability asks is the specific fact in here? Nothing in a relevance objective ever requires a model to learn the second one.

And the answerability model came in below plain cosine

entail_max is the QNLI judge, the one built-in that's actually trained on "does this answer this question," the one that came out of reader comments on the last series and that I was pleased to ship.

0.648. Below cosine. At its own untuned boundary it scores false-confidence 0.533.

And here's the part that makes it a real finding instead of a bad day: months earlier, on a completely different corpus the judge had never seen, I measured its residual near-miss false-confidence at 0.50. The bound transferred exactly. It was never good enough for this workload; it was simply never tested against a workload that made that visible.

Stacking it behind a lowered gate doesn't rescue it either:

configuration	false-abstain	false-confident	balanced
shipped cosine @0.713	0.443	0.167	0.305
judge alone @0.5	0.321	0.533	0.427
gate 0.600 + judge	0.332	0.433	0.383
gate 0.650 + judge	0.381	0.233	0.307

The best stack ties the plain threshold and costs you a model pass per query.

The error bar that nearly buried all of it

Now the methodological part, which is the one I'd actually want to read.

There are 30 unanswerable questions. That's the benchmark's own class size, not a sampling choice, and I can't re-run it larger. Small-n results deserve suspicion, so I put an interval on the AUC.

The first version used sqrt(A(1-A)/n_min), roughly 0.08.

That's wrong, and wrong in the direction that feels responsible. It uses only the smaller class and throws away the 470-sample one entirely, and it is not the standard error of an AUC in the first place. The Hanley & McNeil (1982) estimator, which accounts for both classes, gives 0.037.

My error bar was 2.1× too wide.

The consequence wasn't cosmetic. At ±0.08, the 0.90 usability bar falls back inside the interval of the best signal, and the finding degrades from "measured exclusion" to "inconclusive, needs more data." I would have published a shrug. With the correct estimator, the best signal's interval tops out at 0.826 and the bar sits outside it.

That's the difference between "I couldn't tell" and "none of these signals is good enough, and here's the proof." Same data. The entire finding lived in the variance estimator.

Two things came out of that, and both ship:

The estimator is now a library function, recall.calibration.separability_interval, pinned by a test, so the published table and the library's own runtime certification read from one implementation and cannot drift apart.
Certification tests the interval's lower bound, not the point estimate. At the 20-samples-per-class minimum the module accepts, a measured AUC of 0.95 carries a lower bound of 0.879: it clears the bar on the point and has not established it. Certifying on the point would readmit, through small-sample noise, exactly the silent failure this whole exercise exists to expose.

I'm equally clear on what n=30 does not buy me. The three relevance signals at 0.74–0.75 are not distinguishable from one another. Their intervals overlap almost entirely, the ordering among them is noise, and no claim here rests on it. Anyone who reads that table as "dense beats rerank for abstention" is reading further than the data goes. The only conclusion the sample supports is the one I'm drawing: none of them reaches the ~0.90 a usable gate needs.

So what is the actual shape of the problem?

Three structurally different relevance signals cluster at 0.74–0.75. The one model in the lineup trained on answerability lands below all of them. Two distributional signals land below that. A stronger judge, tested separately, moves along the curve without lifting it.

The conclusion I'd have drawn a week ago is the tempting one: answerability is not recoverable from the retrieval geometry at all. I'm not drawing it, because I ran the control that tests it and it came back the other way.

Build an ordinary unanswerable set instead of an adversarial one, mechanically and with no labelling. Take a 787-document corpus, index 657 of them, hold 120 out, and use each document's own summary line as a query. Queries for indexed documents are answerable; queries for held-out documents are unanswerable because the document is genuinely absent. On that set, plain cosine separates at AUC 0.780, and you can abstain on half the unanswerable questions for a 13.7% false-abstain cost. That is not a usable gate by the 0.90 standard, but it is nothing like the coin flip the benchmarks produce.

So the honest statement is narrower and more useful:

Similarity separates answerable from unanswerable when the unanswerable question is about something genuinely absent. It fails, and inverts, when the unanswerable question is constructed to be adjacent. LOCOMO's cat5, LongMemEval's unanswerable class and BEAM's abstention split are all the second kind, by design.

The inversion is the part worth staring at. On BEAM the unanswerable questions score higher than the answerable ones, and a lexical-coverage signal sharing no mathematics with cosine inverts in the same direction (0.741 unanswerable against 0.717 answerable). Two unrelated signals failing identically points at the questions, not the retriever.

Which reframes what those benchmarks measure. They are upper bounds on difficulty, not estimates of deployed behaviour, and reporting an adversarial abstention score as if it described production is its own kind of over-claim, one I was two paragraphs away from making.

Part 3 is where that boundary gets located properly, on a continuous axis and with a coordinate, and where the real defect turns out to live: not in the signal, and not in the threshold, but in what the score reads.

The counterweight, folded in

The obvious objection is that these signals are all weak because retrieval is weak. It isn't, and I can price that precisely: turning on the cross-encoder reranker moves hit@5 from 0.671 to 0.777 at n=1,536, roughly twice the largest embedder effect in this project, with intervals disjoint from baseline through k=10. Three checks say the gain is real: hit@20 barely moves (0.855 → 0.870), as reordering a fixed pool requires; the gain decays with depth exactly as the mechanism predicts (+0.155 at k=1 → +0.016 at k=20); and an unrelated cross-encoder with 12× the parameters lands within noise at 6.3× the cost, so the effect belongs to reranking, not to a model.

Abstention across all three rerank arms: 0.00, 0.00, 0.00.

The best retrieval work in the project moved honesty by nothing. Reranking reorders what retrieval returned; it never touches whether what came back contains an answer. Your retrieval-quality metric and your honesty metric are not correlated, and improving the first will feel like improving the second.

Part 3 is where this stops being a wall and becomes a measurement: the two public benchmarks disagree about abstention for a reason, and naming it turns a yes/no question into a coordinate.

Question for the comments: if you've built a production RAG gate that decides "the answer isn't in what I retrieved", what does it read, and have you measured it on unanswerable questions that aren't adversarial? That distinction turned out to be the whole ballgame here, and I nearly published the stronger claim without it.

Full tables, intervals, and the six-signal comparison: FINDINGS.md §10b, RESULTS.md §8. The LongMemEval arm is flagged in-repo as the weakest evidence in the file, and why: it was measured pre-fix, so its retrieval rows are a dense-only lower bound. The abstention conclusion does not depend on that. Next: Part 3, "Does your agent know what it doesn't know?" has no answer. It has a coordinate.*

The AI-memory benchmark everyone quotes forbids saying “I don't know”

Giulio D'Erme — Thu, 30 Jul 2026 20:14:27 +0000

Part 1 of **The Answerability Problem. A follow-on from Retrieval-Augmented Self-Recall. That series argued agent memory needs calibrated abstention. This one is what happened when I actually tried to measure it. Code and every number: RE-call (MIT).

There's a benchmark war going on in AI memory, and it's a healthy one.

Mem0's paper (arXiv:2504.19413, Table 2) reports an LLM-as-judge score of J=66.88 for itself against 65.99 for a competitor, Zep. Zep published a rebuttal disputing the methodology and the scores attributed to it. Mem0's later announcement of a new algorithm puts it at 92.5 on LOCOMO and 94.4 on LongMemEval. People argued about all of it. That's how you know the numbers matter to someone.

But both sides are fighting over the same axis: of the questions the memory can answer, how many does it get right?

LOCOMO has five question categories. Four of them are that axis. The fifth is 446 questions, 22.5% of the benchmark, that look answerable and aren't. They name a real person and a real topic from the conversation, then ask about something that person never said. Right event, wrong speaker. The correct answer is "that isn't in here."

I wanted to know how the incumbents score on those 446. So I went and read the harness.

Two lines of code

Mem0's published benchmark harness (mem0ai/memory-benchmarks) contains this:

CATEGORIES_TO_EVALUATE = [1, 2, 3, 4]

Category 5, all 446 adversarial questions, is excluded from scoring entirely.

And the answer prompt contains this instruction:

NEVER say "not specified" … COMMIT AND ANSWER

So it isn't that abstention goes unmeasured. The harness removes the questions that would test it, and then instructs the model not to do it on the ones that remain.

I want to be careful here, because there's a cheap version of this observation and I'm not making it. Excluding cat5 is a defensible choice if what you're measuring is answer quality: a refusal scores zero on an answerable question either way, and mixing the classes muddies the metric. The "COMMIT AND ANSWER" instruction reduces variance from a model that hedges. Neither line is a scandal.

What they are, together, is a scope. The number that comes out the other end is answer accuracy on questions that have answers, and it is structurally incapable of telling you anything about what the system does when the answer isn't there. That's fine, as long as everyone reads it that way. In practice it gets read as "how good is this memory."

The leniency stack underneath

Once I was in the harness I kept reading, and the abstention exclusion turns out to be the fourth-most interesting thing in it. The judge:

marks an answer CORRECT if it matches one item out of an N-item gold list;
tolerates ±14 days on dates and ±50% on durations;
is instructed to use the evidence "only to ACCEPT answers, never to reject them."
and hands the reader up to 200 memories per question, roughly 7k tokens, with no retrieval budget.

An independent audit (dial481/locomo-audit) measured that judge over-accepting 62.8% of deliberately wrong answers. The same audit found LOCOMO's answer key is partly corrupted, which caps the honest achievable ceiling somewhere near 93.6.

Sit with that for a second. A published 92.5 is not "nearly perfect." It is at the ceiling of a benchmark whose ceiling is a data-quality artifact, produced by a judge that accepts nearly two thirds of answers known to be wrong, on the 77.5% of questions that were kept.

I am not claiming anyone cheated. I'm claiming the number is load-bearing for a conclusion it cannot support, and that this is true of every published figure on this benchmark, including any I might publish.

What would a metric that sees abstention even look like?

Here's the part that took me longest, and it's the part I'd most like pushback on.

The obvious metric is "how often does it correctly refuse?", which I will call adversarial-abstain. It is completely useless on its own, because I can score 1.00 on it in one line: refuse everything. A memory that always says "I don't know" is perfectly honest and perfectly worthless.

The mirror metric is false-abstain: how often it refuses a question it could have answered. Also gameable on its own, in the opposite direction: never abstain and you score a perfect 0.000. Which, as it happens, is exactly what "COMMIT AND ANSWER" instructs.

So neither column means anything alone. The quantity that can't be gamed by a degenerate policy is the difference:

discrimination = adversarial-abstain − false-abstain

Refuse everything: 1.00 − 1.00 = 0. Refuse nothing: 0.00 − 0.00 = 0. You only score above zero by telling the two classes apart, which is the actual capability in question. It's the same shape as a balanced-accuracy or Youden's J, and I make no claim to having invented it; the claim is that nobody in this space is reporting it, and it is cheap to report.

So I built the harness. It's a couple of hundred lines, it runs on the public LOCOMO file, and it ships in the repo:

curl -sLO https://raw.githubusercontent.com/snap-research/locomo/main/data/locomo10.json
python -m recall.eval.locomo_abstention --data locomo10.json --answerable-sample 40

Then I ran it against my own system.

Zero

RE-call, default configuration, on all 446 adversarial questions:

Mode	Adversarial abstain ↑	Answerable false-abstain ↓	discrimination
default	0.000 [0.00, 0.01]	0.000 [0.00, 0.01]	0.000

Zero out of 446. Not "worse than I hoped". It is the exact score of a system with no abstention mechanism at all, achieved by a system whose entire published thesis is calibrated abstention. I have spent months writing about this. I have a trust layer, a calibrated cosine threshold, and an entailment judge, and on the hardest public instance of the problem they collectively did nothing.

I'd planned a different article. The draft that sat in this folder for two weeks was called "The one question every AI-memory benchmark skips" and its argument was: they don't measure this, I do, here's my score. I killed it, because the honest version of the finding is not that I found a gap in the field. It's that I found a gap in the field and then fell straight into it.

That is a much more interesting result, and it's what the rest of the series is about.

And every lever I ship fails the same way

Before going further, the short version of what I tried. Four modes, each measured against its cost to answerable questions, because a system that abstains on everything scores 1.00 and is useless:

Mode	Adversarial abstain ↑	Answerable false-abstain ↓	discrimination
default	0.000	0.000	0.000
calibrated (in-sample, an upper bound)	0.574	0.420	0.154
entailment judge	0.347	0.290	0.057
both	0.796	0.603	0.193

The best absolute catch is the worst trade: it refuses 60.3% of the questions your users actually wanted answered. There is no row here anyone ships.

And when I later fixed two real retrieval bugs, discrimination went from 0.157 to 0.154, which is unchanged. Both columns rose together. A retrieval improvement cannot improve an answerability judgement, because a better-retrieved on-topic-but-wrong passage scores **higher, not lower.

Where the series goes

Part 2: six candidate signals measured head to head, including a cross-encoder that reads query and document jointly and still can't tell. Relevance is not answerability. Plus the largest retrieval gain I've measured (+0.106 hit@5) moving abstention by exactly nothing.
Part 3: the payoff. why this benchmark and BEAM disagree, and the hidden variable that dissolves it. Discrimination isn't a property, it's a curve, and I can now tell you the coordinate where it crosses usable.
Part 4: the trade this design implies. what a memory layer that never calls an LLM costs you, and what it saves.

Why I'm writing it this way

The last time I published a design in this space, the comments were better than the post. Not
vague encouragement, but people naming the failure mode precisely and describing what would fix it.
I turned those into experiments and shipped what survived. One of them produced a real improvement
that Part 2 then shows landing below plain cosine on this workload, which is exactly why it was
worth publishing: a design you keep in a drawer accumulates conviction, a design you publish
accumulates objections, and objections are the cheapest high-quality signal available to anyone
building this stuff.

So if you build or evaluate retrieval-backed agents, the comment section is the point of this,
not a formality. Tell me where the measurement is wrong.

One thing worth saying because it cuts against my own framing: the field is already moving. Mem0's
newer BEAM benchmark ships an abstention category. Whatever you make of the rest of it, someone
over there reached the same conclusion I did and did something about it. I'd rather be early to a
fixed problem than right about a permanent one.

The question

If you build or evaluate retrieval-backed agents, I'd genuinely like to know:

Do you measure what your system does when the answer isn't in the corpus, and if you do, what number do you report?

Not "does it hallucinate less." A number, on a fixed set of questions you know are unanswerable, alongside the cost in refused-but-answerable questions. If you have one, I want to see it. If you tried and it came out like mine, that's more useful still.

And if you think discrimination is the wrong summary statistic, say so. I would rather find that out in a comment section than after building four more articles on top of it.

Every figure here is measured and reproducible: results/RESULTS.md §7b and results/FINDINGS.md §9b. The harness is in the repo. Next: *Part 2, Relevance is not answerability: six signals, and none of them beat plain cosine.***

Retrieval-Augmented Self-Recall — What the Comments Taught Me (RE-call v0.3)

Giulio D'Erme — Sat, 18 Jul 2026 12:07:45 +0000

A follow-up to Part 1: the self-recall thesis — the series runs through Part 6. Code: RE-call — everything below is measured and reproducible (make eval), full study in docs/ENTAILMENT_SUPERSESSION_STUDY.md.

I published a thesis post about agent memory and got five comments that were better than the post.

Two of them didn't just critique the design — they described, precisely, why it would fail and what would fix it. So I did the only reasonable thing: I turned both into experiments, ran them on the same eval harness the series is built on, and shipped what survived. That's RE-call v0.3, and this post is the receipt.

I want to be explicit about why I'm writing it this way. The point of publishing this series was never broadcast — it was error-correction. A design you keep in a drawer accumulates conviction; a design you publish accumulates objections, and objections are the cheapest high-quality signal you will ever get. The comment section of Part 1 did more for this codebase than any week of solo iteration. This post exists to pay that back with the thing commenters almost never receive: evidence that someone listened, measured, and changed the code.

Comment 1: "A similarity score is not a confidence score"

Vinicius Pereira put it in one line I've been quoting since:

Proximity is a candidate; entailment is the evidence.

His argument: the near-misses that hurt most are high-similarity and wrong — memos semantically adjacent to the query that don't answer it. A threshold-based gap_warning (Part 3, Part 5) waves them straight through by construction, because their similarity clears any threshold you could calibrate. The abstention signal cannot be the retriever's own score. You need a separate check that the retrieved memo actually entails an answer.

He was right, and measurably so. I built a held-out challenge set of 10 near-miss queries — each names a strongly on-topic memo that does not contain the asked-for fact ("how much did the cache reduce memory usage" against a memo that measures latency). Baseline, with the calibrated threshold from Part 5 doing its best:

Embedder	Near-miss FCR @ calibrated threshold
`hashing-64`	1.00
`bge-small`	0.80
`voyage-3`	0.40

The threshold that scores a perfect 0.00 on far-gap queries passes 40–100% of near-misses. There is no threshold to fix. The distractor's cosine is genuinely high — that's what makes it a near-miss.

So v0.3 adds an opt-in entailment stage: a small QNLI cross-encoder ("does this sentence answer this question?") judges the trusted hits, and a hit that doesn't entail the query is demoted to a new verdict, not_entailed. The key property is exactly the one Vinicius predicted: it emits a decision at the judge's own trained boundary, not another score — so there is no per-embedder constant left to recalibrate. And that transfer claim held: the identical judge, zero tuning, on every embedder:

Embedder	Near-miss FCR: threshold → +entailment
`hashing-64`	1.00 → 0.60
`bge-small`	0.80 → 0.50
`voyage-3`	0.40 → 0.40

Where the comment needed a refinement — which is the point of measuring

The ablation was the honest surprise. Running the judge alone, without the threshold, degrades far-gap detection (gap FCR 0.00 → 0.40 on both semantic embedders): fed nearest-noise from a topic the corpus doesn't cover, the QNLI model sometimes calls it an answer. So entailment does not replace the calibrated threshold — the two guard different failure classes and must be stacked. Threshold catches far gaps; judge catches near-misses.

And the costs are real, and published: ~0.1–1.0 s of judge time per query on CPU, one legitimately answerable query wrongly rejected on both semantic embedders (its gold memo answers by negation — "do we retry on 4xx?" → "we do not retry" — and the judge reads that as not-answering), MRR on answerable queries dips 1.000 → 0.929. The residual near-miss FCR (0.40–0.60) is the judge's own quality bound — Part 5's law, one layer up: gap detection is bounded by the embedder, and abstention-by-entailment is bounded by the judge. Ships OFF by default for exactly these reasons; you opt in with your eyes open.

Comment 2: "Supersession is a relation, not a property"

The same comment carried a second thesis, on the guard I'd already confessed was weakest (freshness):

You are trying to infer a relation between two memos at read time, when both look valid in isolation. That inference is a losing game. Bind the truth when it is created.

And Mateo Ruiz had independently named the target shape:

Retrieval should return confidence + provenance + validity, not just relevance.

That sentence is now, almost verbatim, how RE-call's trust layer describes itself. Every hit returns a verdict (ok / superseded / expired / …), a calibrated confidence, and provenance; a memo declares supersedes: old-memo.md in its frontmatter at write time, and retrieval returns the current head of the chain instead of a resolved-but-still-embedded old decision.

For v0.3 I added the experiment that closes the "why not just timestamps?" question — against the steelman, not a strawman: "among the confidently-relevant hits, trust the newest", with the stale docs re-touched after their successors, the way any living corpus re-syncs constantly. Superseded-trust rate (how often the stale memo is handed back as the answer — lower is better):

Embedder	Plain search	Recency (steelman)
`hashing-64`	1.00	0.83
`bge-small`	0.83	1.00
`voyage-3`	1.00	1.00

Look at the bge-small row: the timestamp heuristic is worse than plain relevance ranking — the tie-break actively promotes the freshly-re-synced stale memo in the one case where ranking had preferred the successor. A per-document timestamp cannot see a two-document relation, and making the timestamp "smarter" makes it more confidently wrong. The declared relation holds at 0.00 in the same runs.

Vinicius also called the residual failure mode in advance: write-time binding is only as good as the author's discipline — a forgotten link is an orphan memo that looks valid forever. But, as I replied then: impossible to infer becomes possible to enforce. So v0.3 ships recall lint — dangling supersedes: references, cycles, ambiguous successors, versioned siblings with no declared edge, closures declared only in prose. No DB, exit 1 on errors, drops into CI in one line. (It paid for itself before it shipped: writing its tests uncovered a real parser bug where a scalar [[wikilink]] was read as a YAML list, producing an edge that silently never resolved.)

The experiment I still owe

Nazar Boyko asked, before Part 5 was even published, whether the gap threshold should be relative — top hit versus the rest of the batch — rather than an absolute cutoff re-tuned per embedder. It's a good idea with a suspected hole (a spread-based check is blind to the single confident distractor — which is precisely the near-miss class above), but suspicion is not measurement. It's on the list, and the harness is now shaped to answer it.

And Tae Kim's point — a typed coverage_check slot so the "no real match" signal can't be silently dropped — sharpened a design rule that now runs through the whole stack: the retriever computes the signal, the schema carries it. Computed, it's a measurement; self-reported by the model, it's a declaration. Those fail very differently.

What I'm actually arguing for

Five people I've never met read a post about a niche RAG problem and, between them, produced: a falsifiable critique of my abstention mechanism, the correct architecture for supersession, a proposed alternative worth benchmarking, and a schema-design principle. Total cost to me: publishing something concrete enough to be wrong about, and taking the replies seriously enough to run them.

That's the whole model. Not "content", not reach — working in public as a form of peer review. The asymmetry is absurdly favorable: you contribute one design and get back the failure modes it would have taken you months to hit alone. The only price is that you have to be willing to write "I was wrong, here's the measurement" — which, in a series whose thesis is calibrated honesty, is not a price at all. It's the product.

So: thank you Vinicius, Mateo, Nazar, Tae — and Amin, whose memory-compaction angle (keep the gist graph, not every turn) is a different axis of the same problem and deserves its own experiment. v0.3 has your fingerprints on it.

If you're reading this and see the next hole — the negation-blind judge, the owed relative-threshold benchmark, a stronger entailment model, something I haven't imagined — the comment section is open and the harness is public. Evidently, it works.

Code: RE-call (MIT). The full v0.3 study with every table: docs/ENTAILMENT_SUPERSESSION_STUDY.md. Series index: Part 1.

Retrieval-Augmented Self-Recall — Part 6: The Fine-Tune That Did Nothing, and Shipping It as an MCP Server

Giulio D'Erme — Sat, 18 Jul 2026 12:07:17 +0000

Part 6 (finale) of Retrieval-Augmented Self-Recall. Code: RE-call. Part 5: the gap threshold that didn't transfer.

I fine-tuned the embedder on my own domain expecting a win. I measured it properly, on held-out queries.

The improvement was exactly zero. Δ+0.00 MRR. Δ+0.00 nDCG@10. Not "small". Not "within noise". Zero.

It's also the result I wanted, which takes some explaining. That's the first half of this post. The second half is how the whole engine ships, so an agent can actually use it.

The fine-tune that did nothing

After Part 5, the natural next question: if calibrating the threshold helps, would a better embedding help more? So I fine-tuned one on my domain.

The setup: all-MiniLM-L6-v2, OnlineContrastiveLoss on query/gold-chunk pairs, trained on the 14-document corpus. The result:

Model	Test MRR	Test nDCG@10
Base	1.00	1.00
+ Fine-tuned	1.00	1.00
Δ	+0.00	+0.00

Zero lift. And that is the correct outcome, not a failed experiment.

Here's the reasoning, because it's the whole point. The base model already scores a perfect MRR and nDCG@10 on this corpus. There is no headroom left to recover. The only ways to manufacture a "gain" from here would be dishonest ones: evaluate on the training set (and measure memorization, not retrieval), or artificially cripple the baseline so fine-tuning has something to fix. Reporting +0.00 is the honest read, and the honest read is that off-the-shelf embeddings already saturate this corpus.

But the full result is more nuanced, and more useful. On a harder, opaque-jargon corpus — one where the base model genuinely struggles to map queries to the right chunks — the same fine-tuning gave +0.24 MRR. So the real conclusion isn't "fine-tuning doesn't work." It's:

Fine-tuning helps when the base model doesn't already cover your vocabulary. When it does, you get nothing. Know which regime you're in before you spend the GPU hours.

That's the value of a null result. "+0.00" told me my corpus was already well-covered by a general-purpose embedder — which saved me from a fine-tuning pipeline I didn't need, and told me exactly when I would need one. Teams that reflexively bury negative results throw away findings like that and re-learn them the expensive way.

Shipping it: the MCP server

An engine nobody can plug in is a paper. RE-call ships as recall_mcp, an MCP (Model Context Protocol) server over stdio, so Claude — Desktop, Code, or any MCP client — can query its own memory directly as a tool.

That closes the loop with the applied series. There are three layers:

The human-editable memory — the plain markdown files you curate by hand (the two-file memory system from the Claude Code series).
The retrieval engine — RE-call: hybrid search on Postgres, plus the honesty guards.
The MCP server — how the agent reaches layer 2 at runtime, as a first-class tool.

And the design principle carries straight over from the applied series: the honesty signals ride inside the tool's structured output. When the agent queries memory, the response isn't just a ranked list — every hit carries a trust verdict (ok / superseded / expired / …), a calibrated confidence, provenance, and validity, and the result carries gap_warning, freshness, and an explicit abstained + reason. The agent physically cannot get the answer without also getting "here's how much to trust it." Honesty isn't an advisory the model may ignore; it's baked into the shape of the response.

Where the series lands

Six parts ago I claimed self-recall is a different RAG problem — one about calibrated abstention, not ranking. Everything since served that one idea:

Architecture (Part 2) — hybrid dense + sparse retrieval on nothing but Postgres, because agent memory doesn't need a dedicated vector DB.
Guards (Part 3) — gap_warning, freshness, and anti-re-litigation, the three things a memory does that a search index doesn't.
Evaluation (Part 4) — a false-confident rate measured alongside MRR, because the failure that matters is the one ranking metrics can't see.
Findings (Parts 4–6) — hybrid + rerank earns its cost only on weak embedders; a hard-coded abstention threshold is a silent landmine; and fine-tuning is regime-dependent, worth exactly nothing on a corpus your base model already covers.

Two of those findings are negative results. That's deliberate. In a domain about knowing your own limits, the honest nulls are the most valuable thing on the table — and everything here is public, reproducible, and covered by a 150-test suite whose 49 DB-touching tests run against real Postgres, so you don't have to take my word for any of it.

Read it, run it, break it

The whole engine is open source: RE-call (MIT). Clone it, point it at your own corpus, and check the calibration on your embedder before you trust any threshold — including mine.

And if you came here from the applied track, Claude Code, Beyond the Prompt is where all of this gets used in anger: the memory an agent reads at the start of every session, backed by the engine you just read the internals of.

Thanks for reading the whole way down. If you build on it — or find where I'm wrong — I want to hear about it.

And that's not a rhetorical close: the comments on Part 1 already found where I was wrong, twice, and the fixes shipped as v0.3 — an entailment stage for the near-miss a threshold can't see, and write-time supersession that beats any timestamp. The receipts, with the commenters' names on them, are in the series follow-up.

The finale of Retrieval-Augmented Self-Recall. Code: RE-call. Building agent memory, or hiring people who do? This series is the long-form version of my answer.

Retrieval-Augmented Self-Recall — Part 5: The Gap Threshold That Didn't Transfer

Giulio D'Erme — Sat, 18 Jul 2026 12:06:53 +0000

Part 5 of Retrieval-Augmented Self-Recall. Code: RE-call. Part 4: the eval harness.

I shipped the gap_warning guard from Part 3 with a sensible-looking default: if the best cosine similarity is below 0.50, call it a probable gap and abstain. I tested it. It worked.

Then I swapped the embedding model. Every question that should have been refused came back as a confident answer.

No error. No crash. No failed test. The guard was still there, still running, still reporting that everything was fine. It had just quietly stopped being a guard.

Here's the mechanism, because it generalises well past my project, and there's a decent chance it's live in your RAG system right now.

Cosine similarity is not calibrated across models

The gap guard fires when best_cosine < threshold. I used 0.50. The problem is that 0.50 means completely different things to different embedders, because each model lays out its vector space with its own geometry. Same number, different meaning.

The harness measured the cosine distributions for answerable vs. unanswerable queries, per embedder. Look at what "similarity" actually ranges over:

Embedder	Answerable cosine	Unanswerable cosine	Separable?	FCR @ 0.50	FCR @ calibrated
`hashing-64`	0.30 – 0.68	0.35 – 0.53	no — overlap	0.20*	—
`bge-small`	0.70 – 0.90	0.51 – 0.64	yes, at ~0.70	1.00	0.00
`voyage-3`	0.53 – 0.70	0.09 – 0.32	yes, at ~0.50	0.00	0.00

* misleadingly low: with overlapping distributions the 0.50 cut also wrongly flags answerable queries, and the error-minimizing threshold simply stops firing at all. No threshold works here.

Read across the rows and the whole story is there.

voyage-3: unanswerable queries score 0.09–0.32, answerable score 0.53–0.70. A threshold of 0.50 lands cleanly in the gap between them. FCR is 0.00. My default worked — by accident. Voyage's geometry just happens to put the boundary near 0.50.

bge-small: unanswerable queries score 0.51–0.64 — entirely above my 0.50 threshold. So the guard, which only fires below 0.50, never fires on them at all. Result: FCR 1.00. Every unanswerable query was confidently answered. The guard was switched off, and nothing told me. Recalibrate the threshold to ~0.70 and FCR drops to 0.00 — the distributions are separable, I was just cutting in the wrong place.

hashing-64: answerable (0.30–0.68) and unanswerable (0.35–0.53) overlap. No threshold separates them, because the embedder is too weak to distinguish "relevant" from "vaguely near." The right lesson here isn't "pick a better threshold" — it's "this embedder can't support abstention at all."

The lesson: never ship a hard-coded abstention threshold

A magic constant that "works" is the most dangerous kind of code, because it works right up until the context shifts — and then it fails silently, which is the worst possible failure mode for a safety guard. My 0.50 wasn't a good threshold that I'd validated. It was a coincidence that held for one embedder and collapsed the moment I changed one.

The fix is cheap and non-negotiable: calibrate the abstention threshold per embedding model, against a small labeled set. Twenty-odd queries — a handful answerable, a handful not — is enough to see where the two distributions actually sit and cut between them. Do NOT ship a constant. The writeup says it in one line: calibrate per embedding model against a small labeled set; do not ship a hard-coded constant.

Why this generalizes past RAG

Any decision that thresholds a similarity score inherits this exact trap:

semantic caching ("is this query close enough to a cached one?")
near-duplicate / dedup detection
"is this document relevant enough to include?"
clustering cutoffs, entity-matching thresholds

In every one of these, the threshold is a property of the (embedder, corpus) pair, not a universal constant. Swap the model and your carefully-chosen number is now cutting in the wrong place — and unless you're measuring the failure explicitly, you won't know.

Which is the meta-point, and the reason Part 4 mattered: this failure is invisible without the eval harness. The system ranks well, returns plausible results, and lies on gaps. You only catch it by measuring a false-confident rate per embedder. Ranking metrics would have shown me green the entire time.

Two footnotes from after this was drafted. First: when I trailed this finding in Part 1, a commenter guessed the mechanism before the post existed — and proposed a relative threshold (top hit versus the rest of the batch) instead of an absolute one. That experiment is still owed; my worry is that a spread-based check is blind to the single confident distractor. Second, and worse: there is a whole failure class no threshold can catch, by construction — the near-miss that scores high. That one needed a different kind of fix, and it's the subject of the follow-up post.

If a better threshold helps, would a better embedding help more? The intuitive next move is to fine-tune the embedder on my domain. I did. The result was zero — and Part 6 is about why that was exactly the right outcome, and the one case where fine-tuning actually did move the needle.

Part 5 of Retrieval-Augmented Self-Recall. Code: RE-call. The "measure the failure that matters, not the one that flatters" discipline runs through Claude Code, Beyond the Prompt too.

Retrieval-Augmented Self-Recall — Part 4: Benchmarking Retrieval and Honesty

Giulio D'Erme — Sat, 18 Jul 2026 12:06:26 +0000

Part 4 of Retrieval-Augmented Self-Recall. Code: RE-call. Part 3: the honesty guards.

A standard RAG benchmark would have handed my retriever a perfect score on a day it was confidently answering questions it had no data for.

That's not a bug in my system. It's a gap in what those benchmarks ask. Every one of them asks the same thing: when there was an answer, did you rank it first? None of them ask the question that decides whether agent memory is safe: when there was no answer, did you say so?

So RE-call ships its own harness. Here's how it works, and the first finding it produced.

The test set: the unanswerable queries are the point

The evaluation runs on 14 answerable queries + 5 unanswerable queries over a synthetic corpus.

Those 5 unanswerable queries are the whole reason the harness exists. They're questions the corpus genuinely cannot answer, where the correct behavior is to abstain — to fire gap_warning, not to confidently return the nearest memo. Standard retrieval benchmarks are built entirely from answerable queries; they have no way to score "did it correctly say nothing?" This harness is built around that case.

Two families of metrics

Because there are two jobs — rank well when there's an answer, abstain when there isn't — there are two families of metrics:

Ranking quality (for the answerable queries):

precision@k, recall@k
MRR (mean reciprocal rank)
nDCG@10

Guard quality (for the unanswerable queries):

False-confident rate (FCR) — the fraction of unanswerable queries that the guard failed to flag. High FCR means the system confidently answered questions it should have abstained on. This is the honesty metric, and it's the one almost nobody reports.

Why you need both is the crux: a system can post excellent MRR and terrible FCR. It ranks beautifully whenever an answer exists, and lies confidently whenever one doesn't. If you only look at ranking metrics — as most RAG evals do — that failure is completely invisible. FCR is what drags it into the light.

The ablation: every embedder × every fusion stage

The harness runs the full matrix: each embedder (HashingEmbedder, bge-small, voyage-3) crossed with each fusion configuration (dense only → hybrid → hybrid + rerank). That's what lets you answer "which component actually earns its cost?" instead of cargo-culting a reranker into every pipeline.

And it runs against the real thing: 49 integration tests on a live pgvector container (of a 150-test suite), in CI, no mock database. The benchmark exercises the actual retrieval path, not a stand-in.

Finding 1: hybrid + rerank helps most exactly where you'd expect — and nowhere else

Here's the ablation on the weak (hashing) embedder — quality climbs monotonically as you add stages:

Configuration	MRR	nDCG@10
Dense only	0.63	0.72
+ sparse (hybrid)	0.74	0.80
+ cross-encoder rerank	1.00	1.00

Now the same pipeline on the strong bge-small embedder: dense retrieval already achieves a perfect nDCG@10. Hybrid fusion and reranking add nothing — there's no headroom left to recover.

The conclusion, stated plainly in the writeup: hybrid + rerank buys the most on weaker embedders or harder corpora; on an easy corpus with a strong embedder it's redundant.

That is a genuinely useful engineering result, because the reflex in RAG is to stack a reranker onto everything. This says: don't pay for stages your embedder has already made unnecessary. Measure first. A cross-encoder rerank on every query is real latency and real cost — and on a strong embedder over a well-covered corpus, you're buying zero.

The rigor is the point

None of these numbers come from an in-memory toy. They come from the same harness that runs in CI against real Postgres — with a dependency audit — every commit. The reason to trust the findings is that the measurement is reproducible. That's the whole pitch of this track: measure honestly, including the parts that make your work look less impressive.

Which is a good segue, because the same ablation surfaced something that made a chunk of my gap_warning design look worthless on certain embedders. I did not see it coming.

Part 5 is the finding I keep leading with: I shipped a sensible-looking abstention threshold, switched embedders, and watched every query that should have been refused sail through as a confident answer. Why a hard-coded similarity threshold is a landmine — and what to do instead.

(The harness itself has kept growing since this was drafted — it now also scores declared-supersession versus timestamps, and a held-out "near-miss" challenge set that no threshold can catch by construction. Both came out of reader comments, and both are covered in the follow-up post.)

Part 4 of Retrieval-Augmented Self-Recall. Code: RE-call. The eval-first discipline here is the same one behind Claude Code, Beyond the Prompt.

Retrieval-Augmented Self-Recall — Part 3: Teaching RAG to Say \"I Don't Know\

Giulio D'Erme — Sat, 18 Jul 2026 12:06:01 +0000

Part 3 of Retrieval-Augmented Self-Recall. Code: RE-call. Part 2: hybrid retrieval on Postgres.

Ask your agent "have we tried this filter on this market before?" when the honest answer is never. A ranking retriever hands back the three closest memos anyway — something about a different filter, on a different market — and the agent, looking at three confident results, concludes: yes, we've looked at this.

It just made a decision on a hallucination. Nothing in the stack noticed. No error was raised, because from the retriever's point of view nothing went wrong: you asked for the nearest neighbours and it gave you the nearest neighbours.

Everything in Part 2 made retrieval good. Good ranking makes this failure worse, not better — it returns confident noise faster. This post is about making retrieval honest, which for agent memory is the part that actually decides whether you can trust it.

So RE-call wraps retrieval in honesty guards — this post covers the original three, each answering a question ranking metrics never ask. (The current repo has grown that table to six, and the growth story is its own post: two of the new guards exist because readers of Part 1 pointed at exactly the weaknesses you're about to see me describe. I'll flag those spots as we go.)

Guard 1: `gap_warning` — "is the best match good enough to trust?"

After retrieval, look at the best dense cosine similarity. If it falls below a calibrated threshold, the top result isn't the answer — it's the least-bad noise. The system sets gap_warning = true.

The important design idea: this is a second-order signal. Retrieval still returns its ranked list; the guard annotates how much to trust it. That annotation is what lets the calling agent do something other than blindly act — it can abstain, ask a human to confirm, widen the search, or explicitly note "no prior memory on this" before proceeding.

That single flag is the difference between an agent that says "we've looked at this before" and one that says "I have nothing relevant on this — treat it as new." In a system that makes decisions, that distinction is worth more than any ranking improvement.

There's one buried landmine here: what threshold? The obvious move is to pick something like 0.50 and move on. That obvious move is quietly, dangerously wrong — and it's the biggest finding in this series, so I'm giving it its own post (Part 5). For now, the load-bearing word is calibrated: the threshold is fit to data, never hard-coded.

Guard 2: freshness — "is this memory still current?"

Every memo has a timestamp. The freshness guard reports the age of retrieved content and warns when it's stale relative to the re-index cadence (my corpus re-indexes daily, so "stale" has a concrete meaning).

This one is specific to memory in a way document QA rarely deals with. A documentation corpus is mostly static — last year's page is still roughly true. Agent memory is a moving target: a decision recorded in April may have been reversed in June. Without a freshness signal, April-truth and June-truth are indistinguishable at retrieval time, and the agent will happily act on a superseded conclusion. Freshness lets it weight recency, or at least flag the risk.

Honest update, because this section aged: freshness turned out to be the weakest guard of the three, and a commenter on Part 1 put a finger on why — supersession is a relation between two memos, and no per-document timestamp can see a relation. We later measured it: even a steelmanned "trust the newest relevant hit" heuristic still hands back the stale memory 83–100% of the time, while an explicitly declared supersedes: link holds at 0.00. The fix (a trust layer that binds the relation at write time) and the measurement are in the follow-up post.

Guard 3: anti-re-litigation — "did we already settle this?"

The most agent-specific guard of the three. Before the agent proposes an idea, it queries memory for closed decisions on that topic — the "we tried X, it failed, here's why" memos — and the guard surfaces them.

The failure it prevents is subtle and expensive: an agent re-proposing a dead idea because the memo that killed it three months ago didn't happen to rank in the top results for today's phrasing. Ranking-optimized retrieval is bad at this specifically, because a settled-decision memo is often lexically distant from the fresh proposal even though it's the most decision-relevant document in the store.

The implementation leans on structure: decision-type memos (closed hypotheses, postmortems) are typed, and a targeted retrieval path prioritizes them when the agent is in "propose" mode. Memory that can't defend its own past decisions is condemned to relive them.

The unifying idea

Retrieval answers one question: what's closest? The guards answer the three that actually govern whether the agent should act:

Should you trust it? (gap_warning)
Is it still current? (freshness)
Did we already decide this? (anti-re-litigation)

That's the whole difference between a search index and a memory. A search index ranks. A memory knows its own limits.

(Since this was drafted, the guard table grew: trust verdicts with declared supersession, an opt-in entailment judge for the high-similarity-but-wrong case a threshold can never catch, and a write-time lint for the supersession graph. All three exist because readers argued with this post's ancestors — that story, with measurements, is the follow-up.)

One rule, or the guards are theater

A guard only helps if its signal reaches the decision layer. A gap_warning that gets computed and then dropped before the agent sees it is worse than useless — it's false assurance that the system is careful when it isn't. So in RE-call the honesty signals ride inside the retrieval result: you cannot get the answer without also getting "here's how much to trust it." (If you read the applied series, this is the same principle as making the tool enforce the rule instead of trusting the prompt to.)

The guards make claims: this is a gap, this is stale. Claims demand measurement — and "how well does it know when it doesn't know?" is a metric that standard RAG benchmarks don't even have. Part 4 builds the eval harness that measures it, and delivers the first finding about which pipeline components actually earn their cost.

Part 3 of Retrieval-Augmented Self-Recall. Code: RE-call. This is the layer that makes Claude Code, Beyond the Prompt's memory trustworthy, not just searchable.

Retrieval-Augmented Self-Recall — Part 2: Hybrid RAG on Nothing but Postgres

Giulio D'Erme — Sat, 18 Jul 2026 12:05:37 +0000

Part 2 of Retrieval-Augmented Self-Recall. Code: RE-call. Part 1: the self-recall problem.

Say "vector search" and the reflex is a dedicated vector database: Pinecone, Weaviate, Qdrant. I didn't install one. RE-call keeps the dense vectors, the full-text index, and the metadata you filter them by in a single Postgres.

Not as a shortcut, and not out of allergy to new infrastructure. Because for agent memory a separate vector store is the wrong shape, and it quietly costs you the one property this whole system depends on.

Here's the argument, and the retrieval pipeline it buys you.

Why not a dedicated vector DB

Agent memory has three properties that make Postgres the natural fit:

It's already relational. Memos have timestamps, source types, tags, decision status. That's structured metadata you want to filter and join on — exactly what a relational database is for. A separate vector store means keeping two systems in sync and losing transactional consistency between the vectors and the metadata.
pgvector gives you real vector search inside Postgres. Approximate-nearest-neighbor cosine search, in the same database as your rows. And Postgres already ships full-text search. So you get dense and sparse retrieval in one transactional store — no sync layer, no second system to operate.
The scale doesn't justify the complexity. My corpus is ~700 memos, about 5 MB, re-indexed daily. Even orders of magnitude larger, a read-mostly, latency-tolerant memory is nowhere near the regime where a distributed vector DB earns its operational cost. Reaching for one here is over-engineering.

One store, one source of truth, ops you already know. Now the pipeline.

The retrieval pipeline

RE-call retrieves in up to four stages. The first two run in parallel; the last two refine.

1. Dense retrieval

Embed the query, run a pgvector cosine-similarity search, take the top-k. This is semantic matching — it finds memos that mean the same thing as the query even with no shared words. Strong on concepts, weak on exact tokens.

2. Sparse retrieval

Run a Postgres full-text search (tsvector/tsquery) over the same corpus. This is lexical matching — it nails exact terms: a specific error code, a ticker, a piece of domain jargon, a proper noun. Strong on precision, blind to paraphrase.

3. Fusion with RRF

Now you have two ranked lists that disagree. You fuse them with Reciprocal Rank Fusion (k=60):

score(doc) = Σ  1 / (k + rank_in_list_i(doc))

Each document's score is the sum, across both lists, of one over its rank (plus a constant k). Documents that rank high in either list bubble up; documents high in both dominate.

The reason RRF specifically: dense cosine scores and full-text scores aren't on the same scale — you can't just add or average them without arbitrary normalization. RRF sidesteps that entirely by fusing on rank instead of raw score. k=60 is the well-established default and it's robust; you rarely need to tune it.

Why fuse at all? Because dense and sparse fail in opposite directions — one misses exact tokens, the other misses meaning. Combining them gives you the concept-matching of embeddings and the precision of keyword search. (If you read the applied series, this is the "hybrid beats either alone" lesson — here's the actual mechanism under it.)

4. Cross-encoder rerank (optional)

The fused shortlist can be reordered by a cross-encoder (ms-marco-MiniLM). Unlike the bi-encoder embeddings in stage 1 — which encode query and document separately and compare vectors — a cross-encoder encodes the query and a candidate together and scores their relevance jointly. It's meaningfully more accurate and meaningfully slower, so you only run it on the top handful of candidates, never the whole corpus.

Whether stage 4 is worth its cost turns out to depend heavily on your embedder — which is the subject of Part 4's benchmark.

One property of this stage matters more than I realized when I first drafted this: the cross-encoder reorders, but what it emits is still a score you end up thresholding somewhere downstream. That distinction — a score you tune versus a decision you can trust — comes back with force later in the series.

Pluggable embedders

RE-call treats the embedder as a swappable component, with three shipped:

HashingEmbedder — deterministic, offline, no model download. It exists so the test suite and CI can run vector retrieval with zero external dependencies. Weak, but reproducible.
FastEmbed (bge-small) — a strong local model, no API calls. Good default for privacy or air-gapped runs.
Voyage (voyage-3) — a cloud model, the strongest of the three.

Pluggability isn't just tidiness. It lets you test deterministically offline, run locally for privacy, or call the cloud for maximum quality — same pipeline, different tradeoff. And it sets up the single most important finding in this series: the embedder you choose changes how you have to calibrate everything downstream (Part 5). Hold that thought.

Is "just Postgres" actually enough?

Fair challenge, and I don't want to hand-wave it. The claim is backed by the harness, not vibes: of RE-call's 150-test suite, 49 integration tests run against a real pgvector container — no mock database — in CI. The retrieval you just read about is exercised against the real engine on every commit.

And to be honest about the boundary: when would you want a dedicated vector DB? Billions of vectors, sub-10 ms p99 under heavy concurrent QPS, distributed sharding across nodes. Agent memory is none of those. It's small, read-mostly, and perfectly happy with tens-of-milliseconds retrieval. Match the tool to the regime — and this regime is Postgres-shaped.

Retrieval now returns the closest memos. But "closest" is not "relevant" — the closest match to a question with no answer is still just noise wearing a high similarity score. Part 3 is about the guards that let the system tell the difference: how RE-call learns to say "I don't know."

Part 2 of Retrieval-Augmented Self-Recall. Code: RE-call. If you came from Claude Code, Beyond the Prompt, this is the retrieval layer under Part 5's semantic search.