Part 4 of Retrieval-Augmented Self-Recall. Code: RE-call. Part 3: the honesty guards.
A standard RAG benchmark would have handed my retriever a perfect score on a day it was confidently answering questions it had no data for.
That's not a bug in my system. It's a gap in what those benchmarks ask. Every one of them asks the same thing: when there was an answer, did you rank it first? None of them ask the question that decides whether agent memory is safe: when there was no answer, did you say so?
So RE-call ships its own harness. Here's how it works, and the first finding it produced.
The test set: the unanswerable queries are the point
The evaluation runs on 14 answerable queries + 5 unanswerable queries over a synthetic corpus.
Those 5 unanswerable queries are the whole reason the harness exists. They're questions the corpus genuinely cannot answer, where the correct behavior is to abstain — to fire gap_warning, not to confidently return the nearest memo. Standard retrieval benchmarks are built entirely from answerable queries; they have no way to score "did it correctly say nothing?" This harness is built around that case.
Two families of metrics
Because there are two jobs — rank well when there's an answer, abstain when there isn't — there are two families of metrics:
Ranking quality (for the answerable queries):
- precision@k, recall@k
- MRR (mean reciprocal rank)
- nDCG@10
Guard quality (for the unanswerable queries):
- False-confident rate (FCR) — the fraction of unanswerable queries that the guard failed to flag. High FCR means the system confidently answered questions it should have abstained on. This is the honesty metric, and it's the one almost nobody reports.
Why you need both is the crux: a system can post excellent MRR and terrible FCR. It ranks beautifully whenever an answer exists, and lies confidently whenever one doesn't. If you only look at ranking metrics — as most RAG evals do — that failure is completely invisible. FCR is what drags it into the light.
The ablation: every embedder × every fusion stage
The harness runs the full matrix: each embedder (HashingEmbedder, bge-small, voyage-3) crossed with each fusion configuration (dense only → hybrid → hybrid + rerank). That's what lets you answer "which component actually earns its cost?" instead of cargo-culting a reranker into every pipeline.
And it runs against the real thing: 49 integration tests on a live pgvector container (of a 150-test suite), in CI, no mock database. The benchmark exercises the actual retrieval path, not a stand-in.
Finding 1: hybrid + rerank helps most exactly where you'd expect — and nowhere else
Here's the ablation on the weak (hashing) embedder — quality climbs monotonically as you add stages:
| Configuration | MRR | nDCG@10 |
|---|---|---|
| Dense only | 0.63 | 0.72 |
| + sparse (hybrid) | 0.74 | 0.80 |
| + cross-encoder rerank | 1.00 | 1.00 |
Now the same pipeline on the strong bge-small embedder: dense retrieval already achieves a perfect nDCG@10. Hybrid fusion and reranking add nothing — there's no headroom left to recover.
The conclusion, stated plainly in the writeup: hybrid + rerank buys the most on weaker embedders or harder corpora; on an easy corpus with a strong embedder it's redundant.
That is a genuinely useful engineering result, because the reflex in RAG is to stack a reranker onto everything. This says: don't pay for stages your embedder has already made unnecessary. Measure first. A cross-encoder rerank on every query is real latency and real cost — and on a strong embedder over a well-covered corpus, you're buying zero.
The rigor is the point
None of these numbers come from an in-memory toy. They come from the same harness that runs in CI against real Postgres — with a dependency audit — every commit. The reason to trust the findings is that the measurement is reproducible. That's the whole pitch of this track: measure honestly, including the parts that make your work look less impressive.
Which is a good segue, because the same ablation surfaced something that made a chunk of my gap_warning design look worthless on certain embedders. I did not see it coming.
Next
Part 5 is the finding I keep leading with: I shipped a sensible-looking abstention threshold, switched embedders, and watched every query that should have been refused sail through as a confident answer. Why a hard-coded similarity threshold is a landmine — and what to do instead.
(The harness itself has kept growing since this was drafted — it now also scores declared-supersession versus timestamps, and a held-out "near-miss" challenge set that no threshold can catch by construction. Both came out of reader comments, and both are covered in the follow-up post.)
Part 4 of Retrieval-Augmented Self-Recall. Code: RE-call. The eval-first discipline here is the same one behind Claude Code, Beyond the Prompt.
Top comments (0)