Nine published systems. One of them refuses 2.3× more often than gpt-4o and still finishes 4th on answer quality. It calls no LLM.
Part 5 of The Answerability Problem. Parts 1 to 4 argued the field measures accuracy on questions that have answers. This one is about the benchmark that doesn't. Code: RE-call (Apache 2.0).
MTRAG (IBM, TACL 2025) is 842 human-written multi-turn tasks with an official gpt-4o-mini judge that pays a full 1.0 on every metric for correctly saying "I do not have that information."
Almost no other benchmark scores a refusal as anything but a miss. So this is the one board where the thing my system is built for is actually worth points.
Here is what nine published systems do with that.
Nobody is collecting the free money
Correct refusals on the 55 unanswerable tasks. Same judge, same tasks, every system:
| # | system | correct refusals | |
|---|---|---|---|
| 1 | llama-3.1-8b | 18/55 | 32.7% |
| 2 | 🔹 gpt-4o + RE-call |
16/55 | 29.1% |
| 3 | llama-3.1-70b | 16/55 | 29.1% |
| 4 | gpt-4o-mini | 13/55 | 23.6% |
| 6 | gpt-4o (benchmark's own retrieval) | 7/55 | 12.7% |
| 7 | llama-3.1-405b | 3/55 | 5.5% |
| 9 | qwen-2.5-72b | 1/55 | 1.8% |
| 10 | mixtral-8x22b | 0/55 | 0.0% |
Two rows in that table read the same document set with the same model. Row 6 is gpt-4o on the benchmark's own retrieval. Row 2 is gpt-4o on RE-call's. Same reader, same prompt, same judge, only the contexts differ, and the refusal rate goes from 7 to 16 out of 55.
Then read the rank order. It is close to inverted against model size. The 8B model refuses best. The 405B model refuses 5.5% of the time. One model refuses nothing at all. Scale does not buy this, and on this evidence it costs it.
RE-call is second of ten, tied exactly with llama-3.1-70b at 16 of 55, and it refuses 2.3× more often than gpt-4o and 16× more often than qwen-2.5-72b.
The usual objection, answered
Abstention is cheap if you don't care about answering. So here is what it cost, on the same 842 tasks:
| # | system | answer quality |
|---|---|---|
| 1 | llama-3.1-405b | 0.5691 |
| 2 | qwen-2.5-72b | 0.5625 |
| 3 | gpt-4o (benchmark's own retrieval) | 0.5591 |
| 4 | 🔹 gpt-4o + RE-call |
0.5527 |
| 5 | c4ai-command-r-plus | 0.5502 |
RE-call is the only system in that top four that is also top two on abstention. The three above it refuse 5.5%, 1.8% and 12.7%. The gap to gpt-4o is 0.0064.
That is the whole claim, and it is a narrow one: not "best", but the only one on the board that is near the top of both columns at once. Everyone else picked a column.
Same two rows again, 3 and 4, same reader. Swapping the benchmark's own retrieval for RE-call's costs 0.0064 on answer quality and more than doubles what the system correctly refuses. That is the trade, priced.
And against the benchmark's own retrieval directly, identical generator, prompt and judge, only the contexts swapped, RE-call is ahead: 0.5527 against 0.5516.
Small. Positive under both prompts I tested. I am not going to inflate it.
(Every row above is scored by MTRAG's official gpt-4o-mini-2024-07-18 judge, hard-coded by the benchmark and applied identically to my rows and theirs. The reader on my rows is gpt-4o, the same model as the row I am compared against.)
One engine, whatever you can afford
The other half of this. The same pipeline runs at both ends of the price range, and every rung is a named flag with a measured price, not a tier you buy.
| configuration | nDCG@5 | cost |
|---|---|---|
| 🔹 SPLADE learned sparse (the free default) | 0.3573 | local, $0 |
| + Voyage rerank (one flag) | 0.4342 | paid API |
A 48% relative span between two flags, measured on 777 judged queries.
The reranker is worth +0.0769 nDCG@5 and is worse on 162 of the 777. That is exactly why it is off by default. I would rather ship a flag with two published numbers than a default with a marketing claim.
Everything dashed and amber below is opt-in and switched on by name. The solid path is what runs if you change nothing, and it costs $0 at any scale, because there is no LLM anywhere in it:
Writing a memory is a local embedding. No LLM call, so the marginal cost of a memory is $0 and stays $0 at any scale, on any model, forever. There is no upstream pricing change that can alter that.
Air-gapped and fully local, or cloud embedder and reranker when a jargon-heavy corpus makes it worth paying. Same engine, same guarantee at every setting: superseded or expired memories are demoted rather than served, and the confidence threshold is fitted per embedder and corpus at install time, never a shipped constant.
The number I nearly published
One thing before you trust any of the above.
An earlier version of this run had RE-call beating every baseline, including llama-3.1-405b. It was an artifact and it was one edit away from going out.
The official scorer reads the answerability label in lower case. The release files ship it capitalised. So the conditioning that pays 1.0 for a correct refusal silently never ran on my data: 2,526 lookups, zero label matches. My metrics were raw, the published baselines' were conditioned, and I was comparing two different quantities in exactly the place where my system is strongest.
Both sides were individually correct. The error lived entirely in the join.
Two signals should have caught it earlier and I explained both away. I had even written the memo describing the two-key problem, that same day, and never connected it. Having the fact is not the same as applying it.
Corrected, the honest reading is: RE-call at 0.5527 against gpt-4o's 0.5591, so −0.0064. Fourth, not first. The abstention result survives intact, because it was measured by the official judge rather than by the string-matching detector that had earlier inflated it from 29% to 43.6%.
The anomaly you can explain is still an anomaly, and the one that flatters you is the one to chase hardest.
So which rung is yours
Take the free default if the marginal cost of a memory has to be zero, your data cannot leave, or you need it to work offline. 0.3573, local, $0, no API key.
Add the reranker when a human is waiting. One second is invisible next to the generator call that follows it, and it is the largest single gain in the project. Check it on your corpus first: it lost on 162 of 777 queries here.
Use something else if abstention is worth nothing to you and you only need the top of one column. Three systems beat me on answer quality. All three of them will confidently answer a question whose answer is not in the corpus, between 87% and 98% of the time.
That is the trade. A system that invents a plausible answer and one that says nothing are not two points on one scale.
Every number, the six generation runs behind them, the correction in full, and the scoring bug I reported upstream: docs/MTRAG_BENCHMARK.md. RE-call is Apache 2.0, pip install recall-rag.
Top comments (0)