Ten questions over the SEPA rulebooks. A second model grades each answer against a reference, zero to a hundred, and the run came back healthy.
Except for question five, which asked which currency SCT Inst payments are executed in. The reference answer is one word long.
A zero on a one-word question reads like a system that does not know the first thing about its own subject matter. That is not what happened.
It was doing what it was told
The production prompt is explicit about the case where retrieval comes up short:
Answer the question using ONLY the sources below. If they do not contain the
answer, say so.
Retrieval never put the euro page in the top five results. So the sources handed to the model genuinely did not contain the answer, and the model was under written instruction to say exactly that.
The grader took that response, compared it to the word "Euro", and returned a zero.
The grader cannot see the difference
The judge receives the question, the reference answer and the answer text. It does not receive the sources, and that is deliberate: a grader holding the retrieved chunks would end up grading the retrieval instead of the answer.
That choice has a cost. "Correct, given what it was handed" is not a judgement this grader can make. It has one axis, factual match against the reference, and everything that fails to match lands at the bottom together.
Which means a fabricated answer about euro payments would have scored exactly what the refusal scored. Zero, either way. The eval could not tell the two apart.
The saved run does not help either. Each question is stored as an id, a score and a one-line critique. The answer text itself is not kept, so nothing downstream can recover the distinction the grader could not draw.
The failure was upstream
If the refusal had been the problem, changing the answering model would have moved the number. It was not, and it did not.
Two separate experiments recovered that question, and neither one touched generation. Keyword search found the euro page instantly, because the page contains the actual word. Vector search matches on meaning rather than
words, and it ranked that page outside the top five. A cross-encoder reranker, kept to the eval path only, found it as well and lifted recall@5 from 0.60 to 0.70.
Neither is a free fix. Leaving keyword search on permanently left recall@5 exactly where it started, at 0.60, because it dropped a question the vectors had been getting right. The blind spot moved. It did not close. The reranker is the only arm that moved the number, and it is not in production.
Make a wrong answer cost something
Retrieval will keep missing pages. Improving that is a separate job, so what matters here is how the eval reacts when it happens.
The grader was not being harsh. My rubric just stops at zero, so both failures pile up there together. One of them wastes the reader's time. The other sends them into a payments rulebook holding a rule that does not exist. My eval scored those the same.
So the strictness belongs on the wrong answer. Meta's CRAG
benchmark, which I only found after this run, grades four outcomes instead of one axis: a correct answer scores 1, a useful answer with minor errors 0.5, a missing answer 0, and an incorrect answer minus one. "Missing" is defined there as the system replying "I don't know".
That minus sign is the part I was missing. Saying nothing costs you nothing. Being wrong costs you a point.
Nothing was fabricated in this run. The system declined, as instructed, and the number punished it for that. But the rubric that could not credit that refusal is the same rubric that would not have charged for a made-up answer, and a made-up answer is the one that reaches the reader.
Charge the model for being wrong. Let it say nothing for free.
Top comments (0)