A RAG answer says [c091] and everybody relaxes. There are two checks you can run on that bracket, and they catch different things.
Citation resolution is pure code: is c091 actually one of the chunks we assembled for this question? Zero model calls. Faithfulness is model-scored: does the assembled context support the claim?
👉 Live, runs in your browser: https://dev48v.infy.uk/ai/days/day80-citation-verification.html
The 2×2, and why both off-diagonals matter
420 seeded answers, 7 defect classes, 60 each:
| faithfulness says supported | says unsupported | |
|---|---|---|
| resolution passes | 120 | 60 |
| resolution fails | 180 | 60 |
If either off-diagonal cell were empty, one check would subsume the other and you could drop it. Both are populated by measurement: 180 answers where the identifier is wrong and the prose is fine, 60 where the identifier is fine and the prose is invented.
A pipeline running one check is not running a cheaper version of the other. It is running a different check whose blind spot is exactly what the other one is for.
The class people get wrong
stale_id: a real store id that was not in this assembled context. It looks real. It resolves perfectly against your vector store.
| check | catches |
|---|---|
| resolution against the assembled context | 60 / 60 |
| resolution against the chunk store | 0 / 60 |
Same check, one wrong set. That is the entire difference between a working guard and a decorative one:
// check 1 — resolution. No model. This is the entire thing.
const inContext = new Set(answer.assembled); // ids WE assembled
const resolves = answer.cites.every(id => inContext.has(id));
const pass = answer.cites.length > 0 && resolves;
// the bug: checking the STORE instead of the assembled context
const wrong = answer.cites.every(id => store.has(id)); // c091 is real...
// ...just not here
Both blind spots, stated as numbers
Resolution never reads the claim. An unsupported sentence carrying an id that genuinely is in the context is a clean pass — 0 of 60 caught.
A context-scoped faithfulness grader is id-blind. Claim in, assembled chunks in, supported-or-not out. An invented id, a stale one, or no id at all does not change its input by one token — 0 of 180 citation defects caught. That is not inferred: the self-check replaces every citation with nonsense and measures the verdict move on 0 of 420 answers.
Neither number moves by tuning a threshold. Both are structural.
The residual neither one catches
60 answers cite a chunk that is in the assembled context and does not support the claim, while some other assembled chunk does. Resolution passes — the id is in the context. Context-scoped faithfulness passes — the context supports the claim. Both answer their own question correctly; neither question is "does this chunk support this sentence".
A citation-level grader does catch all 60, and costs one call per citation rather than per answer — 180 here against 420, and far worse on answers citing five sources each. It also cannot run on a citation that did not resolve: you cannot hand a grader a chunk you could not find. Third layer, never a replacement.
What the free check is worth
| defects caught (of 360) | |
|---|---|
| faithfulness alone | 120 |
| resolution alone | 240 |
| union | 300 |
| plus citation-level | 360 |
Zero false alarms from either check on the clean class. Resolution costs 360 comparisons and 0 model calls over the whole corpus; grading everything costs 420 calls; running the free check first and grading only what survives costs 180 — 240 model calls removed with the union catch rate unchanged.
One cost that is not money: everything short-circuiting skips sits in the cell where the answer text was fine. For those the repair is re-citing, not re-answering, and an ordering that dumps them into one undifferentiated "failed" bucket sends correct answers back to be rewritten.
The limitation to read first
The entailment check here is a stand-in, not a model — a stated token rule over 12 disjoint topic vocabularies, which separates supported from unsupported with no overlap at all. Real graders are wrong a great deal, and that error is part of why the free check is worth keeping. So read the faithfulness column as a ceiling and the resolution column as exact, because that one is code. The corpus is synthetic, one claim per answer, one citation per claim, no multi-hop, no reranker, no abstention. And nothing here asks whether the source is any good.
38 in-page checks, 138 verifier asserts, 0 failures.
Top comments (0)