Originally published at ai.bedvibe.studio.
My portfolio RAG agent holds 1,003 indexed chunks. A question retrieves six and answers.
Six of 1,003 is 0.598%. Nothing in the response, the logs or the trace says the other 997 were never looked at. It reads exactly like an answer built from reading everything.
That is fine for "what does he say about X." It is not evidence for "does he mention X anywhere" — and there is nothing in a normal RAG stack that tells those two apart.
The claim retrieval cannot support
A compliance corpus: 2,431 policy documents. Someone asks whether there is a remote-work reimbursement policy. Retrieval returns eight chunks. None mentions reimbursement. The agent answers:
There is no remote-work reimbursement policy in the corpus.
That claim requires knowledge of 2,431 documents. It was made from eight. The agent did not lie and it did not hallucinate — it correctly reported what it found, and the shape of the sentence quietly upgraded a statement about eight documents into a statement about the corpus.
Absence of evidence in a retrieved fragment is not evidence of absence in the whole. Everyone knows this. No retrieval stack I have used records enough to enforce it, because the denominator is not carried anywhere near the answer.
"But our faithfulness score is 1.0"
This is the first objection and it deserves a straight answer: faithfulness cannot catch this, by construction.
Faithfulness asks whether the answer is supported by the retrieved context — did the model invent anything beyond what it was given. The compliance answer above invents nothing. It scores a perfect faithfulness and may be false about 2,423 documents.
Faithfulness scores answer against context. Coverage scores context against corpus. Different axes, and a system can be perfect on the first while completely silent on the second.
| Metric | What it needs | Available at answer time? |
|---|---|---|
| Faithfulness | answer + context | Yes |
| Context precision | answer + context | Yes |
| Context recall | ground-truth annotations | No — offline evaluation only |
| Corpus coverage | corpus size + retrieved count | Yes — and it is not reported |
Context recall is the metric that would catch it, and it needs labelled ground truth, so it lives in your evaluation harness and not in production. Corpus coverage needs two integers you already have.
The control is the shape of the sentence, not a percentage
This took longest to see, and it is why there is no threshold anywhere in the implementation.
0.598% is not a bad number. It is a bad number for one class of sentence. It is entirely adequate for "he mentions Rust" — you need the one chunk you are quoting and nothing else. It cannot support "he never mentions Rust." Same retrieval, same six chunks, opposite verdicts, because the claim changed.
A single coverage threshold cannot serve both. Set it low and it licenses the second sentence; set it high and it forbids the first.
| Claim | Coverage required |
|---|---|
| "The policy says X" | Any — you need only the units you cite |
| "There is no policy about X" | Exhaustive |
| "All policies require X" | Exhaustive |
| "The most recent policy is…" | Exhaustive — the unread remainder may hold the true maximum |
| "There are three mentions" | Exhaustive — a count over a sample is an estimate |
Every row needing exhaustive coverage has the same reason: it asserts something about the units that were not read. 999,999 of 1,000,000 is 99.9999% and still cannot establish absence — a threshold there would be a lie with a decimal point on it.
Three failures wearing one label
The second thing the missing denominator costs you is diagnosis. When a RAG answer is wrong, the post-mortem usually terminates at the model hallucinated. That sentence hides at least three separate engineering problems:
| What actually happened | What to fix |
|---|---|
| The relevant chunk was never retrieved | The retriever, embedding or query — not the model |
| It was retrieved, then dropped during context assembly | Context construction — the model never saw it |
| It reached the model, which reasoned past it | The prompt or the model — the only reasoning failure |
Three different people, three different days, one identical symptom. And the distinction is trivially recordable — retrieved, in_context, and whether the answer was right — but almost nobody logs the middle number, so the first two are permanently indistinguishable after the fact.
What I built
An MCP server for notchecked, the coverage-accounting schema I wrote after hitting the silent-skip failure in four domains. The library types the gaps a program leaves. The server does it for an agent, which is where the failure moved.
coverage_retrieval(
target = "portfolio corpus",
query = "Rust experience",
corpus_size = 1003,
retrieved = 6,
in_context = 6,
claim_type = "absence",
)
→ scope: 6 of 1003 corpus units (0.598%)
exhaustive: false
claim_supported: false
REFUSE_THIS_CLAIM: An ABSENCE claim asserts something about every unit
you did NOT read, and 997 were never read. Downgrade the answer to what
you found, or inspect the whole corpus.
retrieved = 0 is not a thin answer, it is a retrieval failure, and it records as one. Everything retrieved and then dropped is a context failure and records as that instead. Neither is a gap in the agent's reasoning, and calling them one sends someone to debug a prompt for a day.
The bug this found in its own implementation
I wrote a suite that replays six investigations of my own that produced wrong claims — a page judged from 3,000 of its 10,828 words, a search that had stripped the HTML so anything named only in an href was invisible, three different counts from three broken filesystem walks.
On its first run it failed, and it failed on my code rather than on the cases. exhaustive was a bare assertion: an agent could pass exhaustive: true alongside scope: "3,000 of 10,828 words" and the absence warning was dropped. The tool committed the exact failure it exists to prevent, one layer above the schema it protects. Where the scope carries "N of M", the contradiction is machine-visible and is now refused.
That is the third time this idea has caught its own implementation. I have stopped finding it funny.
What this does not do
If your retriever reports corpus_size=100000, retrieved=20, in_context=12, cited=3, this records those numbers and what they can support. It does not know whether the retriever chose the right twenty. It is not a retriever, a vector database, a reranker or a context assembler, and it is not competing with the one you have.
Three limits are recorded as passing tests rather than left out of the README: an exhaustive search of the wrong instrument is still exhaustive; the target list is self-declared, so nothing can know what the caller failed to think of; the retrieval counts are self-reported, with only internal consistency enforced.
What it removes is the silence. That is a smaller claim than "this makes agents honest," and it is the one the evidence supports.
I have measured this in one live system, my own. I have not established what other RAG deployments report, and I am not claiming it from a sample of one.
pip install notchecked · github.com/Mormolykos/notchecked — MIT, zero runtime dependencies, MCP over stdio written from the JSON-RPC wire format.
The eight coverage states were reviewed publicly by Boris Teplitsky, an IBM Certified Expert IT Architect who hit the same shape in infrastructure compliance and gave three corrections that changed the schema. They are frozen.
If you run RAG in production and you do record retrieval coverage alongside answers, I would like to know — that would make this a solved problem I had not found the solution to, which is a better outcome than being right.
Top comments (0)