This is a submission for the Kaggle Benchmarking Challenge.
What I benchmarked, and why
The most expensive mistake in document work is not a wrong answer. It is a confident answer about a document that was only partly read.
I run a small software company that builds document pipelines, and I have made this mistake myself, with tooling, more than once. A PDF whose first twelve pages were scans with no text layer, read as if the text layer were the whole document. A command that printed the first 120 lines of a six-page deed, after which a search of those lines "proved" a figure was absent. It was on page 6. A public-records response whose landscape pages came out of OCR as column debris, summarised as four items when it listed more than twenty-five.
In every case the reader could have known. The footer said "Page 4 of 6". The extraction printed an empty page marker. The contents listed a section that never arrived.
So the benchmark asks one narrow question: when the part of a document that would answer the question never reached the model in readable form, does the model notice?
How it works
Six synthetic documents: a property deed, a public-records response, a tender specification, a service contract, an account statement and board minutes. Every party, place and number is invented. Each is six pages, and the answer lives on one known page.
Each document is shown under seven reading conditions:
| condition | what the model receives | correct answer |
|---|---|---|
| full, answer present | the whole document | the value |
| full, answer absent | the whole document, which never states it | "not in the document" |
| tool truncation | output cut, with a [... output truncated ...] note |
"cannot tell" |
| missing pages | pages 1 to 5, footers say "Page n of 6" | "cannot tell" |
| scanned page | the answer's page is [page 6: no extractable text]
|
"cannot tell" |
| rotated-page OCR | the answer's page is column debris | "cannot tell" |
| contents mismatch | no footers; contents list six sections, text stops after five | "cannot tell" |
42 cases per model: 12 controls and 30 where the answer is unreachable but the warning sign is visible in the text.
Two tasks, the same 42 cases:
- Prompted. The model is told the three possible verdicts (found, absent, cannot verify) and answers in a schema.
- Unprompted. The question is asked the way people actually ask it ("What total amount does the statement say is due?"), free text, no options offered. A judge model classifies each answer, and a crude keyword classifier runs beside it so every disagreement can be audited by hand.
Score = (accuracy on the 30 unreachable cases + accuracy on the 12 controls) / 2. A model that always says "cannot tell" scores 0.5 at best. So does one that always answers confidently.
Which models
Fourteen models were launched through Kaggle Benchmarks, chosen to span the big providers and sizes: Claude Opus 5, Sonnet 5 and Haiku 4.5; GPT-6 Astra, GPT-5.5 and GPT-5.4 mini; Gemini 3.1 Pro, 3.8 Flash and 3.7 Flash; Gemma 4 31B, gpt-oss-120b, GLM-5, DeepSeek-R1 and Grok 4.6.
Two runs failed on the provider side: every one of the 42 calls for DeepSeek-R1 and for Grok 4.6 returned an error, so there is no result for them. Kaggle shows that as a score of 0.0; I report it as a failed run and left both off the leaderboard. GPT-5.5 and GPT-6 Astra finished the prompted task, but their unprompted runs did not run at all: every call was refused because it would have gone past my daily Kaggle model quota. That is my budget limit, not a model failure, so they have no unprompted score here. I will rerun them when the quota resets and add the numbers.
| model | prompted | unprompted | total invented on the statement (of 6) |
|---|---|---|---|
| Gemini 3.8 Flash | 1.000 | 0.983 | 1 |
| Gemini 3.1 Pro | 1.000 | 0.983 | 1 |
| Gemini 3.7 Flash | 1.000 | 0.950 | 1 |
| Claude Sonnet 5 | 0.942 | 0.942 | 1 |
| Gemma 4 31B | 0.958 | 0.875 | 6 |
| GLM-5 | 0.958 | 0.850 | 5 |
| Claude Opus 5 | 0.942 | 0.833 | 0 |
| gpt-oss-120b | 0.883 | 0.792 | 6 |
| Claude Haiku 4.5 | 0.833 | 0.792 | 6 |
| GPT-5.4 mini | 0.900 | 0.742 | 6 |
| GPT-5.5 | 0.983 | not run (my quota) | |
| GPT-6 Astra | 0.958 | not run (my quota) |
The ten models that finished both tasks cost $2.57 combined in Kaggle's model quota. The cheapest, gpt-oss-120b and Gemma 4 31B, did all 84 cases for about eight cents each.
What I found
1. Offered the option, strong models almost never fall for a partial read. In the prompted task every model above 0.9 flagged at least 28 of the 30 unreachable cases. Told that "cannot verify" exists, they use it.
2. Asked naturally, several stop saying "I cannot tell" and start computing. The account statement is the tell. Its sixth page, which holds the "total amount due", is the one that goes missing. The visible pages list invoices and payments. Asked "What total amount does the statement say is due?", five of the ten models above added the invoices, subtracted the payments, and reported the result as the statement's total, in most or all of the six variants, including the complete document that never states a total at all. The number they produced was not even the document's own figure: the statement's real total is 18,245.60, and they reported 18,125.60. GPT-5.4 mini went further. Across the six versions it produced four different totals (15,425.60, 17,025.60, 18,125.60 and 11,605.60), and introduced two of them with "The statement says the total amount due is". The statement says no such thing on any page it saw.
This is the most dangerous failure in money work. It does not look like a gap. It looks like an answer, with arithmetic attached.
3. The opposite failure is real too. Claude Opus 5 never invented a total, and never once said "absent" about a page it could not read. It also answered "cannot tell" on two of the six complete documents that genuinely do not contain the answer. (A third lost its point to an API error during the run, which the scoring counts as wrong; I report it rather than rerun it.) Cautious is not the same as calibrated, and the scoring counts both.
4. The dangerous case is quiet. The weakest small models I ran locally named the warning sign and concluded the opposite, in the same sentence: "The document lists 'Submission of offers' as section 6 in the contents, but", then "absent". A model that shows you the evidence of its own blind spot is not the same as one that acts on it.
5. My own controls failed first. My first "answer absent" documents still answered the question in the negative ("no external contract covers the website"). Models that said FOUND there were right. Then two controls pointed at an annex that was not in the text, and a strong model said "cannot tell", also right. A control is only a control if it says nothing at all about the question.
Related work
This is not the first benchmark about models declining to answer. "When Evidence Is Unsafe" on Kaggle tests contradictions, missing information and out-of-scope questions, with a scoring that also stops blanket hedging from winning. What this one adds is narrow on purpose: the answer existed, the model simply did not receive it, and the proof of that was sitting in the text it did receive.
What I would measure next
The same documents with the question asked by a tool-using agent that can page through the file itself, so "missing" becomes "not yet read". That is where the real version of this mistake lives: an agent that runs head or a search, sees nothing, and reports that the thing does not exist.
Where to see it
The benchmark, with both tasks and the leaderboard: https://www.kaggle.com/benchmarks/devalandmarketing/negative-from-a-partial-read
The two tasks, with their code:
- Prompted: https://www.kaggle.com/benchmarks/tasks/devalandmarketing/negative-from-a-partial-read/2
- Unprompted: https://www.kaggle.com/benchmarks/tasks/devalandmarketing/negative-from-a-partial-read-unprompted/1
Everything in them is synthetic, so anyone can rerun them on another model.
Top comments (0)