The cheap tier doesn't go blank — it writes

#ai #llm #ocr #benchmark

Here is the bank block that azure/gpt-5.6-sol@low returned for a Japanese invoice rendered at 300 dpi — a document sharp enough that you can count the pixels in the 7.5 pt fine print:

"bank_name": "みずほ銀行",
"bank_branch": "本店営業部",
"account_number": "1234567"

None of it is on the page. The printed bank is ほしかげ信用金庫 — a fictional credit union invented for a benchmark, with no real-world counterpart. The printed account number is seven digits that are not 1234567. The model didn't misread any of this. At its resolution tier it cannot see the fine print at all — and instead of leaving the fields blank, it wrote them.

That's the article. The rest is counting how often, and what gets written.

The control column

Last week I published a benchmark about a red seal covering an invoice field. Every occlusion ladder needs a control: L0, the step where the seal sits clear of everything and the document is simply a razor-sharp invoice. Twenty-seven model variants read that control — four materials, five repeats, twenty reads per field per model. I built it to be the boring column.

The boring column turned out to contain its own article, because it is the cleanest measurement I have of a question the earlier pieces only saw at an angle: what does a low-detail image tier actually read, when nothing whatsoever is wrong with the input?

What @low reads: gates, not dials

The invoice has thirteen document fields across four font tiers — a 28 pt title, large fields like the total and invoice number, 10.5 pt body fields, 7.5 pt bank details. Score each field out of twenty for each of the ten @low variants and a pattern appears that I did not expect to be this clean: 113 of the 130 cells are exactly 0 or exactly 20. A field is read every time, or never. "Unreliable" turns out to be the wrong mental model for this tier — reliability implies a dial. These are gates.

Which gates are open depends on the generation, and the direction is the uncomfortable one:

The GPT-5.5 and 5.6 generation @low variants read exactly four things: the title, and the three boxed money figures — total, subtotal, tax. Every name, every date, the invoice number, every bank detail: zero out of twenty (a couple of the lighter variants wobble to 18–19 on the money, nothing more).
The GPT-5.4 generation at the same tier reads more — the invoice number at 20/20, the due date at 19–20 — and it is also the only place in the table with a genuine transition band: issue date at 10 and 17, issuer name at 5 and 9, counterparty at 11. The older generation has a probabilistic middle; the newer one has a cliff.
Gemini 3.5 Flash @low reads all thirteen fields at twenty out of twenty, including the 7.5 pt bank block, at 16 credits per page. azure/gpt-5.6-sol@low costs 76 per page — 4.75× the price — and reads four fields. At the cheap end of the catalog, price does not order capability. It doesn't even correlate.

I wrote in the fabrication piece that the newer generations are stronger generators and weaker witnesses. The per-field table adds a quieter observation: at the low tier, the newer generation also simply reads less — a capability regression that no headline benchmark will ever show, because headline benchmarks don't run the cheap variants.

What fills the other nine fields

So a 5.6-generation @low read of this invoice has four real fields and nine unreadable ones. The question that matters operationally is what arrives in the nine.

Blanks would be fine. Blanks are honest. Across 1,120 reads of eight of those nine fields — the ninth, the bank branch, gets its own section below — the models returned a blank 181 times. They returned an invented value 938 times — an 84% fabrication rate, on a perfectly sharp document. Per response, that is on average 6.7 written fields and 1.3 blanks.

And the inventions are not noise. They are the statistics of Japanese paperwork:

The bank. The fictional credit union came back as one of Japan's three megabanks in 98 of the 101 runs that invented a bank at all — みずほ 54 times, 三井住友 41, 三菱UFJ 3. (One run answered メガバンク銀行 — "Megabank Bank" — which at least has the honesty of a placeholder.)
The account number. It came back as the literal 1234567 in 88 of 140 reads, with or without a 普通 prefix.
The dates. Of the 250 invented, 231 said 2025 on a document that says 2026 — the same systematic year shift the fabrication article found, reproducing here on pristine input.
The counterparty. 有限会社ミナト設計 became 株式会社ミナト交通: the distinctive word survived as a silhouette, the rest was regularized to the most common corporate form. Right shape, wrong document.

The earlier pieces each caught one face of this. The fabrication article showed the mechanism — when reading fails, generation fills the gap. The prior-capture piece showed the gravity — a partially legible name drifts to its nearest real neighbor. This column shows the limit case: at zero legibility there is no neighbor to drift to, and the model doesn't need one. It answers with the mode of the entire category. Any Japanese invoice — therefore みずほ銀行, 本店営業部, seven ascending digits, and last year.

The older generation, for what it's worth, blanks about twice as readily: one blank per 2.6 inventions, versus one per 5.2 for the new one. Progress, in this corner of the catalog, has meant becoming a more confident author of other people's invoices.

The cell that lied by being right

Which brings me to the most instructive mistake in my own results table.

Six times, a 5.5/5.6 @low variant scored correct on the bank branch — the only fine-print field that ever flickered on for them. For a day I had it filed as a curiosity: maybe branch names render heavier, maybe the position helps. Then I looked at the six responses. In every one of them, the bank name in the same JSON was fabricated — みずほ銀行 or 三井住友銀行, banks that are not on the page. The branch wasn't read either. It was invented along with the rest of the block, and the invention collided with the truth, because the printed branch is 本店営業部 — the single most common branch name in Japan.

My scorer cannot see that. Nothing inside one field can. Six of the "correct" cells in this benchmark are, on the co-occurrence evidence, fabrications that happen to be true.

This series has leaned hard on fictional ground truth — unguessable values, so that a right answer proves reading. The branch field is where that doctrine shows its edge: fictional ground truth only works if the fictional value isn't the category's mode. 本店営業部 was a real, maximally common phrase, and it turned one cell per model into a coin the model didn't even know it was flipping. The next version of this benchmark gets a fictional branch name, for the same reason the bank got one.

Blank or fiction

Put the two findings side by side and the operational picture is stark. The four fields a 5.6-gen @low actually reads are the title and the money triple — precisely the fields every automated validation looks at. The arithmetic reconciles because it was read. The names, dates, and bank details wrapped around that true arithmetic are, five times out of six, authored. A document that is half real is the hardest kind to distrust, and at this tier it isn't a degradation mode. It's the product.

The classification result from the earlier study still stands — at roughly 300 tokens a page these models see the title tier reliably, which makes @low a genuinely good routing gate. My catalog sentence for these variants says text read from images is unreliable at this resolution. After this column I'd sharpen the word: not unreliable — authored. Unreliable suggests you'll get a noisy version of your document. What you get is a fluent version of the average document, with your totals attached.

Postscript (July 18): The mixed-rate follow-up that the stamp article promised has now run, and its result reaches back into this article. On a sibling invoice where the tax is not a round 10% — so total ÷ 1.1 no longer equals the printed subtotal — the same @low variants return total ÷ 1.1 for the subtotal and total − total ÷ 1.1 for the tax, at every occlusion level, including zero. On that evidence, two of the four fields I counted as read above were most likely never read here either: they were derived from the total under a flat-10% assumption that this document's round tax rate made indistinguishable from reading. What a 5.6-generation @low reads on this invoice may be two things, not four — the title, and the total. The operational conclusion gets stronger, not weaker: the arithmetic doesn't reconcile because it was read. It reconciles because two of its three numbers were manufactured from the third. The harness and the recorded results are public at ldxhub-io/examples › analyzedoc/hanko-benchmark-v2. Full write-up to follow.

Method notes: this is the L0 (zero-occlusion) slice of the seal benchmark — 540 of its 3,240 jobs — read from the same scored output; twenty reads per model per field, criteria frozen in code before the run. The harnesses, ground truth, and run summaries are public at ldxhub-io/examples › analyzedoc/hanko-benchmark; the per-field analysis script behind this article ships in the same directory. Provider vision pipelines change — re-run before trusting any of this for anything current.