I ran 110 vision-extraction jobs against a synthetic invoice. The low-resolution modes of the newest models never once returned the correct document — and the way they failed is worse than random noise.
All measurements in this post are as of July 10, 2026. Vision pipelines change; if you're reading this later, re-run the test before trusting the numbers.
The invoice that would have passed review
Here is a fragment of what GPT-5.6 (Sol, low-detail image mode) returned when I asked it to extract a synthetic invoice from a PNG:
{
"vendor": "K Northwind Trading Ltd",
"bill_to": "Accora Manufacturing Inc.",
"invoice_number": "INV-2025-0731",
"date": "2025-07-31",
"subtotal": 757.5,
"tax": 75.75,
"total": 833.25
}
Every number is correct. Subtotal, tax, total, all four line-item amounts, all quantities, all unit prices — perfect, down to the cents. The vendor is right too.
The bill-to company does not exist. The real document says Aozora Manufacturing Inc.; the model wrote Accora (and, on other runs, Alcora). The invoice number is wrong in one systematic way: the year. The document says INV-2026-0731; the model wrote INV-2025-0731 — and then, consistently, dated the invoice 2025-07-31 to match. The two invented values agree with each other.
That last detail is the one that bothers me. A misreading scatters; over five runs you'd expect 2020, 2028, a garbled digit. This didn't scatter. Twenty out of twenty runs — five repeats, two image formats, two providers — said 2025, and the date field followed along. The model didn't fail to read the year — it composed a coherent document in which the year is 2025.
An extraction where the totals reconcile but the counterparty is fictional is precisely the kind of error that sails through an accounts-payable check. Nobody re-verifies the customer name when the arithmetic is clean.
The false positive that came first
Before I trust a finding like this, I have to tell you about the bug I almost blamed on the model — because it changed how I ran everything after.
My platform derives the output schema from an example_output the caller provides. My first test used example values like 10.0 and 22.0. Somewhere between my MCP client and the Java layer that infers the schema, JSON serialization collapsed 10.0 into 10 — an integer. The inferred schema said integer, and every model dutifully returned integer totals. 833.25 came back as 833, and in one configuration as 83325 — the decimal point simply gone.
For about an hour I had a tidy, wrong conclusion: "the new flagship's low mode corrupts decimals." Then I asked the question I should have asked first — did the example even survive the JSON round-trip? — and the whole finding evaporated. With examples like 12.34, every model produced clean decimals.
Lesson one, before any lesson about models: when output looks corrupted, suspect your test harness before the model. X.0 is not a safe way to say "this field is a float" in any pipeline that round-trips JSON.
That embarrassment is why everything below is n=5, scripted, with the pass/fail criteria frozen in code before the runs started.
The numbers
The setup: one synthetic invoice (all names fictional), rendered to JPEG and PNG. Ground truth: vendor K Northwind Trading Ltd., bill-to Aozora Manufacturing Inc., invoice number INV-2026-0731, total 833.25. A result counts as OK only if all four fields match. Eleven model configurations, two formats, five runs each — 110 jobs through my document-analysis API, which maps an @low / @high variant onto each provider's image-detail setting.
The low-detail modes of the GPT-5.5 and GPT-5.6 generations (Sol, Terra, Luna; direct and Azure-hosted): zero correct extractions in 70 attempts. Not "low accuracy" — zero. The failures were the kind shown above: arithmetic intact, identities invented.
The controls, same images, same five repeats:
| Configuration | Result |
|---|---|
| GPT-5.5 / GPT-5.6 family, low detail (7 configs) | 0 / 70 correct |
| GPT-5.4 (Azure), low detail | 18 / 20 correct; 2 near-misses (a dropped space: "KNorthwind") |
| GPT-5.4 mini (Azure), low detail | 10 / 10 correct |
| Gemini 3.5 Flash, low detail | 10 / 10 correct |
| GPT-5.6 Sol, high detail | 10 / 10 correct |
An earlier run of the same protocol used a version of the invoice with smaller, lighter text. Same models, same 0-for-70 — but there the failures were total: complete fictional invoices, different every time. A vendor called Kramerwick Ltd. with a Brussels address. KittenPaws, LLC on Meowth Street. A Japanese company name the document never contained. Line items for services that don't appear anywhere in the image.
So the failure mode is not binary; it slides with legibility. Illegible source → the model invents the whole document. Partially legible source → the model reads what it can (the big bold totals) and invents the rest (the small print), stitching both into one internally consistent answer. The second mode is the dangerous one. A wholesale fabrication looks wrong at a glance. A half-real document does not.
The generational irony
Look at that table again. The token budgets are essentially identical: measured directly against the provider APIs (my gateway meters pages, not tokens), the OpenAI-family low mode spends about 315 tokens per image of this size; Gemini's low setting spends about 258. Gemini reads the invoice perfectly on the smaller budget. And GPT-5.4 — the older generation, same 315 tokens, same hosting path — gets it right, with failures that look like classic OCR noise: a dropped space, a mangled character. It degrades the way you'd expect a reader to degrade.
The 5.5 and 5.6 generations do something different with the same pixels. Where 5.4 returns less, they return other. My best reading — and I'll flag it as interpretation, not measurement — is that the newer generations are stronger generators, and when perception runs out, generation fills the gap with whatever is most plausible. "Aozora" becomes "Alcora": right silhouette, right length, wrong word. 2026 becomes 2025, and the date agrees, because a coherent story beats a faithful blank.
Newer model, better prose, worse witness.
The reversal
At this point the obvious move is to kill the low-detail image modes entirely. I almost did. Then I ran the opposite experiment: instead of asking the same models to read documents, I asked them to sort them.
Round one: five synthetic documents with distinct layouts — invoice, receipt, business card, contract, blank page. Nine configurations (the seven "guilty" low modes plus two controls), three runs each: 135 / 135 correct, including refusing to force the blank page into a category.
Round two was designed to be unfair. Three Japanese business documents — 請求書 (invoice), 御見積書 (quotation), 注文書 (purchase order) — with identical layouts, identical tables, identical amounts, identical document numbers. The only difference is the title and one label line. You cannot sort these by shape; you must read the title. I then degraded them: photocopier noise, a 2° tilt, JPEG quality 25. Nine configurations, seven materials, three runs: 189 / 189 correct.
Same models. Same low-detail budget. 0-for-70 at reading the fine print; 324-for-324 at reading the headline and the shape.
The capability boundary is suddenly crisp: at ~300 tokens per page, these models see the title tier of a document reliably and the body tier not at all — and where the body tier fails, the 5.5+ generations fill it with fiction.
What I did about it
Delisting was the wrong answer — the classification result is real, and a sorting gate that costs a tenth of a high-detail read is genuinely useful (mixed scan folder → cheap low-detail triage → route each type to the right extraction pipeline). Silence was also the wrong answer: my catalog said the low modes were for "clean, large-text documents," and my own test — a clean, large-text document — had just proven that description wrong.
So the fix was one sentence of honesty. Every affected variant in my catalog now reads:
Low-resolution mode for fast, economical PDF extraction and document classification; **text read from JPEG/PNG images is unreliable at this resolution.**
(PDFs are unaffected in my measurements — the OpenAI-family models read PDF input through its text layer, so image downscaling never touches it. That's also why this failure hid so well: every PDF test passed.)
The formats stay listed. The capability stays available. The sentence tells you what 110 jobs taught me: what it's for, and what it will quietly get wrong.
The rule I'm keeping
My pipeline runs on a rule I've kept for a long time: absorb failures with deliberate retries, and always return a result. But a retry only helps when the next attempt can go differently — a network hiccup, a rate limit. This failure is worse than the structural kind. A malformed schema at least announces itself; you can validate before you send. A fabricated bill-to announces nothing. The request succeeds. The JSON validates. The totals reconcile.
You cannot retry your way out of this, and you cannot fully predict it either. What you can do is measure where the boundary sits — n=5, criteria frozen, controls included — and then write the boundary down where your users choose models. Not in a postmortem. In the catalog, in the sentence they read before they click.
The models will keep getting better at writing. That is exactly why "I couldn't read this" increasingly comes back as fluent, internally consistent, confidently formatted text. The shape is real. The details may be fiction. Design — and document — accordingly.
Method notes: 110 extraction jobs + 324 classification jobs against synthetic documents (all names fictional), run July 10, 2026 via my document-processing gateway with per-provider low/high image-detail variants; token figures measured directly against provider APIs. Pass/fail criteria were fixed in the harness before execution. The generation and test scripts, plus the recorded runs, are public: ldxhub-io/examples › analyzedoc/low-detail-study.
Top comments (2)
The shape-vs-text distinction is important for anyone using vision models in workflows.
A model can preserve layout confidence while inventing the actual fields, which is a dangerous combination. The output looks structurally credible exactly when the data needs external verification.
That inversion is the part that unsettles me most. Legibility and structural confidence move in opposite directions: the harder a field is to read, the more the model leans on layout priors to fill it. So the output looks most composed precisely where it's least grounded — structurally credible right when the data needs external verification, exactly as you say.
The one cheap defense I've landed on is a within-document consistency check — e.g. subtotal + tax = total. It won't catch a fabricated vendor name (nothing internal to the doc can), but it turns "plausible" into "verifiable" for the fields that reconcile, which is often enough to route a document to human review instead of trusting it silently.