Where vision models stop reading — and start inventing

Earlier this week I published a strange finding: GPT's low-detail image mode doesn't misread documents it can't see — it invents them, fluently, with reconciling totals. That was one failure mode, in one model family, at one legibility level.

It left an uncomfortable question: where exactly does each model stop reading — and what does it do after that? Leave the field blank, or fabricate something plausible?

One result to hold onto while you scroll: a model that can no longer read a document can usually still tell what kind of document it is. That held across almost the entire board.

So I built a ladder.

Then I ran 27 vision model variants down it: 4,158 jobs, about $141 at list price, one afternoon. This post is the map.

---

The setup, in one paragraph

One Japanese invoice, rendered on a fixed 2480×3508 canvas (A4 at 300 dpi), then degraded through seven simulated scan resolutions: 300 → 150 → 100 → 70 → 50 → 35 → 25 dpi (L0–L6). Degradation is resampling only — no noise, no blur, no rotation — so legibility is the only variable. The invoice carries twelve fields across four font tiers: a 28 pt title, 16–14 pt "large" fields (total, invoice number), 10.5 pt body fields (counterparty, dates, amounts), and 7.5 pt fine print (bank details). Every value is fictional and unguessable, and subtotal + tax = total reconciles — so a plausible but wrong answer is detectable, not just a wrong one. Each variant reads each ladder step five times. The extraction prompt is deliberately neutral: it never says what to do with unreadable text, because that choice is the thing being measured.

Scoring is deterministic, four classes per field: correct / near (edit distance 1, strings only) / blank / fabricated.

---

The map

Rows are model variants, columns are ladder steps, color is the fabrication rate on the 10.5 pt body tier — the tier where invoice counterparties and amounts live. White means the model either read correctly or stayed silent. Red means it filled unreadable fields with invented values.

The companion table below gives each variant's frontier: the deepest ladder step where it still keeps ≥90% field accuracy, per tier (× = below 90% already at the crisp 300 dpi original).

model	title	large	body	fine	body fab @25dpi	classified correctly
openai/gpt-5.6-sol@high	L6	L5	L5	L2	60%	84/84
openai/gpt-5.6-sol@low	L6	×	×	×	64%	84/84
openai/gpt-5.6-terra@high	L6	L5	L4	L3	4%	84/84
openai/gpt-5.6-terra@low	L6	×	×	×	42%	84/84
openai/gpt-5.6-luna@high	L6	L5	L4	L2	96%	84/84
openai/gpt-5.6-luna@low	L6	×	×	×	80%	81/84
openai/gpt-5.5@high	L6	L5	L4	L2	86%	84/84
openai/gpt-5.5@low	L6	×	×	×	62%	84/84
openai/gpt-5.4@high	L6	L5	L4	L2	46%	84/84
openai/gpt-5.4-mini@high	L6	L5	L4	L0	78%	84/84
azure/gpt-5.6-sol@high	L6	L5	L5	L0	90%	84/84
azure/gpt-5.6-sol@low	L6	×	×	×	76%	84/84
azure/gpt-5.6-terra@high	L6	L5	L4	×	56%	84/84
azure/gpt-5.6-terra@low	L6	×	×	×	50%	84/84
azure/gpt-5.6-luna@high	L6	L5	L4	×	98%	84/84
azure/gpt-5.6-luna@low	L6	×	×	×	80%	84/84
azure/gpt-5.4@high	L6	L5	L4	×	98%	84/84
azure/gpt-5.4@low	L6	L5	×	×	58%	84/84
azure/gpt-5.4-mini@high	L6	L5	L4	×	88%	84/84
azure/gpt-5.4-mini@low	L6	L5	×	×	54%	84/84
google/gemini-3.5-flash@high	L6	L6	L6	×	10%	84/84
google/gemini-3.5-flash@medium	L6	L6	L6	L5	10%	84/84
google/gemini-3.5-flash@low	L6	L6	L6	L0	0%	84/84
anthropic/claude-fable-5	L6	L6	L6	L4	10%	84/84
anthropic/claude-sonnet-5	L6	L6	L5	L4	26%	84/84
anthropic/claude-opus-4-8	L6	L6	L5	L4	26%	84/84
bedrock/global.amazon.nova-2-lite-v1:0	×	L5	×	×	80%	63/84

The biggest surprise in this table isn't where the frontiers sit. It's what happens past them — some models go silent, and some keep talking.

Six observations fell out of the map.

---

1. "@low" means different things per provider

google/gemini-3.5-flash@low — the second-cheapest variant on the board — read the body tier correctly at every step down to 25 dpi, with zero fabrications. Under exactly the same conditions, every OpenAI and Azure @low variant collapsed at L0, on the pristine original. Same suffix, opposite behavior. The difference isn't the models' eyesight; it's what each provider's low-detail pipeline does to the image before the model ever sees it.

---

2. For GPT @low, a worse scan is a better scan

The @low accuracy curves are not monotonic. Most GPT @low variants read a 70 dpi scan better than the 300 dpi original — body accuracy climbing from ~40% at L0 to 70–80% at L3–L4 before falling again. My resampling acts as an anti-alias filter for the provider's own aggressive downscale. The practical corollary is genuinely odd: if you are stuck with a @low pipeline, pre-blurring your documents can improve extraction.

---

3. After collapse, models split into fabricators and blankers

What a model does past its frontier is a personality trait, and it's measurable. At 25 dpi, most GPT @high variants fill 75–100% of the body fields they can no longer read with invented values. openai/gpt-5.6-terra@high is the outlier of the entire board: 96% of its failures are blanks. Anthropic and Google models fail less to begin with and fabricate less when they do (0–26%). If your pipeline feeds payment systems, a blanker that admits defeat is worth more than a stronger reader that bluffs.

---

4. Same model, different gateway, different eyes

gpt-5.6-sol@high reads the 7.5 pt fine tier at 100% down to 100 dpi when called via OpenAI — and starts at 92% and degrades immediately when the same model is called via Azure. The failure style shifts too: terra's blank rate drops from 96% (OpenAI) to 39% (Azure). This matches an earlier measurement suggesting the Azure pipeline applies a lower effective-resolution ceiling before the model ever sees the document. Your gateway choice is silently part of your model choice.

---

5. Fabrication doesn't need degradation (teaser)

One fine-print field held a fictional bank whose name is one character away from a real megabank. At 300 dpi — fully legible, five out of five runs — some models "corrected" it to the real one. 48 substitutions across Gemini variants, while a fictional credit union with no real-world neighbor was read perfectly under the same conditions. The trigger isn't legibility; it's the existence of a nearby real entity. This one deserves its own write-up, with the receipts. Coming separately.

---

6. Classification survives reading loss

25 of 27 variants classified all 84 documents (invoice / receipt / business card / meeting minutes) correctly at every degradation step — including variants whose extraction had collapsed completely. A model that cannot read a document can still tell what kind of document it is. The two exceptions are instructive: the cheapest model on the board confuses receipts with invoices (21 out of 21 times — consistently, not randomly), and one @low variant dropped three classifications at the bottom of the ladder.

---

What I'd take into production

• Route by tier, not by document. Titles survive almost anything; fine print dies first. If a field matters, measure the frontier of the tier it lives in.
• Pick blankers for payment fields. A fabricated bank name passes every visual plausibility check. Prefer models that return "" over models that return something convincing.
• Don't assume @low is one thing. Benchmark the variant you'll actually call, on the gateway you'll actually use.

---

Reproduce it (a free key is enough)

Everything — the deterministic material generator, the runner, the scorer, the raw outputs of all 4,158 jobs — is published:

github.com/ldxhub-io/examples → analyzedoc/legibility-benchmark/

The materials are byte-identical on any platform (the generator downloads a pinned, checksum-verified font). A three-variant reproduction subset runs in 147 jobs ≈ 17,600 credits, which fits inside LDX hub's free tier (25,000 credits/month, no card):

python3 gen_materials.py
export LDXHUB_API_KEY=...   # free key: gw.portal.ldxhub.io
python3 run_benchmark.py --models ume --t1-instances A --t1-reps 3 --t2-reps 1 --yes
python3 score_results.py && python3 report.py

Because raw model outputs ship with the results, you can disagree with my scoring rules and re-score everything without re-running a single job.

Full disclosure: I run LDX hub. It builds no models — it's the harness here, not a subject. One API key across OpenAI, Azure, Google, Anthropic and AWS is the only reason a 27-variant matrix fits in one afternoon, and that convenience is exactly what I'm selling. The measurements stand on the published raw data either way.

---

Caveats

Degradation is synthetic resampling, not real scanner noise — claims are limited to simulated legibility. One document type, one language (Japanese; if anything, a harder test than Latin script). The strict scorer counts character-level misreadings as fabrications, which flatters nobody. Results are a July 2026 snapshot; the ladder re-runs on every model addition, so the map will stay current.

The next time a provider ships a new vision model, it gets a row within a day. That's the point of building a ladder instead of writing a review.