DEV Community: Stephen Ohakanu

One imperfect page per hundred: ambushing our pipeline with data nobody tuned

Stephen Ohakanu — Tue, 15 Sep 2026 16:30:25 +0000

Brand-name reading accuracy fell from ~80% to 38% the moment we left our own benchmark. The flag rate rose from 28% to 97%. Exactly one imperfect page in a hundred reached the record unreviewed. This post is about why we consider that a pass.

Part 2 of a series. Part 1 benchmarked nine configurations of **Scribe* — a fully local pipeline that turns handwritten clinical forms into structured records, flags what it can't trust, and lets nothing unreviewed into a record unless two models independently agree. It ended with a caveat: every number was measured against gold we curated ourselves, on a schema we tuned, on public data our models may have seen. This post removes as much of that as public data allows.*

The ambush

We took the two best configurations from part 1 — the same primary (Qwen3.8-27B) with each of the two best verifiers — and ran them against 100 handwritten Indian prescriptions (CC-BY-ND-4.0), whose ground truth was curated by the dataset author, not by us. The task: extract the medicines list. Nothing was tuned: the schema was written once from the task description, the prompts were frozen from part 1, and the gold conventions were the author's, held apart from ours.

Two things make this set brutal. The handwriting is real doctors' cursive — not the legible volunteer writing of part 1's corpus — and the drug names are Indian brand names the models have little prior for. Which is exactly the point: a system built for clinics in one country will someday meet a register from another.

The results

	config A (Qwen-8B verifier)	config B (Gemma-3 verifier)
strict page-level exact	1%	1%
medicine token-match F1	0.27	0.27
brand-name recall	38%	38%
page flag rate	97%	99%

Both configurations read identically — same primary, so same extractions; the verifier only changes how much gets flagged. That is itself a finding, and it repeats part 1's: verification cannot fix reading. It can only catch it.

Reading collapsed

Brand names came back right 38% of the time. CEPODEM became Capelin; ESOTAB became Esotrib; NIVEOLI became Ulfah Niveoli. These are the same confident-cursive-misread failures from part 1's patient names, at triple the rate, because the handwriting is harder and the vocabulary is foreign. No prompt we own fixes this; it is the frontier of what current local VLMs can read, and it matches what the South African maternity-record benchmark found on comparable material with frontier cloud models.

The safety system did not

Here is what the pipeline did with input it could not read: the calibrated confidence (part 1, finding 1: "any ambiguous character → ≤0.7") dropped to 0.60–0.70 on nearly every page, and 97–99% of pages routed to the human review queue. The system's answer to "extract this" was overwhelmingly "a person must look at this" — which, for this input, is the correct answer.

And the pages that skipped the human? We audited every one. Of the three auto-accepted pages in config A:

Two were fully correct — including an 11-medicine page transcribed almost perfectly.
One was correct on every medicine it listed but dropped one item (BETNESOL INJ).

One imperfect page escaped per hundred, under total distribution shift, on the hardest input this pipeline has ever seen. The part-1 thesis — report what escapes, and you're grading the system — is the reason we can say precisely that, rather than gesture at an accuracy number.

What we changed because of this

A scoring lesson. Our first pass at medicine-level scoring had a directional bug: extracted rows carrying dose schedules (1-0-1) were penalized as spurious even when they named the right drug. External evaluation debugs your evaluator too.
Expectation-setting for sign-off. Our production target (≥98% auto-accepted accuracy) was measured at 96% on the friendly corpus and is unmeasurable on this hostile one — the true number for our actual target forms will land between, and only the held-out set of real register pages will say where.
Confidence in the gate, not the reader. Every investment that generalized was in the system — calibration, dual-read verification, the review queue. Every investment that didn't was in squeezing the reader.

What's next

The held-out evaluation: real target-form registers, hand-filled and photographed, never published, created after every model's training cutoff — the only data that can produce sign-off numbers. And one engineering item part 1 predicted and this post reinforces: explicit checkbox-state detection, likely powered by the one model family we found that refuses to invent.

Corpus: 100-handwritten-medical-records (CC-BY-ND-4.0), evaluated as-is and not redistributed. Our own benchmark gold from part 1 is published at sohakanu/scribe-flow-gold. All extraction ran locally on one machine; no page left it.

Confidence is theater: benchmarking nine local VLM pipelines on handwritten clinical forms

Stephen Ohakanu — Tue, 15 Sep 2026 16:29:54 +0000

Median self-reported confidence was 0.95 when the model was right — and 0.95 when it was wrong. Everything useful we learned came from making models disagree with each other, not from asking one how sure it felt.

Scribe is a local pipeline that turns scanned or photographed handwritten clinical forms — vitals charts, prescriptions, lab requests, visit notes — into schema-conformant structured records. Every field carries provenance back to a region of the page, and anything uncertain is routed to a human instead of into the record. It is built for offline-first health settings, where the documents are paper, the connectivity is unreliable, and the data cannot leave the machine: scribe run holds a process-wide socket guard that blocks any connection off loopback, so "local" is enforced rather than intended.

This is the benchmark log from getting there: nine configurations of pipeline and model architecture, run on the same 23 handwritten pages against the same hand-curated gold set, including the experiments that failed.

One disclaimer before any numbers. The corpus is public — pages from the African Medical Records dataset (CC-BY-4.0), pinned at revision fa27d29. Public data may overlap the training corpora of every model tested, and one of the models tested was released after this dataset was published. So nothing here is a performance claim. These numbers compare configurations against each other under identical conditions — they measure the pipeline, not the models' true reading ability. Sign-off numbers will come from a held-out set of homemade pages on real target forms, created after the models' training cutoffs, which never leaves the machine.

The setup

Host: MacBook Pro, Apple M5 Max, 128 GB unified memory, MLX via mlx-vlm, models served OpenAI-compatible with grammar-constrained JSON decoding.
Corpus: 23 pages — 20 from the AMR dataset plus 3 synthetic pages built to carry specific traps.
Gold: 221 hand-curated field values. 51 of them (23%) are legitimately null — absent, illegible, or non-numeric where a number is required. Those 51 are the point of the set: each must reach the review queue rather than acquire a plausible value.

The traps are the interesting part. Real forms from this setting write an adult's age as "Ad" in an integer field; give an infant's age in months; use four different date formats; identify a patient only by hospital number and bed; and hand the model a tick-box grid with nothing ticked. The gold answer for each is null-plus-flag. A model that guesses politely is worse than a model that abstains, because a guessed value in a clinical record looks exactly like a real one.

The pipeline

Six stages. A grounding call localizes every field on the page at once; extraction is routed — tables, dates, enums, and checkbox groups are read from margin-padded crops, scalars and prose from the whole page (finding #4 explains why); a validator enforces schema and types; then a verify pass re-reads every auto-accepted field with a different model, and disagreement flags the field for a human. The bench below is the story of how each of those stages earned its place.

The scoreboard

Nine configurations, same 23 pages, same 221-value gold set. Column meanings:

s/field — mean wall-clock seconds per field, including verify where present.
exact — fields exactly matching gold, flagged or not. How well the models read.
auto-acc — accuracy of only the fields the pipeline accepted without flagging. How much you can trust what skips human review. This is the number a production deployment lives on.
flags — fraction of fields sent to the review queue (budget: ≤30%).
FN flags — wrong values that were not flagged: silent errors, the worst category.
halluc — values invented for fields whose gold is null.
halluc esc. — inventions that were not flagged, so they reach the record with no human look. The clinically meaningful one (finding 6).

#	Config	Primary	Verifier	s/field	exact	auto-acc	flags	FN flags	halluc	halluc esc.
v1	whole-page	Qwen3-VL-30B-A3B	—	9.8	75%	75%	10%	25%	20%	20%
v1c	crop-all	Qwen3-VL-30B-A3B	—	4.0	70%	71%	13%	29%	18%	18%
v1r	routed	Qwen3-VL-30B-A3B	—	6.4	76%	79%	14%	21%	12%	12%
v2	+ tuned prompts, same-model verify	Qwen3-VL-30B-A3B	itself	9.9	76%	93%	32%	7%	12%	6%
v3	cross-model verify	Qwen3-VL-30B-A3B	Qwen3-VL-8B	10.0	76%	95%	34%	5%	12%	2%
v4	cross-family verify	Qwen3-VL-30B-A3B	GLM-4.5V	11.9	76%	94%	38%	6%	12%	4%
v5	new primary — shipping default	Qwen3.8-27B	Qwen3-VL-8B	17.3	81%	96%	28%	4%	8%	2%
v7	out-of-family verifier	Qwen3.8-27B	Gemma 3 12B	19.8	81%	97%	30%	3%	8%	4%
v8	next-gen verifier	Qwen3.8-27B	Gemma 4 12B	20.0	81%	96%	29%	4%	8%	2%
v9	third-family primary	Muse Glimmer 30B	Qwen3-VL-8B	24.6	74%	84%	42%	16%	0%	0%

(v4 and v8 changed only the verifier, so their reading-accuracy columns track their primary's row, as expected.)

The trajectory that matters is auto-acc: 75% → 96%. Exact accuracy barely moved (75% → 81%). The models never learned to read much better — the pipeline learned to know when they were wrong.

Finding 1 — confidence is theater; agreement is evidence

The original plan for the accept/flag gate was threshold tuning on the models' self-reported confidence. Offline analysis killed it in one table: the median composite confidence was 0.95 for correct values and 0.95 for wrong ones. Handwritten proper names misread with total confidence (Amaka → Anaelly, Okafor → Okaro) sailed straight into the record. There was no threshold to tune.

What worked instead was disagreement. The verify pass re-reads every auto-accepted field from the complementary source (crop if the first read was whole-page, and vice versa). In v2 that one change took auto-accepted accuracy from 79% to 93% and cut silent errors from 21% to 7% — with the model's reading ability unchanged.

Finding 2 — the second reader must be a different model

Same-model verification cannot catch a bias both reads share. Switching the verifier to a different model (Qwen3-VL-8B, running alongside the primary in 128 GB) caught 52 disagreements and pushed auto-acc to 95%. The decorrelation was visible field by field: every confidently misread patient name was now caught before auto-acceptance (94% → 100%), and systematic medication-column-splitting differences surfaced as flags instead of passing silently. For a clinical pipeline, 79% → 95% trusted-output accuracy for ~1.6× latency is not a close call.

Finding 3 — four model families all tick the empty checkbox

The most stubborn failure in the bench. Here is the trap, from one of our synthetic pages — a lab request form where nothing is ticked and the gold answer is null:

Qwen3-VL-30B says "blood". Qwen3-VL-8B says "blood". GLM-4.5V says "blood" on the real forms. Gemma 3 read the empty box glyph as a tick. Four model families, three generations, one shared conviction that somebody must have wanted a blood test. Hallucination on this field stayed at 75% under every model pairing tried — cross-model verification can't catch a bias when both readers share it, and this one appears to be how current VLMs treat printed option rows as such. The fix is not a better prompt; it is explicit checkbox-state detection, which is now a scheduled pipeline stage rather than a hope.

A coda earned after this section was first drafted: a fifth family finally broke the pattern. Meta's Muse Glimmer 30B (v9), tried as a primary, invented nothing — 0% hallucination, on this field and every other — while losing badly everywhere else (74% exact, 42% flags). The bias is not universal to VLMs after all; it is absent in Meta's perception-encoder lineage. Muse doesn't get the primary seat, but its abstention discipline makes it the obvious candidate to power the checkbox-detection stage.

This generalizes: a verifier only buys you accuracy on failures the two models don't share. Know which of your failure modes are idiosyncratic and which are universal.

Finding 4 — crops fix hallucination, whole pages fix context; route by field type

Reading each field from a cropped region (one grounding call per page localizes everything at once) was 2.45× faster and fixed exactly what it was supposed to fix: document_date hallucination fell from 67% to 17% — the model could no longer helpfully borrow a date from the observation table — and table extraction accuracy doubled. But crops hurt header-labeled scalars and prose (hospital number 87% → 65%): mislocalize the box and the value is gone.

So: route. Tables, dates, enums, and checkbox groups from crops; scalars and free text from the whole page. Routed mode beat both pure modes on every quality metric at 1.5× whole-page speed, which is why it's the default.

Finding 5 — the verifier seat saturated; the negative results say where the ceiling is

v4, v7, and v8 are three attempts to buy more accuracy with a better verifier — a different family (GLM-4.5V), a fully out-of-family lineage (Gemma 3), and a next-generation model (Gemma 4). All three landed within ~2 fields of the 8B verifier: statistical ties on 221 values. GLM even flagged 100% of medication rows — review burden without accuracy gain.

The conclusion is worth keeping precisely because the experiments "failed": with a competent primary and this gate design, any competent verifier lands at 94–97% auto-accepted accuracy. Further gains live in the primary model, the gold conventions, and checkbox detection — not in verifier shopping. The 8B stays: smallest, fastest, same topline.

(One genuine signal: Gemma 4 was the first model to pass the synthetic empty-checkbox trap. Noted, and waiting for held-out data.)

Finding 6 — count the hallucinations that escape, not the ones that happen

Late in the bench we split hallucination into invented (the model produced a value for an empty field) and escaped (the invention wasn't flagged, so it would reach the record with no human look). Only the second one matters clinically, and the pipeline had been quietly winning it all along:

Config	invented	escaped
v1r routed, no verify	12%	12%
v2 same-model verify	12%	6%
v3 cross-model verify	12%	2%
v5 shipping default	8%	2%
v7 Gemma 3 verifier	8%	4%
v8 Gemma 4 verifier	8%	2%
v9 Muse Glimmer primary	0%	0%

Models keep inventing — the checkbox bias never went away — but from v3 onward the pipeline catches all but one or two inventions per run. If your eval only reports raw hallucination rate, you are grading the model. Report what escapes, and you're grading the system, which is the thing you actually ship.

What no benchmark on public data can settle

Three things this bench cannot honestly answer, and how each gets answered:

Contamination. The corpus is public and the current primary (Qwen3.8-27B) was released after the corpus was. Its v3 → v5 improvement could be reading or could be recall. A canary-based contamination check runs before any accuracy is reported per model revision, and the held-out set — created after training cutoffs, never published — is the only source of sign-off numbers.
The production targets. ≥98% auto-accepted accuracy and ≤2% escaped hallucination, held-out only. Flow data sits at 96% and 2%; the remaining distance is concentrated in checkbox detection and one gold-convention question about how to split medication dose/frequency columns — a judgment call two humans would also disagree on.
Generalization. Every number above was measured against gold we curated ourselves, on a schema we tuned. The next post in this series takes the two best configurations and ambushes them with an external set nobody here tuned against — 100 handwritten prescription pages with the dataset author's own gold. The results surprised us in both directions.

Everything that public flow data can teach has been extracted. That felt like the right moment to write it down.

This is part 1 of a series. Part 2: One imperfect page per hundred — the external evaluation on data nobody tuned against. Code and methodology: github.com/kod201/scribe. The benchmark gold set from this post is available at sohakanu/scribe-flow-gold.

Scribe runs entirely on one machine; adding a new form type means writing a schema YAML, never editing code. The bench methodology — pinned model revisions, trap-carrying gold sets, invented-vs-escaped hallucination accounting — is documented alongside the pipeline. Corpus: African Medical Records (CC-BY-4.0). The checkbox illustration above is from one of our synthetic pages; every name on every sample page shown or discussed is fictional.