Your invoice reader is confidently wrong 4 times in 5. I built one that tells you which fields to check.

Krit — Mon, 19 Oct 2026 04:21:30 +0000

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 2: Side Hustle

I extracted 972 CORD receipt totals with a local 2B vision-language model. It got 59% of them right. When it was wrong, it was wrong confidently: on the held-out test split, 92% of the 111 wrong totals carried a mean token log-probability above −0.05 — an average per-token probability above 0.95.

One receipt's total was 8,800,000.00. The model returned 88000.0, with a mean log-probability of −0.000002. It was certain, and it was off by a hundredfold.

That number is the whole project. A bookkeeper cannot hand a client "about 96% accurate". They need to know which values to double-check. So I built Underwrite: it reads receipts and invoices into structured fields, then issues a certificate per batch —

"Error rate on documents like these: ≤ 10%, with 90% confidence. 48% of totals auto-processed, the rest routed to a human."

— and it earns that sentence with arithmetic, not marketing.

What I Built

Underwrite is a local pipeline that turns invoices into structured fields and decides, per field, whether to auto-accept or send for review — with the accept threshold chosen so the long-run error rate on documents like the calibration set stays under a target you pick.

The customer is the one-person bookkeeping shop: a freelancer, a small accountant, someone digitising a shop's paper bills. Two things matter to them and neither is "accuracy":

What can I safely stop looking at?
What did the machine get wrong that I didn't catch?

The second question is the one nobody answers. Every parser tells you what it read. Almost none tell you which values to trust, and the ones that do sell it as a closed feature. Underwrite answers it locally, on your own laptop, and shows its work.

The certificate is the product. The extraction is table stakes.

Demo

The review UI runs in demo mode off a precomputed batch, so it needs no GPU and no network:

pip install -e ".[dev]"
python scripts/build_demo_batch.py     # freezes one batch from the public CORD corpus
uvicorn underwrite.app.server:app

Left: the document. Right: five fields, colour-coded — green auto-accepted, amber flagged for review, red parse failure. Flagged fields are the only editable ones. The right-hand panel is the certificate: the error budget, the certified threshold, what was auto-processed, how many auto-accepted fields were sampled for audit, and the assumption the whole thing rests on.

In the shipped demo batch: 8 of 20 totals auto-accepted, 12 routed to review, at a certified 10% error budget.

(Figures, screenshots and the screen recording live in reports/figures/ and reports/demo_video/: the risk-coverage curves, the ablation, the validity resplits, the repair curve, the audit alarm, and the review UI.)

demo video link->https://www.youtube.com/watch?v=k6hZj1PyeVc

Code

github.com/Krit-Jain/Underwrite — 85 files, 293 tests, and every number in this post is produced by a script in the repo. reports/tables/*.csv and reports/figures/*.png are committed outputs of those scripts.

How I Built It

The model, measured rather than assumed

google/gemma-4-E2B-it, loaded in 4-bit NF4 with bf16 compute on a 6 GB laptop GPU (RTX 4050).

Two things I had to discover rather than guess:

E4B does not fit. Even at 4-bit it needs more than the 6,140 MiB available, because of Gemma 4's per-layer input embeddings — vocab_size_per_layer_input: 262144 across 42 layers is ~5.6 GB on its own. Transformers quietly spills to CPU/disk and the 4-bit quantizer refuses to load. E2B (35 layers, hidden 1536) fits.
device_map="auto" must not be used. It splits the model onto CPU whenever free VRAM looks tight, which the quantizer rejects. Pinning {"": 0} is the difference between working and not.

Measured on 983 documents: 9.06 s/document, 2.47 hours wall clock, ~6.6 GB peak VRAM, 47 output tokens per document (reports/tables/extraction_cost.csv; VRAM from the smoke test). An image is compressed to 280 vision tokens regardless of receipt height, which is what makes a 6 GB card viable at all. The synthetic Indian invoices were much slower — 40.3 s/document, emitting 62 output tokens against 47 on CORD.

The two-pass trick

Generation gives you the answer but throws away the model's uncertainty. So every document gets a second forward pass: the same (image + prompt + generated JSON) is fed back with output_hidden_states=True, and I record, per field:

realised log-probability of each value token (mean, min), first-token entropy, top-1/top-2 margin;
the hidden-state vector at the value's last token, and the mean over its tokens, at four depths (roughly 25/50/75/90%).

Aligning a field to its tokens is the fiddly part — the chat template wraps the response in its own tokens, so the response is located by subsequence search rather than assumed to start at the end of the prompt. Verified on all 983 documents: 983/983 aligned, checked by reconstructing each value from its selected tokens' character offsets.

Five black-box signals, one white-box signal

The experiment is a fair fight at the same certified error rate:

	signal
S0	mean token log-probability
S1	verbalized confidence (ask the model to rate itself)
S2	validators only (arithmetic, formats, GSTIN checksum, date plausibility)
S2g	source grounding — does the value actually appear on the page?
S3	logprob + validators + grounding (the strongest black-box baseline)
S4	hidden-state probe only (white-box)
S5	S3 + probe (the system)

On CORD, per-field AUROC for the total field (n=292 held-out documents):

S0  logprob            0.632
S2  validators         0.525   ← near chance
S2g grounding          0.725
S3  black-box fusion   0.838
S4  probe only         0.914
S5  probe + fusion     0.952

The hidden states carry information the entire black-box stack does not. S4 — nothing but a PCA + logistic regression on the model's internal vectors — beats S3, which combines three hand-built signals.

One honest gap in that table: S1 is missing. The final run skipped the self-rating call to halve the runtime. It was measured on the earlier run over the same test split at AUROC 0.603 for total — weak, and consistent with the published finding that verbalized confidence is close to useless on receipts.

Does any of that depend on perfect OCR? Grounding is measured against CORD's ground-truth OCR words and the synthetic renderer's own text — oracle text, so absolute coverage is optimistic. Removing grounding entirely makes the comparison stronger, not weaker: S3 falls to 0.729 while S5 holds 0.915. Without grounding, S5 is essentially S4 (0.915 vs 0.914) — the black-box features add nothing, and the probe carries the signal alone.

That is the answer to "why open weights": a closed API gives you token probabilities at best. It does not give you the residual stream. The probe cannot exist there.

The gate: Learn-Then-Test

The accept threshold is chosen by fixed-sequence testing over a pre-specified grid: walk from the strictest threshold to the most lenient, and at each one test H0: risk among accepted > α with an exact binomial test at level δ. Stop at the first failure. Because the order is fixed and you stop at the first non-rejection, the family-wise error rate is controlled.

One correction I had to make to the textbook recipe: the strictest threshold accepts a single document, where P[Bin(1, α) ≤ 0] = 0.95 and no test can ever reject. Taken literally, the procedure never certifies anything — my first simulation certified 0/200 times. Thresholds that cannot reject even with zero errors are skipped.

Results

Held-out test split (total, n=292, base error 38.0%). These coverage columns are empirical: the threshold behind each one is selected on this same test set, so read them as the achievable frontier, not a promise. The certified numbers are in "Is the certificate real?" below.

method                AUROC   coverage@2%  @5%   @10%
S3 black-box fusion   0.838      8.2%     14.7%  46.2%
S4 probe              0.914     17.5%     30.5%  51.7%
S5 probe + fusion     0.952     41.4%     49.7%  62.0%

At a 5% error budget, S5 auto-processes half the totals — 3.4× what the black-box baseline manages.

Is the certificate real? 200 random resplits of the pooled calibration and test documents. "Certifies" is how often a threshold was found at all; the violation column is the share of certified resplits whose realised risk then exceeded α — the rest issued no promise, so they cannot break one.

α=0.10                   certifies   coverage   violates-if-certified
S3 black-box fusion       79/200      10.9%             20.3%
S4 probe                 200/200      18.0%              4.5%
S5 probe + fusion        200/200      48.1%             11.5%

α=0.05                   certifies   coverage   violates-if-certified
S3 black-box fusion        4/200      17.5%            100.0%
S4 probe                  20/200      14.8%             95.0%
S5 probe + fusion        200/200      29.4%              6.0%

S5 certifies on every single resplit — 48% coverage for a 10% budget, 29% for 5%. S3 certifies rarely, and when it does it is usually lying: all four of its α=0.05 certificates violate, and a fifth of its α=0.10 ones do. A score that cannot separate a hard document from an easy one cannot safely make this promise, and the resplits expose it.

I want to be straight about the one number that isn't clean: at α=0.10 the violation rate is 11.5%, above δ=10%. Sampling noise accounts for much of it (with 200 splits the standard error is ~2%), and my simulations show that exceedances should run slightly above δ when you measure realised risk on a finite test part — the guarantee is about population risk, and realised risk fluctuates across it. But it is above the line, and it's in the table rather than in a footnote.

What happens when the documents change

Calibrate on Indonesian receipts (CORD), deploy on synthetic Indian GST invoices, and the certificate does not break — it becomes useless:

CORD-calibrated, deployed on Indian invoices
  source coverage 42.9%   →   target coverage 2.5%   target risk 0.0%

The scores shift even though the target set is easier (90.5% of totals correct, versus 59% on CORD). The threshold calibrated on hard documents is simply never cleared by an easier distribution. No wrong values were auto-accepted — but it auto-accepted almost nothing. I re-ran this after fixing a money-convention bug in the grounding feature and the table came out identical, so the collapse is a property of the score distribution, not an artifact.

So recalibrate. The business question is how many labelled documents a new client's template needs:

α=0.10   k=25 → 76% certify at 93% coverage
         k=50 → 100% certify at 93% coverage
α=0.05   k=50 → 70% certify at 91% coverage
        k=100 → 100% certify at 92% coverage

Roughly 50 labelled documents per new client template to certify a 10% budget; about 100 for 5%. That is a number a bookkeeper can act on.

The audit lane

Flagged-item labels are selection-biased — humans only look at what the system already distrusts, so they can never tell you the error rate among auto-accepted fields. Underwrite instead samples a random slice of auto-accepted fields and checks those.

The alarm asks one question: is the true error rate above α? That needs the Clopper-Pearson lower bound to exceed α. My first version alarmed on the upper bound. That sounds similar and is wrong: with only a handful of audits the interval is so wide that the upper bound is always above α. A reviewer caught it — that rule fired on 52–90% of clean streams.

With the lower bound, and a Bonferroni correction across the repeated checks, over 500 seeds:

true error rate	false alarms
2%	0.0%
4%	0.0%
6%	0.2%
8%	0.4%

against a 5% budget. Detecting a jump to 40% error inside a 500-document window:

audit rate	median delay	missed
5%	410 documents	73%
10%	407 documents	46%
20%	357 documents	12%

On the real pipeline — 616 accepted CORD receipts at 1.1% error, then a template change — the stream I committed happens to be one of the misses: the lane stays silent through the clean segment, and the lower bound climbs to 0.08 after the switch without reaching α. I left that in rather than hunting for a seed that alarms.

The honest summary: the alarm doesn't cry wolf, but at a 10% audit rate it misses about half of drifts within a short window. A quiet, slow monitor is still useful; a noisy one gets switched off.

Why Does Open Innovation Matter?

Three reasons, and the first is decisive.

1. The best signal is invisible to closed APIs. The probe reads the model's hidden states. No hosted API exposes them, at any price. On this corpus a PCA over those vectors alone (AUROC 0.914) beats every black-box signal combined (0.838). That gap is not a tuning difference; it's an information difference, and it exists only because the weights are open.

2. A certificate is only valid for a frozen model. My numbers are calibrated against one specific quantized checkpoint. If Google ships an update, a hosted endpoint might change silently and my calibration becomes a lie. Pinned open weights keep it honest — I pin the revision.

3. The documents never leave the laptop. These are someone's supplier invoices with their margins on them. Running locally isn't a nice-to-have here; it's the reason a bookkeeper can use this at all. The marginal cost is electricity measured in a 2.47-hour run on a laptop GPU, not a per-page bill from an API.

There's a fourth, less flattering reason: I could measure it. Every number above came from a script I can re-run, against a checkpoint I can inspect. With a closed model I'd be calibrating against something I can't see and can't freeze.

Prize Categories

Best Use of Gemma — Gemma 4 E2B is the extractor, and the hidden-state probe is only possible because the weights are open.

Limitations

Being precise about what this does not show:

The guarantee is about the total field only. total has one value per document, so calibration samples are independent. Per-field guarantees on subtotal/tax/date/vendor would be invalid — the same document's fields are correlated, and the natural per-field procedure silently violates its own promise (see arXiv:2608.14639). Vendor and date are shown in the UI but always routed to review.
~290 calibration documents is thin. That is why α=0.05 is at the edge of feasibility, and why the α=0.10 violation rate sits slightly above δ.
A 2B model on a 6 GB card sets the ceiling. Base error is 38% on total. A larger model would raise coverage; it wouldn't change the machinery.
Exchangeability is assumed. The audit lane exists because that assumption fails, and the shift experiment shows exactly how (silently, and by collapsing coverage rather than by producing wrong auto-accepts).
The synthetic Indian invoices have no OCR pass — grounding is measured against the text the renderer drew, so OCR error is not modelled there.
CORD is Indonesian and CC-BY-4.0, and the corpus has no vendor/date ground truth, so the five-field story can only be told as a three-field one on this data.

Prior work

This builds directly on open work and claims no novelty for the machinery:

Learn-Then-Test (Angelopoulos et al., arXiv:2110.01052) — the fixed-sequence threshold calibration.
VerifyDoc (bhaskargurram-ai/verifydoc, Apache-2.0) — per-field calibrated confidence, source grounding, accept/review at a target selective risk. Its reported CORD AUROCs (grounding 0.82, token-probability 0.69, verbalized ~0.50) are why grounding is in my baseline and why my verbalized signal is reported as near-useless.
"Valid Per-Field Selective Risk Control for Document Extraction" (arXiv:2608.14639) — shows the naive per-field procedure violates its own guarantee. That paper is the reason my certified guarantee is restricted to total.
CORD (Park et al.) — the receipt corpus.

The contribution here is narrow and I'll state it narrowly: at a matched certified error rate on a laptop-class open model, a hidden-state probe auto-processes substantially more documents than a strong black-box stack that includes grounding.

DEV Community: Krit