DEV Community

Kevin Li
Kevin Li

Posted on Fully Autonomous

The OCR Was Right. The Invoice Automation Was Still Wrong.

#ai #automation

An invoice can be read perfectly and still be posted incorrectly. That distinction matters more than most OCR accuracy demos suggest.

Consider a partially paid invoice with these three lines:

Invoice total: $1,080
Payment received: $300
Amount due: $780
Enter fullscreen mode Exit fullscreen mode

A text reader can recover all three numbers without a character error. A downstream parser can then assign $780 to invoice_total because it picked the last monetary value. The failure is semantic: it selected a real number with the wrong business meaning.

That is why I would not put a single “OCR confidence above 95%” condition between an incoming PDF and an accounting write. Confidence in the text is not confidence in the proposed transaction.

Three questions, three separate tests

First, did the reader recover the source? If a digital PDF has an embedded text layer, use it when it is reliable. If the page is an image, OCR may be needed. Compare the resulting text with the document, including decimal points, minus signs, currency, and line breaks.

Second, did the extractor assign each value to the right field? A model may return valid JSON and still confuse invoice total, balance due, prior balance, and payment received. Test on documents where those values differ. If every test invoice is unpaid, invoice_total and amount_due may be equal, hiding a mapping defect.

Third, does the proposed record agree with the business system? The PDF alone cannot tell you whether the $300 payment is already in the ledger, whether the supplier is approved, or whether the invoice is a duplicate. Those require trusted records and explicit policies.

My preferred contract for a consequential field is not just name + value + confidence. It is closer to:

{
  "field": "invoice_total",
  "value": "1080.00",
  "currency": "USD",
  "source_label": "Invoice Total",
  "source_page": 1,
  "validation": ["subtotal_plus_tax_matches"],
  "status": "proposed"
}
Enter fullscreen mode Exit fullscreen mode

The value remains proposed until the workflow validates the supplier, relevant PO or receipt, duplicates, and the posting authority. A missing field should remain missing; substituting the nearest plausible amount is a dangerous fallback.

A better test set

Before evaluating a parser or model, include at least these cases:

Case The failure it can expose
Unpaid invoice Basic reading and field assignment
Part-paid invoice Total versus amount-due confusion
Credit or negative line Lost sign or incorrect normalization
Same supplier and invoice number twice Duplicate handling
Invoice with a plausible but stale PO Extraction succeeds while business matching fails

These are categories, not a claim that five documents establish production accuracy. Your test set still needs representative suppliers, formats, scan quality, and exception volume.

I made the total-versus-amount-due failure reproducible with two synthetic invoices and the raw OCR output in this KelenAI field test. It compares text reading with field mapping and states exactly what the demo does not prove.

The broader lesson: measure errors at the boundary where they change a business record. An impressive extraction score is useful, but the wrong total posted to the right supplier is still a failed automation.

Top comments (0)