Invoice extraction has a property almost no other extraction task has: the document contains a checksum of itself. Line items sum to a subtotal, tax is a stated percentage of a stated base, and the total is their sum. If the extracted numbers do not reconcile, something is wrong — and you know it before a human ever looks, without a confidence score and without a labelled test set.
Why invoices are the good case
Most structured extraction fails silently: a model reads a contract, returns a plausible date, and nothing in the world objects. Invoices object. That makes them the right first project for anyone building extraction, and it makes the validator — not the prompt — the interesting part of the build.
The pipeline: get text or an image in front of a model, extract to a strict schema, validate arithmetically and against your own records, and route the failures. Only the middle step involves a model, and it is the step you will spend the least time on.
One decision comes before all of it: text or image. If the PDF has a real text layer, extracting the text and sending that is cheaper, more faithful for long strings like account numbers, and reproducible. Sending the rendered page to a vision model instead preserves layout — which is what tells you that a number in the bottom right is the total — and handles scans, but costs image tokens and reads long digit strings less reliably. The pragmatic answer is text first, image on validation failure, which is the repair path below and which means the expensive route is used on the small fraction that needs it.
The schema
Write the schema first and make every ambiguity explicit. Nullable where a field is genuinely optional; a fixed vocabulary where a field has one; and strings for anything you will parse yourself.
{
"invoice_number": "string",
"issue_date": "YYYY-MM-DD",
"due_date": "YYYY-MM-DD or null",
"currency": "ISO 4217 code, e.g. GBP",
"supplier": {"name": "string", "tax_id": "string or null"},
"buyer": {"name": "string or null"},
"lines": [
{"description": "string",
"quantity": "decimal as string, e.g. '2' or '1.5'",
"unit_price": "decimal as string, e.g. '19.99'",
"line_total": "decimal as string"}
],
"subtotal": "decimal as string",
"tax_rate": "decimal as string, percent, e.g. '20' — or null",
"tax_amount": "decimal as string",
"total": "decimal as string",
"notes_unclear": "string or null"
}
Numbers as strings is not squeamishness. JSON numbers are double-precision floats, and a model emitting 1234.10 can hand you something that is not exactly 1234.10 once it has been through a float. Strings preserve exactly what was on the page, and your code decides how to interpret it.
notes_unclear earns its place: instruct the model to describe anything it could not read rather than guessing. “The tax line is partly obscured by a stamp” is worth more to a reviewer than a confident wrong number.
Money in integers, always
Parse each decimal string into an integer number of minor units immediately, and do every subsequent comparison in integers. Floating point in a reconciliation check produces failures that are not real and successes that are not either.
from decimal import Decimal, InvalidOperation
MINOR = {"JPY": 0, "KRW": 0, "BHD": 3, "KWD": 3, "TND": 3} # non-2 exponents
def to_minor(s, currency):
"""'1,234.50' + 'GBP' -> 123450 (integer pence). Raises on nonsense."""
if s is None:
raise ValueError("missing amount")
cleaned = str(s).replace(",", "").replace(" ", "").strip()
cleaned = cleaned.replace("(", "-").replace(")", "") # (12.00) = -12.00
try:
d = Decimal(cleaned)
except InvalidOperation:
raise ValueError("unparseable amount: " + repr(s))
exp = MINOR.get(currency, 2)
q = d.scaleb(exp)
if q != q.to_integral_value():
raise ValueError("more precision than the currency has: " + repr(s))
return int(q)
Three traps are handled there and each one is real. Thousands separators, which differ by locale and will appear as 1.234,50 on European invoices — detect that pattern separately rather than stripping commas blindly. Parenthesised negatives, which are how credit notes are written. And currencies without two decimal places: yen has none, several Gulf currencies have three, and a hard-coded factor of 100 is wrong for all of them. Representing money as integers is the general rule this follows.
Validation: the arithmetic checks itself
def validate(inv):
cur = inv["currency"]
m = lambda s: to_minor(s, cur)
problems = []
# 1. Each line: quantity x unit_price == line_total (allow 1 unit rounding)
for i, ln in enumerate(inv["lines"]):
qty = Decimal(str(ln["quantity"]).replace(",", ""))
expect = int((qty * Decimal(str(ln["unit_price"]).replace(",", ""))
).scaleb(MINOR.get(cur, 2)).to_integral_value())
if abs(expect - m(ln["line_total"])) > 1:
problems.append("line " + str(i + 1) + ": qty x price != line_total")
# 2. Lines sum to subtotal
line_sum = sum(m(ln["line_total"]) for ln in inv["lines"])
if abs(line_sum - m(inv["subtotal"])) > len(inv["lines"]):
problems.append("lines do not sum to subtotal")
# 3. Tax consistent with the stated rate
if inv.get("tax_rate") is not None:
rate = Decimal(str(inv["tax_rate"])) / 100
expect_tax = int((Decimal(m(inv["subtotal"])) * rate
).to_integral_value())
if abs(expect_tax - m(inv["tax_amount"])) > 2:
problems.append("tax_amount does not match tax_rate x subtotal")
# 4. Subtotal + tax == total
if m(inv["subtotal"]) + m(inv["tax_amount"]) != m(inv["total"]):
problems.append("subtotal + tax != total")
# 5. Sanity, not arithmetic
if inv.get("due_date") and inv["due_date"] < inv["issue_date"]:
problems.append("due before issue")
if m(inv["total"]) <= 0:
problems.append("non-positive total")
return problems
The tolerances are the part to think about. A single unit of the minor currency per line is right, because invoices round each line and the rounding is legitimate; a tolerance proportional to the number of lines on the subtotal check follows from the same reasoning. Tax gets two units because rate rounding compounds. Anything wider than that stops being a validator.
Notice what these checks do not need: a labelled dataset, a confidence score, or a second model. They catch transposed digits, missed lines, a quantity read as a price, and a decimal point in the wrong place — which between them are most extraction errors that matter.
The repair path
A validation failure is not automatically a human’s problem. Two cheap attempts first, in this order:
- Re-ask with the failure attached. Send the same document back with the problem list and instructions to correct only the fields involved. A model told “the lines sum to 480.00 but you reported a subtotal of 420.00; re-read the line items” usually finds the line it missed. One repair round is worth it; a loop is not — cap at one, or two if you must.
- Change the input, not the prompt. If the first attempt used extracted text, retry with the rendered page image, or the reverse. Most persistent failures are a layout the text extractor mangled, and the other modality often reads it perfectly. Different input beats a differently worded prompt nearly every time.
- Then a human. With everything: the document, the extraction, the specific failed check and the model’s
notes_unclear.
Log which path resolved each invoice. If the image retry fixes a third of failures, your text extraction is the problem and that is a cheaper thing to fix than anything about the model.
When it reaches a person
- Show the document and the fields side by side, with the failed check highlighted. A reviewer should not have to work out what is wrong.
- Pre-fill everything. Even a failed extraction gets most fields right; making the reviewer retype them wastes the whole benefit.
- Make the correction the source of truth, and store it against the extraction. That pairing is your evaluation set, and after two hundred invoices it tells you exactly which field is worst.
- Never auto-approve payment on extraction alone. Match to a purchase order and a receipt, or route to an approver. Extraction accuracy is a quality problem; payment authorisation is a fraud problem, and a well-made fake invoice extracts perfectly.
The long tail of document types
The first surprise in production is that a meaningful fraction of what arrives in the invoice inbox is not an invoice. Classify before extracting, because each of these needs different handling and running the invoice schema over them produces confident nonsense.
| Document | Description |
|---|---|
| Credit note | An invoice with negative amounts, or positive amounts that must be applied as a credit. The validator passes and the sign is wrong — detect it from the title, not the numbers. |
| Statement | A list of invoices, not an invoice. Extracting it produces a phantom invoice for the total. The tell is many references and no line items. |
| Purchase order | Looks like an invoice, is a request. Paying one is a real and expensive error. |
| Pro forma | Not a demand for payment. Often says so, in small type, once. |
| Multi-invoice PDF | Several invoices in one file. Split on the pages where an invoice number appears in a header position, then extract each. |
| Remittance advice | Confirms a payment already made. Extracting it as an invoice creates a duplicate. |
The classification call is cheap — the first page, a short prompt, one word out — and it protects everything downstream. Give it an unknown option and route those straight to a human; a document type you have not seen before is exactly the case where a forced choice is worst.
Then the duplicate check, which belongs here rather than in the extractor. Key on supplier plus invoice number plus total, and treat a match as a hard stop rather than a warning. Duplicate payment is one of the most common losses in accounts payable, it is entirely mechanical to prevent, and it is the sort of thing an extraction project is expected to have thought about even though nobody asked.
Measuring accuracy honestly
“95 per cent accurate” means nothing without saying per what. Report all three of these, because they differ enormously:
| Metric | Description |
|---|---|
| Field accuracy | Correct fields over total fields. The flattering one — most fields are easy. Report it, do not lead with it. |
| Document accuracy | Invoices where every field is right. Much lower, and the number that predicts human workload. |
| Straight-through rate | Invoices that passed validation and needed no human at all. The number that determines whether the project pays for itself. |
Then split every one of them by supplier. Extraction accuracy is far more variable across document templates than across models, and one supplier with an unusual layout can be the entire error budget — which is fixable with one template-specific hint, and invisible in an aggregate.
Structured extraction in general and confidence for extracted fields cover the cases where no arithmetic exists to check against, which is most of them.
Top comments (0)