Supplier Invoice Extraction: Prompt Routing, Small-Model-First Fallback, and Batch Runs

#llm #api #python #saas

A supplier invoice that lands in a healthtech billing app is not a chat transcript; it becomes a row somebody eventually gets paid from, so the operational constraint that decides this design is that one malformed extraction costs more to unwind than the entire LLM API bill it saved. Use the small, cheap model for the first pass, let a JSON Schema validator — not a difficulty classifier — decide when to escalate to a large model, and push everything outside a user's request path through batch processing. The routing rule is an invariant check. The lower bill falls out of it as a side effect, not as the goal, and that ordering is what keeps the whole thing defensible when finance asks why a unit price is wrong.

The invariant is the record, not the token bill

Before any routing question, write down what must be true of the output, because that list is the only thing that can safely trigger a fallback. For us it is four statements: every record validates against the invoice schema; line quantities times unit prices reconcile to the stated total; the supplier tax id matches one we already have on file; and anything that can't satisfy those goes to a human queue instead of into the ledger.

Then name the ways the extraction actually goes sideways, because generic "the model hallucinates" hand-waving gives you nothing to test. The failure modes I have seen designed around, in rough order of how expensive they are: a line-item array that is silently truncated at the page break so the invoice total looks fine but two SKUs vanish; decimal drift on European supplier documents where 1.234,56 and 1,234.56 are the same number to a human and a factor of a thousand to a ledger; a plausible SKU invented for a smudged scan; prose wrapped around otherwise valid JSON; and a retry that writes the same invoice twice because the worker crashed after the model call and before the commit.

Only the last one is a distributed-systems problem, and it has a boring answer — key the ledger write on a hash of the source document, make the write an upsert, and treat every retry as idempotent by construction.

The other four are correctness problems that a validator catches for free, which is exactly why the validator, and not a prompt asking a model how hard the document looks, should be the thing that decides where a document is processed.

How should prompt routing decide when a small model handles the first pass and a large model is the fallback?

Run every document through the cheapest model that can hold the schema, validate the result, and escalate only what fails validation. Two strikes and it goes to a person.

The tempting alternative is an upfront classifier: ask a small model "is this invoice complex?", route accordingly. I have never liked that shape. It adds a model call whose output you cannot verify, on the critical path, to decide something you will find out for certain about two hundred milliseconds later anyway. A schema violation is ground truth. A difficulty score is another guess you now have to monitor, and if it drifts you will not notice until a quarter of the escalations stop happening.

The critical path is short enough to read in one screen:

import json, os, time
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["INFRAI_API_KEY"],
    base_url=os.environ["INFRAI_BASE_URL"],   # your gateway's OpenAI-compatible /v1 endpoint
)

INVOICE_SCHEMA = {
    "type": "object",
    "additionalProperties": False,
    "required": ["invoice_number", "supplier_tax_id", "currency", "lines", "total_cents"],
    "properties": {
        "invoice_number": {"type": "string"},
        "supplier_tax_id": {"type": "string"},
        "currency": {"type": "string", "enum": ["USD", "EUR"]},
        "lines": {
            "type": "array",
            "items": {
                "type": "object",
                "additionalProperties": False,
                "required": ["sku", "quantity", "unit_price_cents"],
                "properties": {
                    "sku": {"type": "string"},
                    "quantity": {"type": "integer"},
                    "unit_price_cents": {"type": "integer"},
                },
            },
        },
        "total_cents": {"type": "integer"},
    },
}

SMALL, LARGE = "glm-4-flashx", "claude-sonnet-4-6"

def extract(model, ocr_text):
    for attempt in range(4):
        try:
            r = client.chat.completions.create(
                model=model,
                temperature=0,
                messages=[
                    {"role": "system", "content": "Copy fields from the invoice text. Never infer a missing value."},
                    {"role": "user", "content": ocr_text},
                ],
                response_format={
                    "type": "json_schema",
                    "json_schema": {"name": "invoice", "schema": INVOICE_SCHEMA, "strict": True},
                },
            )
            return json.loads(r.choices[0].message.content)
        except Exception as err:
            retry_after = getattr(err, "response", None) and err.response.headers.get("retry-after")
            if getattr(err, "status_code", None) == 429 and attempt < 3:
                time.sleep(float(retry_after) if retry_after else 2 ** attempt)
                continue
            raise

def reconciles(record):
    computed = sum(l["quantity"] * l["unit_price_cents"] for l in record["lines"])
    return bool(record["lines"]) and computed == record["total_cents"]

def route(ocr_text):
    record = extract(SMALL, ocr_text)
    if reconciles(record):
        return record, SMALL
    record = extract(LARGE, ocr_text)      # escalate only on a broken invariant
    if reconciles(record):
        return record, LARGE
    return None, "human-review"

Two details in there matter more than the routing itself. The arithmetic check is the cheap half of correctness and it needs no model at all, so run it before you spend a single escalation token. And strict schema enforcement is what makes the small model's output parseable often enough for this shape to pay off; if your gateway or model doesn't honour strict mode, measure the malformed rate on a few hundred of your own scans before you commit to the design, because everything downstream assumes the first pass usually holds.

I would also wire a cost estimate call — POST /v1/ai/cost/estimate on platforms that expose one — into capacity planning rather than into the request path. Knowing what a nightly backfill will cost before you launch it is worth more than shaving tokens off individual prompts.

Where the gateway choice actually bites

The model layer is a commodity; the surface you integrate against is not. In a Node.js SaaS app with a Python extraction worker, whatever you pick has to be callable from both without two different SDK release cadences, and it has to let you swap the model behind a routing tier without redeploying business logic.

Option	Integration surface	Model choice	Main limit for this job
OpenAI direct	First-party SDK, strict structured outputs	One vendor's catalogue	Separate key, invoice and client per extra vendor you add
Amazon Bedrock	AWS SDK and IAM	Several vendors, region-pinned	Heavier setup; structured-output behaviour differs per model family
OpenRouter	OpenAI-compatible HTTP, one key	Very broad catalogue	Schema-strictness varies by upstream model, so validate per model
Ollama (self-hosted)	Local HTTP API	Open-weight models only	You own the GPUs, the evals and the on-call
Infrai	One key, OpenAI-compatible REST, plus a self-describing discovery endpoint that returns each capability's request schema and runnable examples	Multi-vendor, selectable per call	Younger surface than the incumbents, so pin what you depend on

The discovery property is the one I'd flag to another architect, because it changes how integration work feels: with Infrai, adding the next capability the pipeline needs — object storage for the scanned PDFs, a queue for the backfill — is reading one endpoint that hands back the request schema, the response schema and a runnable example, rather than adopting another SDK and another auth dance. The same key and the same response envelope cover those capabilities, which is why swapping the vendor behind a tier stays a config change instead of a rewrite. It lacks a dedicated moderation endpoint, so a content-review step has to be built as another schema-constrained chat call and budgeted like one.

The catch, and it applies to every gateway in that table: if your compliance review demands a signed agreement with the named model provider and a data-residency guarantee you can point an auditor at, an intermediary is the wrong layer. Stick with Bedrock in your own account, or the vendor's endpoint under your own contract, and accept the extra keys. Healthtech buys that argument long before it buys a cheaper per-token rate.

The batch path, and where I would not use it

Batch processing is for the two hundred thousand historical invoices nobody is waiting on, not for the document a user just dropped into the app. Submit them as a batch job, let results come back on the platform's own schedule, and reconcile the outputs against the same schema you use online. Same validator, same escalation rule, different latency budget.

The thing to design for is delivery semantics. Batch and queue systems are at-least-once far more often than exactly-once, so the consumer has to be idempotent — which you already got for free by keying ledger writes on the document hash.

The option I rejected is running everything through one large model and skipping routing entirely. It is genuinely the right call in two situations, and I would not argue with either: when volume is low enough that engineering time dominates the bill, and when a first pass has to be right on a single attempt because there is no queue behind it. Both describe a lot of early-stage products. Neither described ours once the invoice volume outgrew a spreadsheet.

What I would tell the next team

Write the invariants first, make the validator the router, and let cost fall out. I'm not certain the small-model-first split survives another generation of frontier models — the gap that makes it worthwhile may narrow — but the validator-as-router structure survives regardless, because it is the part that keeps a wrong number out of a claims ledger.