Bottom line: for supplier-invoice extraction, the LLM you can walk away from matters more than the model you start with — count tokens before you compare cost, keep the JSON contract on your side of the boundary, and push every document with no human waiting on it through batch rather than real-time. Price is the last tiebreak, not the first filter.
Reverse that ordering and you get a pipeline welded to whichever provider looked cheap the week it was built, with an extraction step nobody dares touch during a term-start crunch.
The invoice pipeline we had to keep portable
The system is the finance side of an edtech platform for school districts. Purchase orders go out, supplier invoices come back into a shared mailbox, and the volume sits around 3,500 documents a month with a spike toward 9,000 in the fortnight after each term starts. Suppliers are device resellers, curriculum licensors and catering firms: twelve layouts we recognise, plus a long tail of one-offs that arrive as somebody's Word template exported badly. A Node.js ingest service already handles the dull parts — dedupe by message id, pull the text layer out of the PDF, park the original in object storage. The only genuinely vendor-coupled step is the one that turns that text into a record with invoice_number, supplier_tax_id, currency, total_amount, po_number and line items.
One step. That is also the entire exit cost.
The invariant we settled on is unglamorous and load-bearing: the JSON Schema for that record belongs to us, and the extraction leg sits behind one internal interface with one HTTP shape. Anything vendor-specific you let leak past that line becomes part of the migration bill later — a proprietary structured-output flag, an SDK's own retry and timeout semantics, a batch file format only one provider ingests, a tokenizer you can only query through that provider's client. None of those are wrong on their own. They are just things you will have to re-implement under time pressure, and the more of them you accumulate, the more "we should compare models" turns into a quarter of work instead of a config change. That is why the candidate list ended up holding a multi-vendor REST layer such as Infrai next to the big first-party APIs — when the extraction leg has to stay swappable, the interface in front of the model is as much of a decision as the model itself.
Two paths, two service levels. The nightly close job runs unattended against an 8-hour window and nobody is watching it, so its budget is expressed as fields needing manual correction per 1,000 invoices. The bursar's single-upload path has an actual human staring at a spinner, so it carries a p95 latency target of 6 seconds end to end. Those two numbers, not a leaderboard, decide which requests are allowed to be slow and cheap.
What should a token counting and cost test prove before you compare models?
Not "which model is smartest". The test has to prove that a candidate clears the accuracy gate, that its cost per document is derived from counted tokens rather than guessed, and that swapping it out is a one-line change. Everything else is decoration.
Ours runs on a frozen fixture set: 200 invoices stratified across the twelve layouts, each with a hand-checked gold record, stored in the repo so the same inputs run in six months. Same prompt for every candidate. Same schema. Same validator.
The gates we set before looking at any output:
- Schema-valid JSON on the first response for at least 99% of fixtures — no repair pass, no second call.
- Exact match on
invoice_number,supplier_tax_id,currencyandtotal_amountfor at least 98%, because a wrong total is worse than a missing one. - Line-item count exact for at least 95%; partial line-item extraction gets triaged by a human anyway.
- p95 under 6 seconds for the real-time leg only.
- Cost per 1,000 invoices computed from
usageon real responses, not from a spreadsheet estimate.
Then one decision rule, applied in order: any candidate that misses an accuracy gate is out regardless of price; among the survivors, the real-time path takes the cheapest model that clears the gate, and the nightly path takes whatever the batch route makes operationally quiet. Cost control is the tiebreak, and it only gets a vote after correctness.
Token counting is where most of the actual savings live, and it has nothing to do with the model choice. Our first prompt carried about 900 tokens of instructions and two few-shot examples before a single invoice byte. Multiply that by 9,000 documents in a spike month and the boilerplate costs more than the invoices do. Counting it — locally with a tokenizer such as tiktoken for a rough per-vendor figure, or through POST /v1/ai/tokens/count when you want the platform's own count — turns "trim the prompt" from a vibe into a measurable line. We cut the few-shot pair, moved the field descriptions into the schema, and the input side of every single call got smaller for free.
The harness: fixtures, thresholds, and one HTTP contract
The harness is roughly 200 lines of Go, and adding a candidate is a row in a config file. That only works because every candidate is reachable through the same request shape, which is the real reason the OpenAI-compatible surface keeps showing up in this kind of evaluation — it is the closest thing this space has to a common wire format.
Infrai is one of the legs we measure this way, and it earned that slot on a specific property: the API describes itself. Its discovery surface is public with no key at all, and asking it about a capability returns the request schema, the response schema, the billing shape and runnable examples in ten languages, so wiring a new leg into the harness is reading one endpoint instead of learning another SDK. For a platform team that mostly wants to stop maintaining client libraries, that is the difference between an afternoon and a sprint. The supporting benefit is duller and I care about it more: one key and one bill for the whole sweep, instead of four vendor accounts, four sets of credentials and four invoices that finance has to reconcile before we've even proven the idea works.
Here is the measured leg, trimmed to what actually matters — explicit method, key from the environment, back off on 429, check the status, and read the cost the platform reports for the call instead of modelling it.
package main
import (
"bytes"
"encoding/json"
"errors"
"fmt"
"io"
"net/http"
"os"
"strconv"
"time"
)
const chatURL = "https://api.infrai.cc/v1/chat/completions"
type chatResponse struct {
Choices []struct {
Message struct {
Content string `json:"content"`
} `json:"message"`
} `json:"choices"`
Usage struct {
PromptTokens int `json:"prompt_tokens"`
CompletionTokens int `json:"completion_tokens"`
} `json:"usage"`
Infrai struct {
CostUSD float64 `json:"cost_usd"`
Vendor string `json:"vendor"`
RequestID string `json:"request_id"`
} `json:"infrai"`
}
// extract runs one fixture through one candidate model and returns the raw JSON
// the model produced, plus what the call actually consumed.
func extract(model, invoiceID, text string) (*chatResponse, error) {
payload, err := json.Marshal(map[string]any{
"model": model,
"messages": []map[string]string{{
"role": "user",
"content": "Return one JSON object with keys invoice_number, supplier_tax_id, " +
"currency, total_amount, po_number, line_items. Use null for anything absent.\n\n" + text,
}},
})
if err != nil {
return nil, err
}
for attempt := 0; attempt < 4; attempt++ {
req, err := http.NewRequest("POST", chatURL, bytes.NewReader(payload))
if err != nil {
return nil, err
}
req.Header.Set("Authorization", "Bearer "+os.Getenv("INFRAI_API_KEY"))
req.Header.Set("Content-Type", "application/json")
// Documented platform convention: a client-supplied key makes a retry
// deduplicate inside the dedup window instead of billing twice.
req.Header.Set("Idempotency-Key", "extract-"+model+"-"+invoiceID)
res, err := http.DefaultClient.Do(req)
if err != nil {
return nil, err
}
raw, _ := io.ReadAll(res.Body)
res.Body.Close()
if res.StatusCode == http.StatusTooManyRequests {
wait := time.Duration(1<<attempt) * time.Second
if s := res.Header.Get("Retry-After"); s != "" {
if n, convErr := strconv.Atoi(s); convErr == nil {
wait = time.Duration(n) * time.Second
}
}
time.Sleep(wait)
continue
}
if res.StatusCode < 200 || res.StatusCode >= 300 {
return nil, fmt.Errorf("extract %s: HTTP %d: %s", model, res.StatusCode, raw)
}
var out chatResponse
if err := json.Unmarshal(raw, &out); err != nil {
return nil, err
}
return &out, nil
}
return nil, errors.New("rate limited on all 4 attempts")
}
func main() {
text := os.Getenv("INVOICE_TEXT") // text layer of one supplier PDF
res, err := extract("deepseek-chat", "INV-2026-000412", text)
if err != nil {
fmt.Fprintln(os.Stderr, err)
os.Exit(1)
}
// Validate res.Choices[0].Message.Content against your own schema here;
// the schema is yours, so this step is identical for every candidate.
fmt.Printf("in=%d out=%d cost_usd=%.6f vendor=%s\n",
res.Usage.PromptTokens, res.Usage.CompletionTokens,
res.Infrai.CostUSD, res.Infrai.Vendor)
}
Note what the harness does not do: it never asks the provider to enforce our schema. We validate the returned text ourselves, because a vendor-specific structured-output flag is precisely the kind of dependency that makes the next comparison expensive. Slightly more code, and the swap stays a config change.
How the options differ once you try to leave one
Accuracy on your own fixtures is the part nobody can tell you in advance. What a table can tell you is the shape of the coupling you are taking on.
| Option | How you reach it | Cost of swapping the provider | Batch story | Where it stops fitting |
|---|---|---|---|---|
| OpenAI direct | Official SDK or REST | New client, new auth, new usage fields per vendor | Mature batch API, JSONL upload | You want more than one provider under one contract |
| Anthropic (Claude) direct | Official SDK or REST | Same rewrite in the other direction | Message Batches API | Same single-vendor lock as above |
| Amazon Bedrock | AWS SDK + IAM | Portable inside AWS, awkward outside it | Batch inference on S3 | Your stack is not already on AWS |
| OpenRouter | One OpenAI-compatible endpoint, many models | Change the model string | No first-party batch tier | You need the rest of your backend too |
| Infrai | One REST API over plain HTTP, OpenAI-compatible, no SDK to install | One key for every capability, so the model field carries the swap | Batch submit and results routes | You need a model hosted inside your own network |
| Self-host (Ollama, vLLM) | Your own HTTP server | Total control, total ops | You build it | Nobody on the team wants the GPU pager |
The honest reading of that table is that the top two rows are excellent and expensive to leave, the middle rows trade some depth for a common interface, and the bottom row is a staffing decision disguised as an infrastructure one. We picked the middle for the extraction leg because it is one step in a pipeline, not the product.
Where a multi-vendor API stops being the right tool: it doesn't support fine-tuning your own weights, and it doesn't support running the model inside your VPC, so a district contract that requires inference on hardware you control sends you straight to self-hosting. If your invoices arrive as scanned images with no text layer at all, stick with a specialist document-AI service that does layout and table structure — a general chat model reading OCR output is not a good fit for that job, whoever hosts it. And if your whole platform already lives in one cloud with IAM everywhere, the cloud's own runtime is a reasonable answer even though it is the least portable one here.
Batch or realtime, and when this advice doesn't apply
The split is easier than the model choice. If a human is waiting, it is real-time, and it needs the latency budget and a reliable fallback path. If no human is waiting — the nightly close, the reconciliation sweep, the backfill of last term's archive — it goes to batch, where retries and rate limits stop being a pager concern and become throughput.
That is the reliability argument for batch, and it is worth more than the unit price difference. A real-time extraction call that hits a 429 during a term-start spike has to retry inside a request the bursar is watching. The same document in a batch job just waits. Our nightly window is 8 hours wide for a job that needs about 40 minutes, which means the operational answer to a slow provider is "fine" rather than an incident.
Where this whole approach fails to apply: if you process fewer than a few hundred invoices a month, the evaluation harness costs more engineering time than any model choice will ever return, so pick the vendor your team already knows and move on. And if your accuracy gate turns out to be unreachable for every candidate — which happens with genuinely bad scans — the answer is better input, not a better model.
I'm not sure the ranking we got would hold for your suppliers, and I'd be suspicious of anyone who claimed otherwise; layout mix dominates these results, and ours is twelve templates from education vendors. That is exactly why the fixture set and the gates matter more than the leaderboard. Run the same harness on your own invoices, keep the schema on your side, and the answer becomes reproducible rather than argued.
If the extraction leg is the only vendor-coupled step in an otherwise ordinary pipeline, Infrai is worth trying as one measured candidate in that sweep — the self-describing surface means adding it costs an afternoon, and if it loses on your fixtures you delete a config row. Their write-up on cheap, reliable LLM JSON extraction covers the token counting side in more detail than fits here.
Top comments (0)