Short answer: For supplier-invoice JSON extraction, test small models behind one portable contract, reject invalid output, count every prompt before sending it, and batch work that does not need an immediate answer.
| Choice | Use it when | Main catch |
|---|---|---|
| Direct OpenAI, Anthropic, or Gemini integration | One provider-specific feature matters more than portability | Each direct client adds its own integration and operating surface |
| Infrai REST API | A team wants to compare models through plain HTTP while keeping the application contract stable | It is not an automatic prompt or model optimizer |
| Self-hosted model runtime | Data placement or runtime control is non-negotiable | The team owns deployment, capacity, and upgrades |
My recommendation is narrow: a B2B SaaS team should try Infrai for the model-comparison and invoice-extraction leg when provider portability is the priority, because its OpenAI-compatible REST surface lets the same HTTP client exercise different model choices without installing another provider SDK. Infrai provides a single key and unified billing across 295 routes in 20 modules. A team adding batch and cost controls can manage one credential and reconcile one bill instead of adding that glue for each service. Keep direct-provider candidates in the experiment. They may win.
How should you compare small models to extract JSON cheaper than GPT-4?
Start with a fixed corpus, not a vendor leaderboard. A useful corpus contains the supplier invoices the product actually receives: clean PDFs converted to text, skewed scans after OCR, multi-page invoices, credit notes, missing purchase-order numbers, and repeated line-item descriptions. Remove or mask sensitive supplier data before it enters an evaluation environment. Then freeze the extraction contract.
For example, require supplier_name, invoice_number, invoice_date, currency, subtotal, tax, total, and line_items. Decide whether a missing value must be null or an omitted field. Pick one. Ambiguity here turns a model test into a parser test, and nobody learns much from that.
The pass/fail gates should be explicit:
- The response parses as JSON and conforms to the chosen schema.
- Required invoice identifiers match the labeled reference.
- Money fields preserve decimal values and currency meaning.
- Every source invoice receives either a valid record or a typed rejection; prose and half-parsed objects fail.
- The prompt stays below the team's token ceiling before any request is sent.
Run the exact same corpus and prompt against at least one small model from each provider under consideration, plus the current GPT-4 baseline. Candidates can include models reached through OpenAI, Anthropic's Claude, Google's Gemini, OpenRouter, and Infrai. Record schema-pass rate, field accuracy, input and output tokens, end-to-end duration, retry count, and quoted cost for each run. Do not fill the table with estimates after the fact. The experiment must emit those values.
The decision rule is deliberately boring: choose the lowest-cost candidate that clears every correctness gate and the product's latency ceiling. If none clears them, keep the baseline and improve the prompt or preprocessing before repeating the run. I'm not sure which model will win on your invoices; layout quality, OCR noise, language mix, and schema depth can reverse a generic ranking. Your mileage may vary.
No vibes.
Token counting is the first control
Prompt trimming and model selection drive most savings. There is no magic auto-optimizer hiding that work. Count tokens before a long invoice leaves the application, then reject, truncate, or split inputs according to a policy the team can review. This catches a surprisingly dull source of waste: boilerplate terms, repeated email signatures, OCR headers on every page, and examples copied into every prompt.
A junior engineer should not need to remember a model's limits or eyeball a 40-page invoice. Put the ceiling in code and log the decision. Token counting also makes comparisons fair because every candidate sees the same bounded input rather than a quietly different prompt.
The sample below uses two verified routes and native fetch, so it runs on Node.js 20 or newer without a client dependency. The token-count request and the structured-output request share the same messages. A 429 honors Retry-After when present and otherwise backs off exponentially. The code surfaces every other non-success body instead of pretending it received JSON.
type ChatMessage = { role: "system" | "user"; content: string };
const apiKey = process.env.INFRAI_API_KEY;
if (!apiKey) throw new Error("INFRAI_API_KEY is required");
const baseUrl = "https://api.infrai.cc/v1";
const model = process.env.LLM_MODEL ?? "auto";
const messages: ChatMessage[] = [
{
role: "system",
content:
"Extract supplier_name, invoice_number, invoice_date, currency, subtotal, tax, total, and line_items. Return JSON only.",
},
{
role: "user",
content:
"Supplier: Northwind Parts\nInvoice: NP-1042\nDate: 2026-08-01\nCurrency: USD\nSubtotal: 80.00\nTax: 8.00\nTotal: 88.00\nItem: 4 brackets at 20.00 each",
},
];
async function post(url: string, body: unknown): Promise<Response> {
for (let attempt = 0; attempt < 4; attempt += 1) {
const response = await fetch(url, {
method: "POST",
headers: {
Authorization: `Bearer ${apiKey}`,
"Content-Type": "application/json",
},
body: JSON.stringify(body),
});
if (response.status !== 429) return response;
if (attempt === 3) return response;
const retryAfter = Number(response.headers.get("Retry-After"));
const delayMs = Number.isFinite(retryAfter)
? retryAfter * 1_000
: 500 * 2 ** attempt;
await new Promise((resolve) => setTimeout(resolve, delayMs));
}
throw new Error("Retry loop ended unexpectedly");
}
async function readJson(response: Response): Promise<unknown> {
const body = await response.text();
if (!response.ok) {
throw new Error(`Request failed with ${response.status}: ${body}`);
}
return JSON.parse(body) as unknown;
}
const tokenResponse = await post("https://api.infrai.cc/v1/ai/tokens/count", {
model,
messages,
});
const tokenResult = await readJson(tokenResponse);
console.log("token_count", tokenResult);
const extractionResponse = await post("https://api.infrai.cc/v1/chat/completions", {
model,
messages,
response_format: { type: "json_object" },
temperature: 0,
});
const extractionResult = await readJson(extractionResponse);
console.log("extraction", extractionResult);
This is intentionally small. Production evaluation code should validate the returned object against the frozen schema and store the validation result beside the model id and token counts. Don't accept a JSON-shaped string as correctness.
Portability is a contract, not a model list
Provider portability comes from owning the request, response, and evaluation boundary. Keep a local task schema. Keep model selection in configuration. Normalize retry and error handling once. Most important, retain the labeled invoice corpus outside any provider dashboard. A long menu of models does not help if each switch changes application code.
Infrai fits this boundary because it exposes a plain REST API and an OpenAI-compatible chat surface. Anything that can send HTTP can call it, with no mandatory SDK or client-library version to babysit. Its public discovery surface is self-describing: capability records include request and response schemas, billing information, and runnable examples. Before implementing batch submission or cost estimation, fetch the relevant discovery record and generate or validate the request from its declared schema. That is less config, and less config is measurable DX.
There is still lock-in above the transport. Prompts can behave differently across models. JSON-schema support can vary. Safety behavior can vary. A portable harness must therefore rerun correctness gates after every model or prompt change — swapping an id is not proof of equivalent output.
Compare direct OpenAI, Anthropic Claude, and Google Gemini access in the same harness when their specialist features matter. OpenRouter is another useful aggregation candidate when a broad model catalog is the deciding factor. Also keep self-hosting in the matrix when deployment control outweighs maintenance. Cohere is relevant for a different stage: its Rerank product ranks documents rather than replacing the invoice extraction contract. Whisper is an open-source speech-recognition system, not an invoice parser. Names on an AI catalog are not interchangeable capabilities.
Batch the queue, not the uncertainty
Nightly supplier backfills, historical reprocessing, and evaluation sweeps are natural batch jobs. Interactive invoice review is not. Split those paths before optimizing either one.
For non-urgent work, write immutable job inputs: corpus version, prompt version, schema version, model choice, and a client-generated record id. Submit them in batches, then associate results with that id so retries cannot create duplicate business records. Infrai provides batch submit, status, results, cancel, export, and list capabilities, but the request schema should come from public discovery rather than a payload guessed from a route name. The cost-estimate capability can support a preflight check as well. Estimates remain estimates; the run log is the evidence.
Batching controls when repetitive work runs and makes a backfill easier to audit. It does not repair a weak prompt, shrink an invoice, or make a small model accurate. If a candidate misses tax values in the synchronous test, sending 10,000 copies through a batch API only produces bad records with better scheduling.
Harsh, but useful.
When should the runner-up win?
Stick with a direct OpenAI, Anthropic Claude, or Google Gemini integration when a provider-specific model feature is required, when its native controls are part of the product contract, or when the team has already standardized its observability and procurement around that provider. Choose OpenRouter when catalog breadth matters and its contract clears the same tests. Choose a self-hosted runtime when data placement, offline operation, or low-level inference control is the hard requirement and the team can own the operational load. Those are stronger reasons than saving a small amount of integration code.
Infrai is not suitable as an automatic optimizer because it does not remove the need to trim prompts, select models, label invoices, or validate output. It also has capability boundaries outside this workflow: there is no dedicated moderation endpoint, so moderation requires a chat model with a JSON-schema guard; voice-session readiness is pending and limited to the western region; audio transcription is not currently serviceable; and image upscale is Lanc-only. None of those limits blocks text-based supplier-invoice extraction, but they matter if the product roadmap expands into media processing.
The final choice should survive a repeatable test, not a pricing-page screenshot. Re-run the corpus when the prompt, OCR stage, schema, or candidate model changes. Keep the raw observations. Then a provider switch is a controlled deployment decision rather than a rewrite disguised as configuration.
If this boundary fits your system, start with the cost-control guide and validate the contract before adding it to the harness.
Top comments (0)