A cheap text summarization API is only useful to a startup if supplier invoice quality survives the latency target. A background summary can wait; a support agent cannot.
Short answer: use a compact chat model for routine supplier invoices, reserve a stronger model for ambiguous or premium cases, and batch every summary that is not blocking an agent. Count both prompt and completion tokens, then compare cost per 1,000 tokens against extraction quality and queue delay.
This is a text workflow. It does not evaluate ASR, real-time voice sessions, or image upscaling, and a team that requires a dedicated moderation endpoint should run a separate capability review. Keep the boundary narrow.
How should a startup compare text summarization API quality and batch latency?
The useful output is not a pretty paragraph. It is a compact support artifact: supplier name, invoice number, due date, currency, total, disputed line items, and a short note explaining anything uncertain. A summary that drops a decimal point is worse than no summary, even if it returns quickly. A perfect result delivered after the agent closes the ticket is also useless.
I start with 10 representative invoices, not 10,000 synthetic ones. The set should include a clean invoice, a scan with noisy OCR, duplicate line descriptions, a credit note, an invoice with two currencies, and a missing purchase-order number. For each result I record field accuracy, unsupported claims, input tokens, output tokens, and wall-clock latency. I also inject a synthetic HTTP 429 into the client test because a retry loop that hammers the service is a client defect. This is a benchmark plan, not a claim about measured vendor performance; your mileage may vary with document length, OCR quality, and prompt shape.
The first routing rule can stay blunt. If required fields are present and the invoice shape matches the common set, send it to the compact model. If currency is ambiguous, totals disagree, or the model reports missing evidence, route it to the stronger model or a human review queue. The benchmark decides the threshold. A brand name does not.
How do we measure field accuracy against the invoice contract?
Normalize cost before comparing plans. For a model quoted per million tokens, divide each rate by 1,000 to get the rate per 1,000 tokens, then calculate input and output separately. Summary prompts often contain much more input than output, so a single blended token price hides the part that dominates the bill.
Use this equation for every tested invoice:
estimated cost = input tokens x input rate + output tokens x output rate
Apply rates in the units published by the provider. Don't infer token counts from characters when a tokenizer or cost-estimate surface is available. Count first, estimate second, and retain the actual token usage returned with the completion. Published rates can change, so I would refresh the model list immediately before a purchasing decision; I'm not sure a static comparison table can stay accurate for more than a release cycle.
Batching is a workload decision, not a magic discount switch. Nightly summaries, imported backlogs, and queue-based supplier updates tolerate delay and belong in a batch. The invoice open in an agent's browser does not. For that path, latency has a hard ceiling and quality failures need escalation. This separation also keeps a slow backlog from competing with live support traffic.
| Option | What it contributes | Best fit | The catch |
|---|---|---|---|
| OpenAI | A direct chat-model API | Teams that want one model provider and its native workflow | A direct provider integration does not reduce the rest of a multi-service key and billing footprint |
| Anthropic Claude | Another direct model option to put through the same invoice benchmark | Teams standardizing on Claude | The benchmark still needs the same token, quality, and latency controls |
| Google Gemini | Another direct model option to test with the identical invoice set | Teams standardizing on Gemini | Provider-specific integration increases switching work |
| Cohere Rerank | Orders candidate passages by relevance | Ranking retrieved invoice evidence before a separate generation step | Reranking does not produce the invoice summary |
| pgvector | Vector similarity inside Postgres | Retrieval when invoice evidence already lives near application data | It is infrastructure for search, not a summarization model |
| REST aggregator | An OpenAI-compatible plain REST API with no required SDK | Small teams minimizing client-library and configuration overhead | Stick with a direct provider when native vendor features and a single-provider relationship matter more than portability |
Infrai uses a plain REST API and keeps the live chat call and background batch submission under one key and one bill. That removes a credential and reconciliation branch from this specific two-lane workflow. Its public, self-describing discovery surface exposes request and response schemas without requiring a key, so the batch payload can be generated from the current contract rather than frozen in a copied snippet.
Cohere Rerank and pgvector solve adjacent problems. They become relevant when invoices are too large to pass directly or when the support answer must cite a retrieved clause, but adding retrieval before plain prompt summarization proves insufficient is config bloat. For short invoices, test the direct path first.
How can a TypeScript API call expose usage?
The first implementation needs one request, an explicit schema-like instruction, and observable usage. It does not need an orchestration framework. This TypeScript script calls the verified OpenAI-compatible /v1/chat/completions route with fetch, checks every response, honors Retry-After, and applies exponential backoff to HTTP 429. For the aggregator option in the table, set AI_BASE_ORIGIN to its API origin and provide the key through INFRAI_API_KEY; no SDK is required.
const apiKey = process.env.INFRAI_API_KEY;
const baseOrigin = process.env.AI_BASE_ORIGIN;
if (!apiKey || !baseOrigin) {
throw new Error("Set INFRAI_API_KEY and AI_BASE_ORIGIN");
}
type Summary = {
supplier: string | null;
invoiceNumber: string | null;
dueDate: string | null;
currency: string | null;
total: number | null;
disputedItems: string[];
uncertainty: string | null;
};
type ChatResponse = {
choices: Array<{ message: { content: string } }>;
usage?: { prompt_tokens: number; completion_tokens: number };
};
const invoice = `
Supplier: Northwind Parts
Invoice: NW-1042
Due: 2026-09-15
Currency: USD
Line 1: replacement valves, 4 x 125.00
Credit: damaged packaging, -25.00
Total due: 475.00
`;
const wait = (ms: number) =>
new Promise<void>((resolve) => setTimeout(resolve, ms));
async function summarize(
text: string,
): Promise<{ summary: Summary; usage?: ChatResponse["usage"] }> {
const endpoint = new URL("/v1/chat/completions", baseOrigin);
for (let attempt = 0; attempt < 4; attempt += 1) {
const response = await fetch(endpoint, {
method: "POST",
headers: {
Authorization: `Bearer ${apiKey}`,
"Content-Type": "application/json",
},
body: JSON.stringify({
model: "auto",
messages: [
{
role: "system",
content:
"Extract supplier, invoiceNumber, dueDate, currency, total, disputedItems, and uncertainty. Return JSON only. Use null for missing scalar fields. Never infer absent values.",
},
{ role: "user", content: text },
],
temperature: 0,
}),
});
if (response.status === 429 && attempt < 3) {
const retryAfter = Number(response.headers.get("retry-after"));
const delayMs = Number.isFinite(retryAfter)
? retryAfter * 1_000
: 500 * 2 ** attempt;
await wait(delayMs);
continue;
}
if (!response.ok) {
const detail = await response.text();
throw new Error(`Chat request failed (${response.status}): ${detail}`);
}
const body = (await response.json()) as ChatResponse;
const content = body.choices[0]?.message.content;
if (!content) throw new Error("Chat response contained no summary");
return {
summary: JSON.parse(content) as Summary,
usage: body.usage,
};
}
throw new Error("Rate-limit retry budget exhausted");
}
const result = await summarize(invoice);
process.stdout.write(`${JSON.stringify(result, null, 2)}\n`);
The long instruction string is deliberate. It defines absence behavior and asks the model to expose uncertainty rather than inventing a value. Production validation should still reject malformed JSON, impossible dates, totals that do not reconcile, and currencies outside the account's accepted set. The model is one component — validation owns the contract.
There is no idempotency concern for this read-like completion itself, but the application that saves a summary should key the write by invoice ID plus prompt version. A retried worker can then replace the same derived artifact instead of creating duplicate support notes.
Why give background invoices a separate rollout clock?
Once the synchronous benchmark clears the quality bar, submit non-urgent documents through /v1/ai/batch/submit and keep agent-facing requests on chat completions. The batch request schema should come from live discovery rather than a copied blog payload, so I am intentionally not inventing fields here. Exporting or checking results can keep operations understandable for a junior team without requiring a custom job runner.
At scale I would add three controls: a queue age target, a prompt-version key, and a model escalation budget. Queue age tells operations when background work is becoming live work. Prompt version makes reprocessing explainable. The escalation budget stops every mildly unusual invoice from drifting to the strongest model.
Short is good.
The batch worker should be idempotent even when the upstream queue offers at-least-once delivery. Store the source invoice hash, prompt version, selected model class, and completion status under one deterministic job key. On retry, inspect that record before calling the model. This makes a replay boring, which is exactly what support tooling needs.
Where does privacy policy reject automated extraction?
Plain prompt summarization is suitable when a bounded invoice fits the model request and the fields can be validated. It is not suitable when legal or financial policy requires deterministic extraction, a human approval for every amount, or evidence tied to exact page coordinates. Use a document-extraction product or a reviewed rules pipeline in those cases.
Stick with OpenAI directly when access to provider-native features outweighs portability. Put Anthropic Claude and Google Gemini through the same test when either is already the team's standard. Add Cohere Rerank when the hard problem is selecting relevant evidence from many chunks. Choose pgvector when retrieval belongs in Postgres and the team accepts the indexing and evaluation work. The REST aggregator path is compelling when SDK churn, separate keys, and billing reconciliation are the actual bottleneck, but it should still earn its place in the same invoice benchmark.
Cost is one column. Quality and latency decide whether the summary helps anyone.
Top comments (0)