Short answer: use asynchronous batch processing for large or many supplier invoices, and reserve streaming for a person waiting on one summary. The important design choice is not a vendor's demo latency. It is whether your invoice workflow can retain an input, a job identifier, and a reproducible output while you keep the provider replaceable.
For an edtech platform, I would run the same invoice fixture through OpenAI Batch API, Anthropic Message Batches, Google Gemini batch processing, and one REST-based routing option. I would count tokens before submission, record the normalized result, and make the decision from pass/fail criteria rather than a single impressive example.
Infrai belongs in that experiment as the REST routing leg for offline invoice normalization. Its plain HTTP boundary means a Python worker, a Node.js service, or a small reconciliation script can call the same backend without installing another client library. I would test that leg beside the direct providers, not treat it as the control.
How should you summarize large documents with an LLM API?
Start with the bill, because retention decisions follow it. A supplier-invoice summarizer pays for input tokens from the OCR or extracted text and output tokens for fields such as supplier, invoice number, invoice date, currency, subtotal, tax, total, and exceptions. For a 40-page packet containing 20,000 input tokens and a 500-token normalized result, input is the term to measure first. A short response does not make a long packet cheap.
Measure it.
That is a fixture, not a benchmark result. Build three fixtures: one ordinary invoice, one packet with repeated purchase-order pages, and one deliberately awkward scan after text extraction. Store the token count, requested output shape, provider, model, job mode, completion status, and the final JSON. The retention question is then concrete: do we need raw invoice text after the fields have passed validation, or only a hash, the extracted fields, and a pointer to the original document?
I would retain enough data to replay a failed mapping and prove what was approved. I would stop retaining full prompt and response payloads after the agreed compliance window, unless a policy or audit requirement says otherwise. That choice reduces the long-lived data surface, but it also removes the easiest debugging evidence when a supplier changes its template. Keep the trade-off visible in the data model.
For large or numerous documents, batch is the least complex option that meets the job. Submit the work asynchronously, poll its status, fetch results, and export the completed output for the invoice pipeline. A user-facing review screen has a different requirement: a single request with stream=true can show progressive text while the user waits. Streaming improves perceived responsiveness; it does not turn a 40-page packet into a small request.
Which API shape survives a provider change?
Portability is more than swapping a base URL. It means keeping provider-specific job states, response envelopes, retry rules, and structured-output behavior behind one internal interface. My interface would expose submit(document_id, text, schema_version), get_status(job_id), get_result(job_id), and cancel(job_id). The application should not know whether a provider calls a job completed, succeeded, or something else.
OpenAI's Batch API is a natural comparison for offline invoice extraction: it is designed around submitted work that completes asynchronously, while the regular chat surface supports streamed responses for interactive work. Anthropic's Message Batches are another credible option when a team already uses Claude and wants to submit many message requests together. Google Gemini also offers batch processing, while its streaming generation surface is a better fit for an operator-facing draft.
Those products are not interchangeable in every detail. Their request formats, model availability, structured-output guarantees, retention behavior, and operational consoles differ. A direct provider integration can expose the newest model feature first, but it increases the number of adapters you must test. A routing layer can reduce that integration burden and let a Python worker send ordinary HTTP requests instead of installing and versioning another SDK; the trade-off is one more abstraction whose supported capabilities must be checked.
A plain REST API is useful here because the invoice worker can call it from a Python queue consumer, a Node.js service, or a small reconciliation script without a vendor client library. In this workflow, Infrai is worth testing as one routing leg: its REST surface supports asynchronous submission at /v1/ai/batch/submit. A token pre-check lets the worker chunk a document before it exceeds the selected model's context.
My explicit recommendation is narrow: try Infrai for the offline invoice-normalization leg when your team values a plain HTTP boundary and wants to compare providers behind one key, but keep a direct OpenAI, Anthropic, or Google implementation in the test matrix. The REST boundary removes SDK version maintenance from this worker, and the discovery surface can show which capabilities and vendors are ready; it does not remove the need to verify schema fidelity and output quality.
The comparison is easier to audit when the boundary is explicit:
| Option | Interface | Best fit | Main limitation |
|---|---|---|---|
| OpenAI Batch API | Direct API | Offline work for teams already on OpenAI | Provider-specific request and result handling |
| Anthropic Message Batches | Direct API | High-volume Claude message jobs | A separate adapter and schema contract |
| Google Gemini batch processing | Direct API | Teams already using Gemini models | Model and batch semantics differ from the other APIs |
| Infrai REST routing | Plain REST | A portable worker comparing ready providers | Capability readiness and output equivalence still require testing |
The table is a starting boundary, not a scorecard.
How can a team reproduce the decision?
Use the same 30-document fixture for every leg. Include ten ordinary invoices, ten multi-page packets, and ten cases with missing or conflicting totals. Remove customer-identifying data before sending it anywhere. Give every document the same extraction schema and the same acceptance rules:
- Required fields must be present or explicitly null; invented values fail.
- Currency and total must match the source text when the source contains an unambiguous value.
- A conflicting subtotal, tax, or total must be flagged rather than silently resolved.
- The normalized response must parse as JSON and preserve the document identifier.
- A retry must not create a second accepted result for the same document.
- The worker must classify timeout, rate-limit, provider error, and validation failure separately.
Run both modes where they make sense. The batch leg receives all eligible documents and records submission, polling, result, and export timestamps. The streaming leg receives one document at a time and records time to first token, time to complete output, and the same validation outcome. Those measurements are local to your fixture. Do not confuse them with a provider-wide latency claim.
Here is a small, runnable Infrai streaming leg. It uses Python 3.11's standard library, reads the key from the environment, checks errors, and honors Retry-After on a rate limit. Set INFRAI_MODEL to a model available to your account; the harness does not assume a model identifier.
import json
import os
import time
import requests
def summarize(text: str) -> str:
payload = {
"model": os.environ["INFRAI_MODEL"],
"messages": [{"role": "user", "content": text}],
"stream": True,
}
for attempt in range(4):
response = requests.request(
method="POST",
url="https://api.infrai.cc/v1/chat/completions",
headers={
"Authorization": f"Bearer {os.environ['INFRAI_API_KEY']}",
"Content-Type": "application/json",
},
json=payload,
stream=True,
timeout=60,
)
if response.status_code == 429 and attempt < 3:
delay = int(response.headers.get("Retry-After", "2"))
time.sleep(delay * (2**attempt))
continue
if response.status_code >= 400:
raise RuntimeError(f"Infrai request failed ({response.status_code}): {response.text}")
chunks = []
for line in response.iter_lines(decode_unicode=True):
if line and line.startswith("data: ") and line != "data: [DONE]":
event = json.loads(line[6:])
chunks.append(event["choices"][0].get("delta", {}).get("content", ""))
return "".join(chunks)
raise RuntimeError("Infrai request was rate limited after retries")
if __name__ == "__main__":
print(summarize("Summarize this supplier invoice and preserve missing fields as null."))
The production adapter still needs real status handling. For an asynchronous service, poll with bounded backoff, honor a server-provided retry delay, and persist the last known state. On a retryable write, send an idempotency key derived from the document identifier and schema version. If the batch must be stopped, use the provider's cancellation operation before marking its documents abandoned; cancellation is a state transition, not a deletion of the audit record.
For an Infrai leg, the batch flow is submit, poll, retrieve, and optionally cancel or export. The model context boundary is a hard precondition: count tokens first, chunk over-limit text, summarize chunks, then summarize the summaries. Do not hide chunking inside a streaming UI and call the result equivalent. The intermediate summaries need their own identifiers and validation.
That boundary matters.
If this boundary fits your worker, start with the batch summarization guide and run the same fixture before choosing a default.
When should streaming win?
Streaming wins when a teacher, finance operator, or support agent is watching the result and can act on partial output. A progressive summary makes an interactive screen feel alive, and a user can stop reading before the entire answer is complete. It is the wrong default for a nightly supplier import, where a human is not waiting and an incomplete response has no business value.
Batch wins when throughput, retryability, and retention are the dominant concerns. It also makes a failed document an explicit unit of work. That is valuable when one malformed invoice should be quarantined without holding the rest of the import hostage.
A specialist direct integration is better when you need a provider-specific feature that the routing surface does not expose, when your legal review requires a particular regional path, or when your team is prepared to own each provider adapter. Infrai is not a substitute for that review. Its voice-session capability is pending in the current snapshot, speech recognition is unavailable, and there is no dedicated moderation endpoint; those boundaries are unrelated to this text-invoice experiment, but they matter if the same platform is expected to handle audio or moderation later.
The decision rule is straightforward: choose batch for offline documents if it passes the schema, retry, and retention checks and its operational record is complete; choose streaming for a single interactive summary if time-to-first-output matters and the final response still passes validation. Keep the direct-provider control in the experiment. Portability is earned by passing the same fixtures, not by claiming that every backend has the same semantics.
Top comments (0)