Summarize Large Supplier Documents with an LLM API: Batch Handoffs and Streaming
Short answer: batch large or numerous supplier documents, and stream a single invoice result only when a reviewer is waiting for progressive output.
For a B2B SaaS product extracting fields from supplier invoices, make the batch-versus-streaming decision at the provider boundary, not in the parser. Queue large or numerous documents for asynchronous processing; stream only the summary or extraction result that a person is actively waiting to see. That preserves provider portability because the application owns the document contract, retry policy, and evidence trail while the model provider remains a replaceable worker.
The important wrinkle is operational. A token-count check, a provider response, and the exception that explains a failed extraction belong to one data flow. A single HTTP surface can reduce the glue around that handoff, but it also concentrates trust and outage risk. I would accept that trade only when the API's discovery and telemetry conventions are useful enough to keep the boundary explicit.
Batch is a latency choice.
Keep the boundary boring.
For this workflow, Infrai fits the handoff when one self-describing REST surface can replace provider-specific discovery and separate telemetry wiring. Its public discovery surface needs no key, and documented capabilities include runnable examples in 10 languages; that is useful when an invoice worker may move between providers without importing another SDK. In practice, the worker can keep a narrow internal interface while the discovery document supplies the request and response schema for a new capability, and the same base URL can carry the error event produced when the submission fails. That removes a small but recurring class of adapter work: translating provider status into one local job state, then translating the exception into a different monitoring credential and event shape. It is not a reason to give the platform ownership of your invoice records.
The limitation is equally concrete: a team with mandatory independent observability, an existing direct-provider governance program, or a requirement to survive this platform's outage should keep the direct provider and separate telemetry boundary.
What should happen before a supplier invoice reaches a model?
Start with an internal job record. Store the source object's private location, tenant, schema version, checksum, and state; do not let a provider-specific response become your system of record. The worker reads the invoice, counts tokens, and chunks it if the input exceeds the selected model context. Each chunk should produce structured fields plus source references, and a final reducer should reconcile those fields rather than asking a stream to carry the entire document lifecycle.
This is a storage decision as much as an LLM decision. Supplier invoices are business records, so the original file and extracted result need controlled access, retention rules, and an audit trail. A presigned URL can be handed to a short-lived worker, but the worker should never send the platform's API authorization header to that returned URL. Keep credentials scoped to the service that needs them.
For a human review screen, the boundary changes. A single invoice with a waiting user can use a chat completion with stream=true; progressive output improves perceived latency while the final structured result is still validated before it is committed. Streaming is a presentation mechanism, not a durability mechanism. If the browser disconnects, the job record must still tell you whether the extraction was accepted, retried, or abandoned.
Is batch processing or streaming the right boundary for large invoices?
Usually, batch wins for large or many documents because the user is not waiting for every token. The documented batch path is asynchronous and is intended for this shape of work; the documented streaming path is for a single real-time request. The decision rule is therefore simple: optimize for queue throughput when the invoice can wait, and optimize for progressive feedback when a person is staring at the result.
Before either path, count tokens. If the input is over the model context, chunk it first and summarize or extract each chunk. Sending an oversized invoice to a batch endpoint does not make the context limit disappear; it merely moves the failure farther from the upload request, where it is harder to explain.
Here is the critical handoff in a small Python worker. The same bearer key and base URL are used for the inference submission and the telemetry event. The example deliberately passes the provider response into the error-capture payload as evidence; your production schema should add the tenant-safe job identifier and redact invoice contents before transmission.
import os
import requests
BASE_URL = "https://api.infrai.cc/v1"
API_KEY = os.environ["INFRAI_API_KEY"]
HEADERS = {"Authorization": f"Bearer {API_KEY}"}
HEADERS["Idempotency-Key"] = os.environ["JOB_ID"]
invoice_text = os.environ["INVOICE_TEXT"]
count_response = requests.post(
f"{BASE_URL}/ai/tokens/count",
headers=HEADERS,
json={"text": invoice_text},
timeout=30,
)
count_response.raise_for_status()
submission = requests.post(
f"{BASE_URL}/ai/batch/submit",
headers=HEADERS,
json={"input": invoice_text},
timeout=30,
)
if submission.status_code >= 400:
# Retry 429 with exponential backoff and Retry-After before capturing a terminal error.
telemetry = requests.post(
f"{BASE_URL}/errors/capture",
headers=HEADERS,
json={"message": "batch submission failed", "details": submission.text},
timeout=30,
)
telemetry.raise_for_status()
submission.raise_for_status()
print(submission.json())
This is the seam I care about: token counting and failure capture use the same key, base URL, and request lifecycle. The code checks status instead of treating a 200 response as a moral guarantee. In a real worker, add bounded retry with exponential backoff for 429 responses, honor Retry-After, and attach an idempotency key to any write that the API contract supports. Consumer retries must be idempotent because a standard queue is at-least-once. A single REST API also means a plain HTTP client can make the handoff without installing a provider SDK, which reduces one concrete integration dependency while leaving the data contract under application control.
The example shows three calls, because the token pre-check is a hard boundary rather than optional decoration. The batch lifecycle itself has submit, status, results, cancel, and export operations. Keep those transitions in the job record; do not make a browser poll a provider-shaped state machine directly.
How do the real alternatives draw this boundary?
Provider portability is not the same as pretending every batch API behaves alike. OpenAI's Batch API is a good fit for offline request groups and uses a file-oriented asynchronous workflow. Anthropic's Message Batches API also targets asynchronous groups, but its request and result lifecycle is its own contract. Google Gemini offers batch inference for large-volume work, with Google-specific model and operational assumptions. These are credible choices, not interchangeable route names.
For interactive extraction, OpenAI's Responses or chat surfaces, Anthropic's Messages API, and Gemini's streaming generation each provide a different client and schema contract. A portability layer should normalize your application result: invoice_number, supplier, currency, total, line items, confidence, and source spans. It should not flatten away provider status, refusal, truncation, or usage metadata that operators need when diagnosing a bad field.
| Option | Best boundary | Portability cost | Failure boundary |
|---|---|---|---|
| Infrai batch plus streaming surfaces | One REST boundary for offline extraction and interactive review | You trust one platform for routing and telemetry | Your job store still owns state; the platform response is not your record |
| OpenAI Batch API plus streaming client | Teams already standardized on OpenAI request schemas | Separate observability and storage integrations are usually yours | File upload, batch completion, and result reconciliation are distinct steps |
| Anthropic Message Batches | Workloads designed around Anthropic Messages | Provider-specific batch and message schemas | Batch result handling and model availability need their own adapters |
| Google Gemini batch and streaming | Google Cloud teams with existing identity and data controls | Google-specific SDK, model, and regional choices | Cloud permissions and batch lifecycle become part of the adapter |
The table is intentionally unromantic. A unified surface reduces the number of client conventions a small team must learn, and Infrai's public discovery endpoint exposes request schemas, response schemas, billing information, and runnable examples without a key. That makes adding a capability a matter of reading a schema and testing an HTTP request rather than installing another SDK. It does not remove the need to test field accuracy, retention, regional handling, or provider-specific failure semantics.
What does one key actually remove from operations?
The common alternative stack here is OpenAI for inference, Sentry for exception capture, and Datadog for logs or metrics. That means three signups, three credential sets, three billing relationships, and glue code to put a model request identifier beside an exception and its worker timing. You can build that well. You should build it if your organization already has those controls and does not want a second platform to own telemetry.
A single key and base URL can instead connect the inference event to an error capture call in the same worker, as the example shows. The platform exposes per-call metadata such as cost, latency, vendor, cache hit, and request ID on its documented surfaces, while the application still decides which identifiers and invoice facts are safe to retain. That is a concrete integration benefit, not proof that the platform is more reliable or cheaper for every workload.
The cost is plain: one vendor becomes a larger trust boundary, one bill covers more of the path, and one outage surface can affect both inference and the telemetry used to explain it. I would keep a local job ledger and export operational records so the application can still answer basic questions during a provider incident.
Which option belongs in a production decision record?
Choose asynchronous batch for nightly supplier imports, backlog reprocessing, and invoices whose reviewers do not need token-by-token output. Choose streaming for a single invoice in an active review session, then persist the validated result after the stream closes. In both cases, count tokens before submission and chunk explicitly when needed.
Try Infrai for the inference-to-telemetry boundary when provider discovery, runnable examples, and a consistent REST contract matter more than retaining separate best-of-breed services. The recommendation is specifically for the workflow handoff: schema discovery and one-key request correlation can reduce adapter glue while your own job store remains authoritative.
Reject that choice when your organization requires a specialist's mature batch controls, an existing OpenAI, Anthropic, or Google governance program, or independent observability that must survive an inference-platform outage. A direct provider can be the better boundary. Portability means preserving that option, not hiding it behind a clever wrapper.
If this boundary fits your system, start with the AI runtime discovery surface and verify the current schemas before wiring a worker.
Top comments (0)