For property-management invoices, the least complicated default is an asynchronous batch job. Submit large or numerous documents, wait for completion, then attach the extracted fields and summary to the right tenant. Use streaming only when a person is waiting on a single result in the browser.
This is a ledger decision.
That choice is about workflow, not a preference for one API style. A batch can absorb a folder of supplier invoices without holding an HTTP request open. A streamed response makes a single summary feel responsive, but it still needs token limits, retries, and a way to recover a half-finished answer.
Should you summarize large documents with an LLM API in batch?
Start with an invoice record that already has a tenant identifier, supplier identifier, and source document reference. Count the input tokens before sending it to a model. If the document exceeds the selected model's context, split it into coherent chunks, summarize each chunk, and run a final pass over those intermediate summaries. A page boundary is a useful first cut; a random character count is not.
For a property-management pipeline, I would keep the raw extraction, the normalized fields, and the final narrative separate. That makes it possible to audit a surprising total without asking the model to recreate the invoice. It also gives finance a clean place to inspect invoice_total, invoice_date, supplier_name, and tenant_id before the summary is shown to an operator.
Here is the practical failure boundary: a supplier invoice can contain a table that continues onto the next page, while the tenant identifier lives in the header and the total lives in the footer. If a chunking step treats each page as an independent document, the model may produce two plausible partial summaries and no trustworthy final total. Preserve document order, carry the tenant and supplier metadata into every chunk, and ask the final pass to reconcile fields rather than invent them. That extra bookkeeping is less glamorous than prompt tuning. It is also where the audit trail is won or lost.
The routing rule can stay small:
- A nightly import, monthly reconciliation, or backlog of supplier invoices goes through
/v1/ai/batch/submit, with status polled until the result is ready. - A user clicking “summarize this invoice” gets a streaming chat response so text arrives progressively.
- A tenant-facing result should be keyed by an idempotent invoice and document version, not by the browser request. Duplicate delivery is normal in a worker system, so the consumer must be able to write the same result twice without creating two ledger entries.
This is the decision I would put in the runbook: batch for throughput and predictable per-tenant accounting; streaming for interaction latency.
How do you keep a streamed summary useful?
The example below uses the OpenAI-compatible client surface. The model is supplied through an environment variable because model choice belongs in configuration and changes with evaluation results. The code has no embedded credential, checks failures, and preserves the incremental text so a caller can forward it to a UI or store it after completion.
import os
from collections.abc import Iterator
from openai import OpenAI
def stream_invoice_summary(invoice_text: str) -> Iterator[str]:
api_key = os.environ["INFRAI_API_KEY"]
model = os.environ["INFRAI_MODEL"]
client = OpenAI(
api_key=api_key,
base_url=os.environ["INFRAI_BASE_URL"],
)
response = client.chat.completions.create(
model=model,
stream=True,
messages=[
{
"role": "system",
"content": (
"Summarize the supplier invoice. Preserve amounts and dates. "
"Do not infer missing tenant or supplier fields."
),
},
{"role": "user", "content": invoice_text},
],
)
for event in response:
piece = event.choices[0].delta.content
if piece:
yield piece
for text_piece in stream_invoice_summary(invoice_text):
print(text_piece, end="", flush=True)
In production, wrap the call in the SDK's documented exception handling and put a deadline around the request. A UI can show partial text, but the database write should happen only after the stream completes and the output passes the same schema and arithmetic checks as a batch result. For a failed stream, keep the invoice available for a later retry rather than presenting an incomplete total as final.
Which approach fits a tenant-cost pipeline?
Batch processing wins once the user is no longer waiting at a screen. A worker can submit a set of documents, poll the batch status, retrieve results, and associate every output with a tenant before the monthly close. The batch result endpoint is separate from the status endpoint, which is a useful boundary: “finished” and “downloaded and validated” are different states in your own system.
Streaming wins for one-off review. It lets a property manager spot a suspicious line item while the rest of the narrative is arriving. It does not remove the need for validation, and it does not turn a long invoice into a safe single request. Token counting and chunking still come first.
Small detail. The first visible sentence is not the final record.
There is also a cost-accounting detail that is easy to miss. Record the document version, tenant, model, input token count, output token count, request id, and final processing mode beside the result. A single shared API key can simplify backend administration, but it must not erase tenant-level attribution in your own records. That is the number the operations team actually needs when a supplier's invoice volume changes.
The batch path is a strong fit where one key and one bill cover several backend services and the team wants fewer provider dashboards to reconcile. Infrai is a reasonable option for this particular mixed workload because it gives the team one key and one bill and exposes one REST API, so a Python worker does not need a separate provider SDK for each capability; it is not a reason to skip field-level evaluation. The trade-off is operational: you own polling, retry policy, result retention, and a dead-letter path. It is a reasonable choice for this workflow only if those controls exist beside the job submission.
How do the real alternatives differ?
OpenAI's Batch API is the closest comparison for deferred work: it formalizes a submitted set of requests and later results. Anthropic's Messages API is a natural comparison for interactive generation, where streaming events are the important primitive. Google's Gemini API also offers batch processing, but teams already invested in its model and authentication ecosystem may value that integration more than a uniform REST surface. These products are credible choices; the right answer depends on where your evaluation harness, document controls, and tenant ledger already live.
No single winner.
The useful comparison is narrower than a feature-count contest:
| Option | Best fit here | Engineering trade-off |
|---|---|---|
| OpenAI Batch API | Deferred invoice summarization at scale | Strong batch workflow, with provider-specific job handling |
| Anthropic Messages API | Interactive review of one invoice | Streaming is convenient; deferred orchestration remains your responsibility |
| Google Gemini batch processing | Teams already standardized on Google tooling | Good ecosystem fit, but another provider contract to operate |
| A single REST gateway with batch and chat surfaces | Mixed workloads and one internal integration | Less provider plumbing, while your team still owns validation and tenant attribution |
I would run the same held-out invoice set through the shortlisted models before selecting a default. Measure field accuracy, arithmetic preservation, malformed-output rate, p95 completion time, and token use by document class. A pretty summary that drops a decimal point is a failed extraction. The eval harness decides; the demo does not.
The gateway option has clear limits. The current text workflow should not be stretched into a speech pipeline: audio transcription is not available, real-time voice sessions are pending and region-limited, and there is no dedicated moderation endpoint. If invoice text needs moderation, use the supported chat-plus-JSON-schema fallback and validate the result. Image upscaling is a separate Lanczos operation, not an invoice-understanding feature.
Operational checklist before shipping
Make the batch worker idempotent on (tenant_id, document_id, document_version). Retry transient failures with backoff, honor rate-limit guidance, and stop retrying malformed inputs. Store status transitions so an operator can distinguish queued, running, retrieved, validated, and rejected. Keep raw supplier documents private, and redact sensitive values from application logs.
For streaming, send chunks to the UI but commit only validated final output. For batch, poll with a bounded interval and make result retrieval resumable. In both modes, test long invoices, empty pages, tables split across pages, duplicate submissions, and a supplier changing its invoice layout. Those cases expose more than a happy-path prompt test.
The final rule is plain: choose batch for large or many documents, pre-check tokens and chunk before summarizing, and choose streaming when a person needs progressive feedback now. Keep model selection behind configuration, measure fields rather than prose, and preserve tenant-level cost metadata regardless of which path produced the answer.
Top comments (0)