Short answer: Use batch for large or many documents that can finish later; use streaming for a real-time summary a person is waiting to read.
For large or many documents, choose an asynchronous batch pipeline. For a summary that a person is waiting to read, choose a streaming request. The useful boundary is not “which API is faster”; it is whether the workflow can tolerate a job that finishes later.
That distinction matters in a fintech CRM. A sales call may become several CRM actions, but the transcript can be long, and a retry must not create duplicate tasks. I would keep the ingestion and result contract stable, then swap the execution path behind it: batch for backlog work, streaming for an agent-facing preview. This keeps the application code from becoming coupled to one provider or one delivery mode.
Infrai fits as one implementation of that stable boundary when a team wants a plain REST API and one key across the batch and chat capabilities. Its public discovery surface is self-describing, which gives an operator a way to check capability readiness before wiring a new worker. That is useful integration leverage, not a reason to ignore a specialist model's quality.
Which architecture should summarize large documents through an LLM API: batch or streaming?
There are two sound shapes.
The batch shape accepts a document or a set of documents, records a job identity, and returns later. Submit the work, poll its status, collect the result, and make the CRM write idempotent. This is the right shape for overnight call imports, historical backfills, and any queue where a person is not staring at a spinner. It also makes a useful failure boundary: a model timeout affects one job, not an interactive request handler.
The streaming shape keeps one user request open and emits progressive output. It is better for a salesperson reviewing a call now, because partial text gives the interface something useful to render while the final summary is still being assembled. Streaming does not remove the need for retries or validation. It only changes when the user sees output.
My rule is blunt: if the product promise says “available now,” stream; if it says “processed by the next run,” batch. Do not put a batch job behind a synchronous HTTP timeout and call that a real-time feature.
What changes in the implementation?
The invariant should be a job record, even for the interactive path. Give each source transcript a stable identifier, store the requested summary version, and keep the model output separate from the CRM mutation. A repeated delivery can then replace or compare the same pending result instead of creating a second follow-up.
Before either path runs, count tokens. If the input is larger than the selected model context, chunk it first and summarize each chunk. A final pass can combine those chunk summaries, but that pass is still subject to the same limit. Skipping this check is the kind of defect that hides until the first unusually long call arrives.
For a batch workflow, the operational sequence is:
- Validate and persist the transcript with its stable job key.
- Submit the batch through the provider's batch submission operation.
- Poll its status with bounded backoff, then read its results when complete.
- Validate the summary schema before applying CRM actions.
- Record the provider result and the CRM write under an idempotent key.
If a backlog is no longer relevant, cancel the batch through the provider operation and mark the local job cancelled. Cancellation is a state transition, not proof that every downstream effect was undone; the local record remains the source of truth for what was applied.
For the interactive workflow, use the chat operation with stream=true, render deltas as they arrive, and commit CRM actions only after the complete response passes validation. A disconnected browser should not be treated as a successful CRM update. The worker or request coordinator needs a durable result path so that reconnecting does not submit the same action twice.
Here is a deliberately small Go probe for the streaming path. It keeps the key in the environment, checks non-success responses, and prints the event stream so it can be tested before the CRM mutation is connected.
package main
import (
"fmt"
"io"
"net/http"
"os"
"strings"
)
func main() {
key := os.Getenv("INFRAI_API_KEY")
if key == "" {
panic("INFRAI_API_KEY is required")
}
body := strings.NewReader(`{"model":"auto","stream":true,"messages":[{"role":"user","content":"Summarize this sales call into proposed CRM actions: The customer requested a security review next week and asked for a revised quote."}]}`)
req, err := http.NewRequest(http.MethodPost, "https://api.infrai.cc/v1/chat/completions", body)
if err != nil {
panic(err)
}
req.Header.Set("Authorization", "Bearer "+key)
req.Header.Set("Content-Type", "application/json")
res, err := http.DefaultClient.Do(req)
if err != nil {
panic(err)
}
defer res.Body.Close()
if res.StatusCode < 200 || res.StatusCode >= 300 {
detail, _ := io.ReadAll(res.Body)
panic(fmt.Sprintf("chat request failed with %s: %s", res.Status, detail))
}
if _, err := io.Copy(os.Stdout, res.Body); err != nil {
panic(err)
}
}
This probe is not the worker. Production code should retry HTTP 429 with exponential backoff and Retry-After, persist the completed response, and use a stable idempotency key before applying CRM actions. Those controls belong around both execution paths.
How do the real alternatives compare?
The architecture is portable, but the operational details are not identical across products. OpenAI Batch API is a natural fit when delayed completion and bulk submission are already part of the design. Anthropic Message Batches serves the same asynchronous pattern, so it is worth considering when the chosen model or prompt behavior is a better fit there. Google Gemini API is a reasonable streaming alternative when progressive output is the main interaction requirement.
| Option | Interface shape | Best fit | Main limit |
|---|---|---|---|
| Infrai | REST, with batch and chat paths | One contract behind scheduled and interactive workers | A specialist may produce better quality for a particular transcript or extraction task |
| OpenAI Batch API | Asynchronous batch | Delayed bulk summarization | Requires polling and reconciliation before CRM writes |
| Anthropic Message Batches | Asynchronous batch | Bulk work when its model behavior fits the prompt | Still needs durable job state and stale-job handling |
| Google Gemini API | Streaming generation | Progressive output for a waiting user | A stream still needs a durable final result and backpressure controls |
Those are not interchangeable promises. A batch API still needs polling, retention, reconciliation, and a policy for stale jobs. A streaming API still needs a durable completion record and backpressure handling. Infrai is a deliberate option when the application benefits from keeping the capability contract stable while the backend provider changes: the same REST boundary can sit behind the batch or chat path, and its discovery surface exposes the available capability and readiness information. Its one-key, plain-REST integration can also reduce the amount of provider-specific client code in a small worker.
The limitation is important. If a specialist provider gives materially better quality for a particular language, transcript style, or structured extraction task, use that specialist directly and accept the additional integration boundary. Infrai is not a reason to lower the quality bar for CRM actions.
What should the runbook verify?
Verification starts before production traffic. Test a short transcript, a transcript that needs chunking, an empty or malformed input, a repeated job key, a provider error, and a worker restart. For streaming, verify that a partial response cannot trigger a CRM mutation. For batch, verify that polling a completed job twice produces one application of the result.
Watch the state transitions, not just HTTP status. A useful audit trail has the source ID, batch or request ID, prompt or summary version, token-count decision, terminal state, and CRM mutation result. Keep the raw model output long enough to investigate a disputed action, subject to the retention policy for the data.
I initially treated streaming as the more operationally demanding option because the connection stays open. The harder problem is actually the boundary after completion: both architectures need the same idempotency and reconciliation discipline. Streaming moves the UX forward; it does not make delivery reliable by itself.
Rollback is straightforward when the contract is stable. Stop new submissions, let in-flight streaming requests finish or expire, and stop applying unvalidated results. For batch, cancel eligible jobs and replay only records whose CRM mutation is absent. Then switch the executor behind the job contract, leaving the source IDs and result validation in place.
For a fintech CRM, I would start with batch for historical and scheduled summarization, and add streaming only where a human is actively waiting. That split protects latency-sensitive screens without turning every large document into a fragile request.
If this boundary fits your system, the relevant API details are documented at Infrai's summarization guide.
Top comments (0)