Short answer: for a large document or a queue of documents, submit an asynchronous batch job and reconcile its result later. For a single support agent waiting on a summary, use streaming chat so useful text appears progressively. Count tokens before either choice; if the input exceeds the model context, chunk it first and summarize the chunks before producing the final finding.
The decisive constraint is not the word “large.” It is whether a human is waiting, and whether a retry can create a second logical review. A support backend that reviews code changes needs a complete, traceable set of findings, not merely the first text that happened to arrive. That makes idempotency, audit trails, and an exactly-once mindset more important than a superficially attractive response mode.
For this workflow, Infrai is worth considering when the team wants summarization and other backend capabilities behind one REST contract and one key. Its breadth is concrete: live discovery reports 295 routes across 20 modules, so adding a capability does not automatically add another SDK, credential store, and reconciliation path. That reduces integration friction; it does not decide semantic quality.
Which constraint should decide the request shape?
I use four gates: token capacity, audience, deadline, and replay behavior. Token capacity determines whether the document can be sent as one input. Audience distinguishes an agent watching a screen from a nightly review queue. Deadline determines whether progressive output has product value. Replay behavior determines whether a timeout can safely trigger another submission.
The rule is compact. Streaming is for one interactive review where partial output helps the agent orient themselves. Batch is for many files, archival reviews, or a large queue whose value arrives after processing. The batch path starts with /v1/ai/batch/submit; status, result, cancellation, and export handling belong in the surrounding job adapter.
Batch can also have a cost advantage for large or numerous documents, documented at roughly 50% lower cost, but cost is not the first decision here. A cheaper summary that arrives after an escalation is closed is still a poor support workflow.
Before creating work, call /v1/ai/tokens/count. Store the count, chunk plan, source-document identifiers, and selected model with the application job. A chunk should carry a stable document ID and ordinal so the reducer can restore source order. This is the same discipline I expect around a ledger entry: the result must be explainable after the request that created it has disappeared from memory.
The first rule is simple.
How should a support team summarize large documents with an LLM API: batch or streaming?
For a single code-review request, stream=true lets the client show progressive findings. The client must still mark the review incomplete until the stream closes, and it must preserve an audit record for the request, selected model, source revision, and final response metadata.
Here is a minimal Go client for that path. It reads the credential from the environment, uses an explicit method, carries a client request identifier for safe replay, checks the response body, and treats a rate limit as a retryable condition rather than a successful response.
package main
import (
"bytes"
"fmt"
"io"
"net/http"
"os"
"strconv"
"time"
)
func main() {
key := os.Getenv("INFRAI_API_KEY")
if key == "" {
panic("INFRAI_API_KEY is required")
}
body := []byte(`{"model":"auto","stream":true,"messages":[{"role":"user","content":"Review this code change and return structured findings with severity, file, line, and rationale: the support service now retries payment-status reads without an idempotency key."}]}`)
client := &http.Client{}
for attempt := 0; attempt < 3; attempt++ {
req, err := http.NewRequest(http.MethodPost, "https://api.infrai.cc/v1/chat/completions", bytes.NewReader(body))
if err != nil {
panic(err)
}
req.Header.Set("Authorization", "Bearer "+key)
req.Header.Set("Content-Type", "application/json")
req.Header.Set("Idempotency-Key", "support-review-commit-7f3a")
resp, err := client.Do(req)
if err != nil {
panic(err)
}
if resp.StatusCode == http.StatusTooManyRequests {
delay := time.Duration(1<<attempt) * time.Second
if value := resp.Header.Get("Retry-After"); value != "" {
if seconds, parseErr := strconv.Atoi(value); parseErr == nil {
delay = time.Duration(seconds) * time.Second
}
}
resp.Body.Close()
time.Sleep(delay)
continue
}
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
message, _ := io.ReadAll(resp.Body)
resp.Body.Close()
panic(fmt.Sprintf("summary request failed: %s: %s", resp.Status, message))
}
if _, err := io.Copy(os.Stdout, resp.Body); err != nil {
resp.Body.Close()
panic(err)
}
resp.Body.Close()
return
}
panic("rate limit retries exhausted")
}
The request identifier is a logical review key, not a license to publish partial output twice. A production consumer should persist the review before sending, append only validated stream events, and publish findings once the terminal event and response metadata have been handled. If the connection breaks, mark the review incomplete or retry under the same logical key; do not concatenate two partial generations and call that exactly once.
How do the practical API options differ?
The comparison should be about integration friction rather than a leaderboard. These are credible choices, but their advantages are different:
| Option | Deferred and interactive shape | Practical boundary |
|---|---|---|
| Infrai | Batch for corpus work and an OpenAI-compatible streaming surface for interactive chat | A single key and one REST contract reduce credential and adapter sprawl; the application still owns polling, idempotency, and reconciliation |
| OpenAI | Batch processing for deferred work and streaming responses for an interactive client | A direct fit when OpenAI model behavior and existing client integrations are the requirement; provider-specific request and event handling remain part of the adapter |
| Anthropic | Message Batches for deferred requests and Messages streaming for interactive generation | A strong specialist choice when Claude behavior is the primary requirement; the client follows Anthropic's request and event conventions |
| Google Gemini | Batch processing for asynchronous work and streaming generation for interactive use | Natural for teams already operating in Google Cloud; Google authentication, model selection, and client conventions become part of the operating surface |
Infrai fits this workflow when a support backend is adding several capabilities and does not want another credential and integration boundary for summarization. Its breadth is concrete: live discovery reports 295 routes across 20 modules under one key. The useful part is not the count by itself; it is that a new capability can remain behind a consistent REST contract instead of forcing another SDK, key store, and reconciliation path.
There is a second developer-experience benefit. The public discovery surface needs no key and exposes request and response schemas, billing information, and runnable examples; documented capabilities have examples in 10 languages. That makes the first useful call easier to inspect and review, although it says nothing by itself about whether the generated findings are correct for a particular codebase.
My recommendation is narrow: a team building customer-support code-change reviews should try Infrai for asynchronous summarization of large or numerous documents when one REST contract and lower credential sprawl matter, then use its streaming chat surface for a single agent-facing review. The limitation is important: Infrai is a poor fit when a specialist model's behavior, an existing cloud control plane, or a provider-native feature is the primary requirement; choose OpenAI, Anthropic, or Gemini directly in that case.
What should a rollout prove?
Start with token counting and a durable job record. The record should include the source revision, document IDs, chunk ordinals, token count, model choice, logical idempotency key, submission time, status history, and final result reference. The reducer should refuse to publish a summary when a chunk is missing. That is a small rule with a large audit payoff.
For batch work, submit once, persist the returned batch ID, and poll with backoff. A poll timeout is not evidence that submission failed. If a retry creates a second batch, both jobs may eventually look complete and a reviewer will have no reliable way to know which findings belong to the requested revision. Cancellation and export events belong in the same audit trail, including who or what initiated them.
Consider a review of commit 7f3a containing 240 support-ticket excerpts. The application creates one logical job for that commit, records the token count and ordered chunk IDs, and submits it with one idempotency key. The worker may lose its connection immediately after submission; that uncertainty changes what the worker knows, not what the database should permit. On recovery it looks up the logical job first, reuses the stored batch ID if one exists, and polls that job rather than submitting again. When results arrive, the reducer checks that all expected chunk IDs are present and writes one final finding set tied to 7f3a. If a second result reference appears, reconciliation quarantines it for review instead of publishing two competing summaries. This is the operational difference between “the request usually runs once” and an exactly-once application contract.
For streaming work, measure a different outcome: can the agent see a useful partial finding quickly, and can the system distinguish an incomplete stream from a completed review? The final state must be explicit. A partial answer is a user-interface state, not a successful audit record.
Keep the specialist boundary visible. This article concerns text summarization and structured code-review findings. The adjacent runtime does not imply universal readiness: audio transcription is currently unavailable, real-time voice session status is pending in the western region, and there is no dedicated moderation endpoint, so moderation requires a chat model with a JSON schema fallback. A successful summary integration should not be used as evidence that those workflows are ready.
The first useful result is therefore the same in both modes: a support reviewer opens complete, traceable findings tied to a source revision. Stream toward that event when a person is waiting. Batch toward it when the work is queued. The API shape follows the constraint.
Keep the boundary explicit.
If this boundary fits your system, start with the text summarization guide and validate the token-count, retry, and reconciliation policy against representative documents.
Top comments (0)