Large-document summarization should use batch execution for the catalog's durable result and streaming only for the operator's waiting experience. That split usually gives a logistics application better quality control, predictable retries, and a responsive screen without forcing every partial sentence into the data model. The important design choice is not batch versus streaming as a single global setting. It is deciding which stage needs a final answer and which stage needs early visibility.
For a messy product catalog, I would chunk descriptions, summarize each chunk, combine those summaries, validate the result, and persist the final artifact. A user-facing review page can stream progress or partial text while that work runs. The stored summary remains a completed object, not an accidental concatenation of tokens.
How should an LLM API summarize large documents with batch?
Start with the output contract. If the summary will drive search facets, shipment notes, or an approval queue, the application needs a reproducible final record with a status, source revision, and validation result. That points to a batch-style job. It can run asynchronously, retry a failed unit, and leave the catalog item available for inspection if one description needs human review.
Streaming solves a different problem. It reduces perceived waiting by showing incremental output while the model is producing it. It does not, by itself, make a long document fit into a request, make a partial response safe to store, or guarantee that the first text shown is the text that should be used downstream. Treat it as a delivery mode around a job, not as the job's source of truth.
Small documents do not need a complicated pipeline.
| Work stage | Batch execution | Streaming delivery |
|---|---|---|
| Primary job | Produce a durable result | Show progress or partial output |
| Safe persistence point | After completion and validation | After the stream closes and validation passes |
| Main trade-off | More waiting before the result | More client and recovery complexity |
I first assumed the fastest visible text would be the best user experience. The better rule is narrower: stream when a person is actively waiting, but batch the work that another system will trust.
This distinction matters in logistics because descriptions are often inconsistent: dimensions may use different units, packaging notes may appear in the middle of prose, and one item can contain several variants. A polished paragraph arriving quickly is less valuable than a complete summary that preserves uncertainty and does not silently merge variants.
The following TypeScript sketch keeps those concerns separate. The interface is intentionally generic; an adapter can map it to the chosen inference service without making the rest of the application vendor-specific.
type CatalogItem = {
id: string;
revision: string;
description: string;
};
type Summary = {
itemId: string;
sourceRevision: string;
text: string;
status: "complete" | "needs_review";
};
type TextGenerator = {
generate(input: string): Promise<string>;
stream(input: string): AsyncIterable<string>;
};
function splitIntoChunks(text: string, maxCharacters = 6000): string[] {
const words = text.trim().split(/\s+/);
const chunks: string[] = [];
let current = "";
for (const word of words) {
if (current.length + word.length + 1 > maxCharacters && current) {
chunks.push(current);
current = "";
}
current += current ? ` ${word}` : word;
}
if (current) chunks.push(current);
return chunks;
}
async function summarizeCatalogItem(
item: CatalogItem,
generator: TextGenerator,
): Promise<Summary> {
const chunks = splitIntoChunks(item.description);
const partials: string[] = [];
for (const chunk of chunks) {
partials.push(await generator.generate(
`Extract product facts from this catalog text. Preserve unknowns and units.\n\n${chunk}`,
));
}
const finalText = await generator.generate(
`Combine these product facts into one concise catalog summary. Do not invent missing values.\n\n${partials.join("\n\n")}`,
);
const status = /unknown|conflict|missing/i.test(finalText)
? "needs_review"
: "complete";
return {
itemId: item.id,
sourceRevision: item.revision,
text: finalText,
status,
};
}
The example uses a conservative character limit as an application policy, not as a claim about a particular model's context window. In production, the limit should be derived from the selected model's documented input constraints, the prompt, and the required output allowance. Chunk boundaries should also respect structure where possible. Splitting in the middle of a dimensions table is a quality bug waiting to happen.
The limitation is real: this two-stage approach adds latency and input tokens, so it is a poor fit for a tiny description where a single validated call is enough. It is also a poor fit when the product cannot tolerate an asynchronous result. In those cases, use the simpler synchronous path and accept the narrower validation boundary.
Why does streaming feel faster but fail as a storage strategy?
A stream changes when bytes become visible. It does not change the semantics of the final answer. The first fragment may be a heading, a qualification, or an incomplete value. A browser can render it, but a database should not mark the summary complete until the stream closes and the application has checked the accumulated text.
For a review interface, expose two states: progress and final content. Progress can say that 4 of 9 chunks are complete. Final content can arrive incrementally for the active item, but the save operation should happen after completion and validation. If the connection to the browser closes, the worker should continue or record a resumable job; the user interface is not the execution engine.
A simple stream consumer makes the boundary explicit:
async function previewSummary(
input: string,
generator: TextGenerator,
onText: (text: string) => void,
): Promise<string> {
let completeText = "";
for await (const fragment of generator.stream(input)) {
completeText += fragment;
onText(fragment);
}
if (!completeText.trim()) {
throw new Error("The completed stream was empty");
}
return completeText;
}
There is a subtle trap here: token or fragment boundaries are transport boundaries, not language boundaries. Do not parse JSON, units, or a numbered list from each fragment unless the protocol provides a structured event contract and the parser is designed for incremental input. Buffering until a complete object is available is less exciting and much easier to reason about.
The WebRTC API documentation is useful as a reminder that real-time browser communication has its own connection and event model; it is not a general-purpose document-processing queue. A transport that can deliver data promptly still needs application-level completion, cancellation, and recovery rules.
How do quality and latency trade against each other?
The fastest path is rarely the best path for every catalog item. A short, clean description may need one generation call. A long entry with conflicting measurements benefits from chunk-level extraction, a merge step, and a validation pass. That creates more latency and more input tokens, but it also creates places to detect bad source text before the summary reaches a customer or warehouse workflow.
Use a decision rule based on document shape rather than a universal threshold. For example, route an item to the fast path when it is below the configured size limit and contains no repeated variant markers. Route it to the careful path when it is long, contains multiple SKUs, includes tables, or has conflicting units. Those are engineering signals, not magical model properties. Log the reason for each route so the rule can be tuned against review outcomes.
Cost control follows the same logic. Cache results by source revision, avoid reprocessing unchanged descriptions, and retain chunk summaries only as long as they are useful for audit or retry. Do not optimize by truncating the end of a catalog entry without measuring what information is lost. In logistics, the last paragraph may contain handling restrictions.
I would measure four things separately: time to first visible fragment, time to completed summary, percentage sent to review, and input/output tokens per completed item. A single average latency hides the difference between a fast preview and a slow but correct durable result. Keep a small manually reviewed sample of descriptions with units, variants, and missing fields; compare new routing rules against that sample before changing the default.
What can break after the model call?
Most production failures happen around the call. Source text can change while a job is running. A retry can duplicate a write. A browser can disappear. A stream can end without a usable final object. Each case deserves an explicit state transition.
Use an idempotency key built from the catalog item ID, source revision, prompt version, and processing policy. Store a job record before execution, then update it with started, partial, completed, needs_review, or failed status. A retry should claim the same logical job and write the same revision, rather than creating a second summary that looks current.
Cancellation needs a boundary too. Canceling a browser request should stop delivery to that browser; it should not automatically cancel the worker if the result is still needed. Conversely, an operator cancel action should mark the job and prevent a late response from overwriting the canceled state. These details are mundane. They are also where duplicate summaries and stale catalog data come from.
Observability should attach the item ID, source revision, route decision, chunk count, attempt number, duration, and completion status to each job. Keep prompts and model responses out of ordinary logs when they may contain sensitive commercial data. A redacted sample or content hash is usually enough to connect an incident to the source revision.
One implementation detail deserves more space because it affects both quality and operations. With a 6,000-character application limit, a description that becomes 9 chunks should not look like a single opaque request in the job record. Record each chunk's index, source revision, attempt count, and completion time, then expose a progress value such as 4 of 9 to the review screen. If chunk 4 fails, retry chunk 4 and preserve the successful outputs from chunks 1 through 3; do not restart the entire item unless the prompt version or source revision changed. The merge step should receive an ordered set of completed partials, and the final write should compare the revision it read with the revision currently stored. This is a little more bookkeeping than concatenating fragments in a request handler, but it prevents a late result from an old description from quietly becoming the current catalog summary. It also gives the operator a useful answer when a result is delayed: the system can say which unit is waiting, whether the delay is a retry, and whether human review is required.
A practical operating rule
Use batch execution for the canonical summary, and add streaming to the review surface when early feedback has real value. Keep chunking, retries, validation, and persistence in a worker boundary. Keep transport details in an adapter. Make source revision part of every stored result.
Before shipping, test short descriptions, descriptions larger than the configured chunk limit, malformed input, mixed units, repeated variants, an empty completed stream, a dropped client connection, a retry after a timeout, and a source revision change during processing. Read the resulting summaries, not only the status codes.
The final decision is straightforward: choose the path that protects the catalog's meaning, then optimize the waiting experience around it. For most large logistics documents, that means a durable batch pipeline with optional streaming progress, plus a review state for uncertainty. The architecture stays portable because it depends on an application contract rather than a provider-specific workflow.
Top comments (0)