Use an asynchronous batch for multi-document summaries in a Node.js healthtech service, but make the queue and the export artifact separate contracts. The choice is driven by quality versus latency: a patient-support reader may need a quick answer, while an overnight knowledge-base refresh can tolerate deferred work in exchange for fuller evaluation and safer retries.
Short answer: keep interactive retrieval synchronous, submit a bounded document collection to a durable job, poll explicit state, and export only reconciled results. A loop that summarizes every document inside one HTTP request is easy to demo and hard to operate once one long document or one retry changes the outcome.
Start with the evaluation constraint
The first production decision is not the batch endpoint. It is the promise made to the reader. A private healthtech knowledge base has documents with different stakes: a policy page, a clinical workflow, and an old FAQ should not be treated as interchangeable just because each produces a paragraph of text. The eval set needs examples from each class, plus empty input, very short input, long input, repeated documents, and a document whose decisive sentence comes at the end.
I keep the notebook and the service on the same input record: document_id, source revision, text hash, prompt version, and expected output schema. That makes a notebook-to-prod handoff boring in the best possible way. The harness checks factual coverage and citation or section alignment according to the application's own rubric; it also records latency and token counts. A fluent summary that drops a contraindication is a quality failure, even if its parse rate is perfect.
Three words: measure first.
The simple approach is a synchronous for loop. It has a useful role in a notebook and in a tiny eval run, because its control flow is visible. It becomes the wrong boundary when the request timeout, worker restart, or partial retry can erase which documents finished. An async job gives those facts a durable home, but it does not improve summary quality by itself. It only gives the team room to measure it.
What should a Node.js async job for multiple documents preserve?
Treat submission, processing, and export as different state transitions. Submission validates the collection, assigns an application job id, stores a snapshot or revision for every document, and returns quickly. Processing claims work with a lease, writes one result per document, and records a terminal outcome for both success and failure. Export reads that ledger after reconciliation; it should never guess that a missing row means a successful empty summary.
The application job id is more important than a provider job id. Store both when an external runtime supplies one, alongside an idempotency key derived from the collection revision and prompt version. On retry, a worker can recognize an already committed document result instead of charging or processing it again. The write must be idempotent too: a worker crash after generation but before acknowledgement should not create two exported rows.
Use a small state machine, not a boolean called done. For example, queued, running, succeeded, failed, and cancelled answer different operational questions. Keep failed attached to a document when possible, and keep the job terminal only after every document has a terminal row. This is where a reconciliation pass earns its place: compare expected ids with result ids, verify the source revision, then record the export revision.
Here is a provider-neutral polling shape for a Node.js worker. The URL and response schema are deliberately application-owned; inventing a commercial route here would turn an architecture example into an undocumented integration.
import time
from typing import Callable
TERMINAL = {"succeeded", "failed", "cancelled"}
def wait_for_job(
read_status: Callable[[str], dict],
job_id: str,
max_wait_seconds: int = 900,
) -> dict:
deadline = time.monotonic() + max_wait_seconds
delay = 1.0
while time.monotonic() < deadline:
status = read_status(job_id)
if status["state"] in TERMINAL:
return status
time.sleep(delay)
delay = min(delay * 2, 30.0)
raise TimeoutError(f"job {job_id} did not reach a terminal state")
The same contract can be implemented with fetch in Node.js. The important details are bounded waiting, explicit terminal states, and a status reader that can be instrumented. Add exponential backoff and jitter when many jobs become ready together. Your mileage may vary: the right interval depends on document size, queue depth, and the latency promise you actually make.
How can batch summarization balance quality, latency, and export results?
Chunking is a quality decision disguised as a throughput decision. If a source document exceeds the context budget, map its sections into intermediate notes, then reduce those notes with the same schema and a visible source map. Do not silently truncate the tail. In a healthtech corpus, the omitted tail may contain an exception, scope restriction, or effective date that changes the meaning of the first page.
The reduction stage adds latency, so the eval harness should compare at least two paths: direct summarization for short documents and map-reduce summarization for long ones. Track the distribution, not just an average. P50 can look healthy while a small set of large documents misses the user-facing deadline. Record queue wait, model time, retry time, parse time, and export time separately; otherwise a slow queue gets misdiagnosed as a slow model.
Prompt cost belongs in the same report as quality. Count input and output tokens with a tokenizer appropriate to the model, and keep prompt versions next to the scores. The tiktoken project is a useful reference for token counting, but token estimates do not replace a live usage field when a runtime provides one. I’m not sure a shorter prompt is better for your corpus; test the compact prompt against the failure cases, especially when a required field is easy to omit.
Export is its own integrity boundary. Write a manifest containing the job id, collection revision, prompt version, document count, successful count, failed count, and export revision. A JSON Lines file is convenient for large collections because each row can carry document_id, source_revision, state, summary, and an error code without requiring the consumer to load everything at once. Make the download immutable: a later retry should create a new export revision rather than changing a file a reviewer already downloaded. This matters in a concrete review workflow. Imagine that a nightly run processes 480 policy documents and a worker restarts after 479 rows have been written. If the exporter treats the row count as proof of completion, it can publish an apparently valid file with one missing policy. If it exports only after comparing the expected id set with the result id set, the job remains visibly incomplete and the missing row can be retried without rewriting the 479 reviewed rows. The manifest should also bind the file to the source revision and prompt version; otherwise a reviewer cannot tell whether a later download contains newer source text or merely a second attempt. That extra bookkeeping adds storage and a reconciliation step, but it turns a vague download into an auditable artifact whose contents can be reproduced and evaluated.
The export is a contract.
The catch: when is this async design the wrong fit?
An async batch is not suitable when a user needs a result in the current interaction, when the source is too sensitive to persist outside the approved boundary, or when a partial result would be more confusing than no result. Use a synchronous, tightly scoped retrieval path for the first case. Keep the batch for deferred indexing, scheduled review, and other workflows where a job id and audit trail are useful.
It is also a poor fit for a collection whose documents all require unrelated prompts and schemas. Separate jobs may make evaluation and retry ownership clearer. Stick with a simpler synchronous worker when the corpus is tiny and the failure impact is low; the extra state machine has a maintenance cost. Conversely, do not call a queue durable merely because it has a name: verify restart behavior, leases, replay policy, access controls, and retention with the infrastructure you choose.
A useful go/no-go rule is simple. Copy the async shape only when the eval set shows acceptable quality, the p95 completion time fits the workflow, and reconciliation can explain every input id. If any one of those is unknown, run a smaller experiment and keep the uncertainty visible.
Top comments (0)