TL;DR
For marketplace sales calls, use an async job when several documents must become one reviewed set of CRM actions; use an inline request only when one short document can finish inside the caller's latency budget. Preserve one result per input, expose partial progress, and export only records that carry their source ID, outcome, and schema version.
Start with this decision table:
| Pick | Use it when | Quality and latency consequence | Operational burden |
|---|---|---|---|
| Inline request | One short transcript produces one independent summary | Fast feedback, but the request deadline limits retries and review stages | Low until traffic spikes or callers retry |
| Bounded parallel calls | A small set of independent transcripts can finish separately | Lower wall time, with variable completion order | The caller owns concurrency, backoff, and reconciliation |
| Durable async job | Multiple documents feed one CRM export or need validation | More queue latency, but enough room for retries and quality checks | Requires job state, idempotency, metrics, and retention rules |
The important boundary is not "batch or no batch." It is ownership. If the API accepts a collection, the service should own that collection through terminal results and a verifiable export. Don't make a client reconstruct truth from whichever promises happened to resolve.
What should a Node.js batch summarization API do with multiple documents?
It should turn an admission request into a stable job record, process every document under a declared concurrency limit, and publish an item-level outcome before it declares the job complete. The result model needs at least four identities: job, input document, processing attempt, and export. Without them, a duplicate submission can look like new work, a retry can overwrite useful evidence, and an export can silently omit a failed call.
For the marketplace example, imagine that a seller has three calls about the same account: discovery, pricing, and legal review. The desired CRM update is not merely three paragraphs. It may include a consolidated next step, an owner, an objection, and evidence that points back to a transcript. Running three promises in parallel is easy. Deciding whether the legal call's failure permits a partial CRM update is the actual product decision.
Use an explicit state machine: accepted means the request and idempotency key are stored; running means at least one item may be in flight; completed means every item has a terminal outcome; and cancelled means no new attempt will start. Keep completed_with_errors out of the top-level state if callers might confuse it with total failure. A clearer contract is completed plus counts for succeeded, failed, and skipped items.
Diagram in words: admission writes the job, the queue releases item IDs, workers fetch text and summarize it, a validator checks structured actions, the reducer decides whether cross-call consolidation is allowed, and the exporter freezes a manifest. Metrics observe every arrow. The database remains the source of job truth.
This separation matters. A queue receipt proves delivery, not a usable summary.
How can an async Node.js job export batch summarization results safely?
Define the contract before choosing the queue. The following TypeScript types make partial completion visible and make every exported action traceable to its document. They also keep model-specific response shapes out of the rest of the application.
type JobState = "accepted" | "running" | "completed" | "cancelled";
type ItemState = "pending" | "running" | "succeeded" | "failed" | "skipped";
type DocumentInput = {
documentId: string;
accountId: string;
transcript: string;
};
type CrmAction = {
kind: "follow_up" | "update_stage" | "record_objection";
owner: string | null;
text: string;
sourceDocumentIds: string[];
};
type ItemResult = {
documentId: string;
state: ItemState;
attemptCount: number;
summary?: string;
actions?: CrmAction[];
errorCode?: "input_too_large" | "invalid_output" | "rate_limited";
};
type BatchJob = {
jobId: string;
idempotencyKey: string;
state: JobState;
schemaVersion: 1;
createdAt: string;
items: ItemResult[];
};
The API should acknowledge admission, return a job ID, and let the caller read job state without holding the original connection open. A separate export operation should read only a terminal snapshot. If exports are generated while workers are still updating rows, two downloads for the same job can disagree even though both appear valid.
Idempotency closes another gap. Bind the client's key to a canonical digest of the ordered input IDs and processing options. A repeated key with the same digest returns the existing job; the same key with a different digest is a conflict. This rule prevents a network retry from generating duplicate CRM actions, while still surfacing a caller that accidentally reused a key for different work.
Consider the awkward case, because it is where a plain Promise.all design stops being convincing. A client submits discovery, pricing, and legal transcripts; discovery finishes first, pricing is retried after rate limiting, and legal fails validation. While the retry waits, the client loses its connection and submits the same request again. The idempotency record should lead that second request to the original job, not create three more items. Once pricing succeeds and legal reaches its retry limit, the job becomes terminal with two successes and one failure. Its export still contains three rows, including the failed legal row and its error code, so the CRM importer can propose supported actions without pretending the collection was complete. If policy says all three calls are mandatory, the reducer marks every proposed action as review-only instead. Same mechanics. Different business rule.
Missing rows are bugs in the contract.
Here is the deeper implementation path. It uses a generic summarizer rather than a vendor SDK, caps concurrency, records each outcome, and refuses to produce an export until all items are terminal. Production storage must make the state transitions atomic; the interfaces keep that requirement visible.
type SummaryOutput = { summary: string; actions: CrmAction[] };
interface Summarizer {
summarize(input: DocumentInput): Promise<SummaryOutput>;
}
interface JobStore {
load(jobId: string): Promise<BatchJob>;
markRunning(jobId: string, documentId: string): Promise<void>;
saveSuccess(jobId: string, documentId: string, output: SummaryOutput): Promise<void>;
saveFailure(jobId: string, documentId: string, errorCode: ItemResult["errorCode"]): Promise<void>;
finishIfTerminal(jobId: string): Promise<void>;
}
async function runWithLimit<T>(
values: T[],
limit: number,
work: (value: T) => Promise<void>,
): Promise<void> {
const pending = [...values];
const workers = Array.from({ length: Math.min(limit, pending.length) }, async () => {
while (pending.length > 0) {
const value = pending.shift();
if (value !== undefined) await work(value);
}
});
await Promise.all(workers);
}
async function processJob(
jobId: string,
inputs: DocumentInput[],
summarizer: Summarizer,
store: JobStore,
): Promise<void> {
await runWithLimit(inputs, 4, async (input) => {
await store.markRunning(jobId, input.documentId);
try {
const output = await summarizer.summarize(input);
await store.saveSuccess(jobId, input.documentId, output);
} catch (error) {
const code = error instanceof RangeError ? "input_too_large" : "invalid_output";
await store.saveFailure(jobId, input.documentId, code);
}
});
await store.finishIfTerminal(jobId);
}
async function exportJob(jobId: string, store: JobStore): Promise<string> {
const job = await store.load(jobId);
if (job.state !== "completed") throw new Error("job_not_terminal");
return job.items
.map((item) => JSON.stringify({ jobId, schemaVersion: 1, ...item }))
.join("\n");
}
The limit of 4 is an example, not a universal tuning value. Measure it. Increase concurrency until queue age improves without pushing rate-limit responses, memory pressure, or downstream latency beyond their budgets. Then leave headroom; a setting discovered under clean test traffic is too optimistic for a burst of morning calls.
Token counting belongs before admission or before an item enters the expensive processing stage. A BPE tokenizer can estimate model input size, but encoding details depend on the model and tokenizer configuration. I'm not sure a character-count shortcut is ever worth the ambiguity for hard limits; verify with the tokenizer used by the chosen runtime. The tiktoken project documents its BPE implementation and supported usage.
Observe quality and latency as one system
An async design can hide slowness. Fix that by measuring queue delay separately from processing time and export delay. A single end-to-end percentile cannot tell you whether workers are slow, capacity is short, or the exporter is waiting on one poisoned item.
Track job admission count, runnable queue age, item attempt count, item duration, terminal outcome, validation failure, and export generation duration. Attach jobId and documentId to logs, but keep transcript text and generated summaries out of routine telemetry. High-cardinality IDs are useful for logs and traces; they can be expensive or unusable as metric labels, so aggregate metrics by bounded dimensions such as outcome and workload class.
Quality needs operational signals too. For CRM actions, validate the schema, allowed action kinds, source-document references, and account consistency. Then sample completed jobs for human review using a stable rubric: factual support, missing commitments, incorrect owner, and unsafe stage changes. Automated validation catches malformed output. It does not prove that a summary preserved the decisive sentence in the legal call.
Set two service objectives rather than forcing a fake compromise into one number: a completion objective for eligible jobs and a quality-acceptance objective for reviewed results. Alert on sustained queue age before the completion objective is breached. Alert separately when schema-valid output loses human acceptance, because adding workers won't repair that regression.
One more practical detail: retries should follow the error class. Rate limiting may be retried with bounded backoff and jitter; invalid input should terminate immediately; invalid structured output may justify a limited regeneration attempt. Store the attempt count and final code. Otherwise, a job that consumed five attempts looks identical to one that succeeded cleanly, and capacity planning becomes guesswork.
Know when this pattern is the wrong pick
The catch is added machinery. A durable job is not suitable when a user needs one small summary during an interactive edit and can safely retry the whole request. Keep the inline path there. Bounded client-side parallelism can also be reasonable for an internal script with a tiny, disposable input set and no shared export.
Do not use automatic cross-document consolidation when each call belongs to a different account, consent boundary, or retention class. Split the jobs first. Likewise, do not publish CRM stage changes directly from generated text when policy requires human approval; export a proposed action with its evidence and let the review system own the decision.
There is no universal concurrency limit, chunk size, or polling interval. Your mileage may vary with transcript length, runtime quotas, storage contention, and the acceptable delay between a finished call and a CRM update. Load-test representative distributions, including one very long transcript among several short ones, and verify that the long item cannot erase or indefinitely delay the completed results around it.
Ship the ledger first. Optimization can follow.
Top comments (0)