For Node.js customer-support triage, multiple documents turn batch summarization into an accounting boundary: every async API job and exported result must still identify the tenant and the documents behind it. A batch that produces good summaries but cannot say which tenant submitted which document is not an optimization; it's an accounting problem waiting to become a support problem.
Short answer: in Node.js, treat multi-document summarization as a durable async job, attach tenant and document identity before submission, reconcile API results by identifier rather than position, and export only after completeness checks; use per-document workers when tickets need different prompts or immediate results.
This is an architecture decision record for a support system that receives tickets, groups the associated conversation documents, and gives agents a short triage summary. The model call is the easy part. The boundaries are the work.
Build the tenant ledger before the summarizer
Start with an input manifest owned by the application. Each row needs a tenant_id, a stable document_id, a source revision, and the prompt or schema version. The remote async job gets its own identifier, but that identifier is not a substitute for the application's run identifier. A provider can identify a job; only the application can define what “complete” means for a tenant's ticket queue.
The accounting record should be written before a worker submits work. It should contain the tenant, the number of documents, a redacted input digest, the selected model policy, and a status such as prepared. Once the submission response is durably recorded, the status can move to submitted. This ordering matters after a deploy: a process that dies between submission and database commit must not blindly submit the same manifest again.
I keep the cost ledger separate from the content store. It records usage fields returned by the API, request counts, retry counts, and allocation rules; it does not need to retain every private sentence in a ticket. The storage key for a result can be deterministic, for example tenant/{tenant_id}/runs/{run_id}/documents/{document_id}.json. Determinism makes a repeated export idempotent and makes a missing result visible.
The join key is the contract.
One small rule pays for itself: never join results by array position. A batch may finish documents in a different order, and a future export format may add metadata rows. Join on document_id, then verify that the returned tenant and run identifiers match the local manifest.
Can a Node.js API prove that every document belongs in the export?
The request path should do validation and registration, not wait for summarization. A coordinator submits the manifest, stores the job identifier, and returns a local run identifier to the caller. A separate poller reads that local record, asks for status at a bounded cadence, and records each observed state. A reconciler then compares the result set with the manifest before an export is marked ready.
That gives the system three different clocks: the HTTP request clock, the remote job clock, and the retention clock for exported results. Confusing them causes familiar failures. A request timeout does not prove remote work stopped. A terminal job state does not prove every expected document was returned. A successful download does not prove the file belongs to the requested tenant.
The critical path is small enough to express as a generic Node.js adapter. The adapter deliberately does not pretend that a particular service has a universal path or payload shape; the concrete transport is supplied by the capability contract selected at deployment time.
async def run_triage_batch(tenant_id, run_id, documents, api):
manifest = [
{
"tenant_id": tenant_id,
"document_id": document["id"],
"revision": document["revision"],
}
for document in documents
]
await api.record_run({
"tenant_id": tenant_id,
"run_id": run_id,
"document_count": len(manifest),
"status": "prepared",
})
job = await api.submit_batch({
"run_id": run_id,
"items": manifest,
"instruction": "Summarize this support ticket for triage.",
})
await api.record_submission({
"tenant_id": tenant_id,
"run_id": run_id,
"job_id": job["id"],
"status": "submitted",
})
return {"run_id": run_id, "job_id": job["id"]}
The api object is an intentionally boring seam. In production it should implement authentication, timeout handling, retry policy, and the service-specific request shape. The caller should never place a secret in a ticket, log a full prompt, or use a tenant-provided URL as an object-storage key.
Polling needs the same discipline as submission. Retry a 429 with bounded exponential backoff and jitter, honor a server-provided delay when the contract defines one, and cap the total polling budget. Do not turn every transient response into a new job. I use the local run_id as the idempotency key for a submission; your mileage may vary if the chosen API defines idempotency differently, and that difference belongs in the adapter contract, not in scattered route handlers.
Where should tenant cost and export evidence live?
The dangerous state is not always “failed.” It is “looks finished.” A useful reconciliation report names the failure boundary instead of hiding it behind a green status.
| Failure boundary | Evidence to retain | Safe action |
|---|---|---|
| Request ended before submission was committed | Local run state and submission key | Resume from the durable state; do not infer success from a timeout |
| Remote job is terminal but an item is absent | Manifest IDs and result IDs | Keep the export pending and raise a reconciliation event |
| Result has the wrong tenant or revision | Tenant, document, and revision fields | Quarantine the item; never attach it to the ticket |
| Polling is rate-limited | Status history and retry timestamps | Back off, then continue observing the same job |
| Export download is repeated | Deterministic object key and content digest | Overwrite the same artifact or reject a different digest |
| Usage data is incomplete | Raw usage response and ledger status | Mark cost attribution incomplete instead of inventing a number |
The last row is where per-tenant cost visibility becomes an engineering property. If the API reports usage only at job level, the application needs an explicit allocation policy, such as equal allocation by input token count or a documented shared-cost bucket. That estimate must be labeled as an estimate. It is not acceptable to copy a whole job's usage onto the largest tenant because that makes dashboards look decisive. I would store both the raw job-level usage and the allocation inputs: the document count, the measured input-size basis, the policy version, and the list of tenants sharing the run. That lets finance or an operator recompute the allocation after a policy change without asking the summarizer to run again, while keeping the exported support artifact independent from the ledger calculation. If one document is retried, the ledger must show whether the retry replaced the first attempt, added billable work, or was absorbed under the API's retry semantics; otherwise a weekly tenant report can be numerically tidy and still be impossible to defend.
I also log state transitions, not private ticket bodies: prepared, submitted, running, reconciling, exported, and blocked. A short error code such as 429 is useful operational evidence; a copied customer conversation is usually not. Retention and deletion rules should be applied to source text, summaries, usage records, and exported files independently. They are different data classes even when one request created them.
No guessing.
Is a batch job the right boundary for each support document?
Batching is unsuitable when a support agent needs one ticket summarized immediately, when tickets require materially different instructions, or when a single item must trigger its own tool calls and priority policy. A queue with one durable message per document is a better fit there. It gives the team independent retry, priority, and dead-letter controls, at the cost of more coordination and more opportunities to misattribute usage.
Batching is a good fit when the support operation can tolerate asynchronous completion, the instruction and output schema are stable, and the unit of review is a collection rather than a single ticket. The catch is that a batch boundary can hide per-document latency and can make partial completion harder to explain to agents. A UI should expose “12 of 15 reconciled,” not collapse that state into “done.”
The decision should be tested with a replay harness containing redacted tickets from several tenants. Measure missing-result detection, duplicate submission behavior, export integrity, and ledger completeness before comparing summary quality. A tokenizer can help estimate input size before submission, but it does not tell you whether a support summary is faithful; that needs a labeled evaluation set and a human review rule.
For evidence and repeatability, keep the manifest, schema version, status history, usage response, and exported-file digest together. If an operator cannot reconstruct the relationship between a tenant, a document revision, a remote job, and an export, the pipeline is not auditable yet.
References
- OpenAI tiktoken tokenizer library: https://github.com/openai/tiktoken
- ElevenLabs documentation: https://elevenlabs.io/docs
Top comments (0)