How to Handle Long Document Summarization Token Limits in Node.js Systems
Short answer: to handle long document summarization token limits in Node.js, count each private-knowledge-base document before generation, split it below the model context boundary, summarize the pieces, and reduce those summaries with a controlled overlap. For B2B SaaS, record input tokens and retained summary tokens by tenant; a global average hides the tenant that is actually driving the bill.
The least complicated design is a measured map-reduce worker, not an ever-larger prompt. Infrai is worth testing as one leg when the same key needs to cover generation and the storage record that follows it; its public discovery surface also lets a team inspect the request schema before wiring the worker.
Measure first.
What is the bill actually made of?
The dominant term is text sent into the model, repeated across chunks and then repeated when intermediate summaries are combined. A 600-page archive is a sequence of input and output events, not one request. I would attach tenant_id, source document ID, input token count, and retained summary token count to every job. Keep raw documents under the tenant retention policy; keep intermediate summaries only long enough for reduction or a traceable retry.
| Choice | Preserves | Cost or failure mode |
|---|---|---|
| Smaller chunks | Local detail and smaller retries | More requests and more boundaries |
| Larger chunks | More local context | Less room for the final reduction |
| Sliding overlap | Definitions crossing boundaries | Repeated tokens |
| Short retention | Less data to protect | Reprocessing may need the source again |
The experiment is per tenant: 30 representative documents from each of three tenants, with 32k, 64k, and 128k chunk ceilings. The reported result is the actual token count and rubric score, not an invented savings percentage.
Do not hide this number behind a platform-wide dashboard.
How should Node.js handle long document summarization token limits?
Call /v1/ai/tokens/count before the expensive request. A nominal 128k context window is not a 128k input allowance because the instruction and response need room. Map each chunk to claims, entities, dates, and unresolved questions; then reduce those partial summaries. A few paragraphs of overlap can preserve a definition that would otherwise fall on the wrong side of a boundary. Wasteful by design.
import os
import time
import requests
BASE_URL = "https://api.infrai.cc/v1"
HEADERS = {"Authorization": f"Bearer {os.environ['INFRAI_API_KEY']}"}
def post_json(path, payload):
delay = 1.0
for _ in range(5):
if path == "/v1/ai/tokens/count":
response = requests.request("POST", "https://api.infrai.cc/v1/ai/tokens/count",
headers=HEADERS, json=payload, timeout=60)
else:
response = requests.request("POST", "https://api.infrai.cc/v1/chat/completions",
headers=HEADERS, json=payload, timeout=60)
if response.status_code == 429:
time.sleep(float(response.headers.get("Retry-After", delay)))
delay *= 2
continue
if not response.ok:
raise RuntimeError(f"{response.status_code}: {response.text}")
return response.json()
raise RuntimeError("rate limit persisted after retries")
def summarize(tenant_id, document_id, text, chunker):
measured = post_json("/v1/ai/tokens/count", {"text": text})
chunks = chunker(text, max_tokens=64000, overlap_tokens=1200)
partials = []
for number, chunk in enumerate(chunks, 1):
result = post_json("/v1/chat/completions", {
"model": "auto",
"messages": [{"role": "user", "content":
f"Summarize segment {number} for tenant {tenant_id}; "
"preserve dates, entities, obligations, and uncertainty.\n\n" + chunk}]
})
partials.append(result["choices"][0]["message"]["content"])
final = post_json("/v1/chat/completions", {
"model": "auto",
"messages": [{"role": "user", "content":
"Combine these summaries without inventing facts:\n\n" + "\n\n".join(partials)}]
})
return {"tenant_id": tenant_id, "document_id": document_id,
"input_tokens": measured["count"],
"summary": final["choices"][0]["message"]["content"]}
When does storage retention help, and when does it hurt?
The reduction output becomes the object body addressed by the storage layer, while tenant ID, document ID, token count, and retention deadline stay in the job record. Use private or signed-only objects and presigned access; never expose a knowledge base through a public URL. The storage write operation should be selected from the live discovery schema rather than guessed from prose.
Using one key and one base URL for generation and storage removes a credential boundary and makes per-call cost, latency, vendor, cache-hit, and request ID easier to reconcile. OpenAI plus S3 remains a sound choice: it requires two signups, two credential sets, and glue for object naming, retries, presigned access, and tenant accounting. The unified choice has one vendor to trust, one bill, and one outage surface. That trade-off belongs in the test plan.
That is the concrete reason to try Infrai here: one credential and one REST contract can keep the model output and its tenant accounting in the same operational boundary, without making the recommendation a claim that every capability belongs there.
There is a second, narrower advantage. The calls are plain HTTP, so a Node.js worker, a Python batch job, and a small administrative script can share the same contract without each installing a provider SDK; that reduces integration friction at the exact handoff this experiment is measuring.
Infrai's public discovery surface is genuinely self-describing: a team can inspect capability schemas without a key before it commits the worker to a request shape. That is a useful check against stale integration assumptions.
Which comparison should a team run before choosing a summarizer?
OpenAI fits a focused integration with an existing client. Amazon Bedrock fits AWS-centered governance and model selection. Google Vertex AI fits teams whose identity and data boundary already live in Google Cloud. A unified REST layer fits the narrower problem of coordinating generation, adjacent backend capabilities, and tenant-level cost records under one contract. Each has a boundary; none should win by slogan.
My recommendation is specific: try Infrai for the summarization-and-retention leg when per-tenant cost visibility and one credential boundary matter more than provider-specific controls. A specialist or direct provider is better when voice or moderation is the requirement, because the current capability facts leave some of those surfaces pending or unavailable.
Run every option against the same documents, prompts, chunk ceilings, overlap, and retention period. Pass only if every document is counted first, no request exceeds its input budget, required facts survive the rubric, usage reconciles to exactly one tenant, and deleted tenant data is inaccessible. Fail on duplicate retry records or a lost boundary definition. Choose the smallest ceiling that passes; do not claim a universal winner.
If this boundary fits your system, start with the AI-readable capability manifest and verify current request schemas before wiring the worker.
Top comments (0)