Short answer: treat summarization as a metered data pipeline, not a single API call. Normalize the document, count tokens with the tokenizer for the selected model, split on semantic boundaries, and reject a job when its worst-case input and output cost exceeds the user's budget.
That decision matters more than finding a provider with the smallest headline rate. A SaaS feature needs predictable latency, a useful failure message, and a way to resume after a worker restart. The model is one replaceable stage.
What should a Node.js SaaS feature measure before splitting long text?
Start with a model profile stored in configuration: context limit, reserved output limit, tokenizer identifier, and input/output rates. A character count is fine for displaying rough progress. It is not an admission check. Token boundaries vary by tokenizer, and the bill includes generated tokens as well as source tokens.
For a map-reduce job, estimate every level. If 18 chunks each reserve 180 output tokens, the first pass reserves 3,240 output tokens. Those summaries then become input to the reduction pass, which needs its own output allowance. The ceiling will be higher than many real invoices; that is intentional. A preflight check prevents a surprise, it does not predict the invoice to the cent.
Keep the estimate with the job record. Store the profile version, estimated tokens, actual usage returned by the API, and the remaining budget. That makes a later rate change auditable instead of turning billing support into archaeology.
Measure twice.
Here is the failure pattern I design around: a user pastes a 90-page export, the web request accepts it, and a worker starts eight calls before anyone notices that the output reservation is larger than the tenant's allowance. The first seven calls can succeed, which makes a later rejection confusing and leaves partial results to clean up. A planner at the boundary computes the complete map and reduction ceiling first, records the decision, and only then queues work. If the estimate is over budget, the response can offer a smaller output cap or ask the user to narrow the source. If it is under budget, each worker still enforces a per-call timeout and writes an idempotency key before retrying. That sequence turns an expensive surprise into an explicit product choice.
Put a copyable planner in front of the transport
The following TypeScript is the part I want to test without making a network request. The caller supplies a tokenizer and a model profile, so a migration does not leak into every request handler.
type Tokenizer = { count(text: string): number };
type ModelProfile = {
contextTokens: number;
outputTokens: number;
inputUsdPerMillion: number;
outputUsdPerMillion: number;
};
type SummaryPlan = {
chunks: string[];
estimatedInputTokens: number;
estimatedOutputTokens: number;
estimatedUsdCeiling: number;
};
function units(text: string): string[] {
return text
.replace(/\r\n/g, "\n")
.split(/\n{2,}|(?<=[.!?])\s+/)
.map((part) => part.trim())
.filter(Boolean);
}
function pack(unitsToPack: string[], limit: number, tokenizer: Tokenizer): string[] {
const chunks: string[] = [];
let current = "";
for (const unit of unitsToPack) {
if (tokenizer.count(unit) > limit) {
throw new Error("One paragraph is larger than the configured chunk budget");
}
const candidate = current ? `${current}\n\n${unit}` : unit;
if (tokenizer.count(candidate) <= limit) {
current = candidate;
} else {
chunks.push(current);
current = unit;
}
}
if (current) chunks.push(current);
return chunks;
}
export function planSummary(
text: string,
instructionTokens: number,
profile: ModelProfile,
tokenizer: Tokenizer,
): SummaryPlan {
const chunkLimit = profile.contextTokens - instructionTokens - profile.outputTokens;
if (chunkLimit <= 0) throw new Error("The model profile leaves no input space");
const chunks = pack(units(text), chunkLimit, tokenizer);
const sourceTokens = chunks.reduce((sum, chunk) => sum + tokenizer.count(chunk), 0);
const estimatedInputTokens = sourceTokens + chunks.length * instructionTokens;
const estimatedOutputTokens = chunks.length * profile.outputTokens;
const estimatedUsdCeiling =
(estimatedInputTokens * profile.inputUsdPerMillion +
estimatedOutputTokens * profile.outputUsdPerMillion) /
1_000_000;
return { chunks, estimatedInputTokens, estimatedOutputTokens, estimatedUsdCeiling };
}
There is one deliberate hard stop: an individual unit cannot fit. Reject it.
In production I fall back from paragraphs to sentences, then to smaller text units, while preserving a small margin for the instruction and output. Never split UTF-8 bytes or cut in the middle of a sentence just to fill a quota. A document with one enormous pasted table is a good example: the planner should report that unit and ask the caller to preprocess it, rather than silently slicing rows into unrelated fragments and producing a confident summary that cannot be traced back to the source.
How do chunking, token count, and cost estimates survive real traffic?
Give each chunk a stable job ID and sequence number. A worker can write a completed chunk result before asking for the next one; a retry then checks for that record and avoids charging twice. Limit concurrency per tenant, because ten simultaneous documents can multiply the same budget error.
The map prompt should request a fixed shape, such as JSON with a short summary and cited source offsets. The reduce prompt should consume only those fields. This bounds intermediate growth and makes malformed output a validation error instead of an invisible context expansion.
Streaming is a presentation concern. Send progress events from the worker to the browser over Server-Sent Events, while the authoritative state remains in your database. SSE reconnects are ordinary HTTP requests, so include the last completed sequence in the event stream and replay missing progress. A final event should carry the stored summary ID, not a giant blob.
Three numbers belong on every request log: planned input tokens, planned output tokens, and provider-reported usage. Add queue wait, model latency, retry count, and chunk count. Those fields reveal whether a “cheap” design is actually losing money to retries or to a reduction pass that keeps growing.
Trade-offs worth making explicit
| Choice | Useful when | Cost or risk |
|---|---|---|
| Larger chunks | Fewer round trips and better local context | Higher tail latency and a larger failure unit |
| Smaller chunks | Parallel workers and easy retries | More boundary loss and a larger reduction bill |
| Map-reduce | Documents exceed one context window | Extra prompts, output tokens, and ordering work |
| Single pass | Short, bounded documents | One oversized request can consume the whole allowance |
| Hosted API | You need to ship before operating models | Vendor limits, variable rates, and data residency review |
| Self-hosted gateway | Stable routing and policy control matter | Capacity planning, upgrades, and on-call load |
The catch is that this pipeline is not suitable when the product needs exact cross-document reasoning in one context, strict real-time latency, or a model feature unavailable behind your chosen interface. Stick with a single bounded call for short text; choose a retrieval or batch system when the job is really search or offline analytics. Your mileage may vary across languages and document formats, so replay a representative corpus before setting the margin.
Before enabling the feature, pin a tokenizer and profile version, then test an empty document, one paragraph over the limit, and a document that requires two reduction levels. Set a tenant budget and return a 4xx-style validation response before any model call when the ceiling is too high. During rollout, compare planned versus actual usage and adjust the output reservation from evidence, not optimism.
Keep prompts, profile changes, and chunking rules versioned. Redact source text from ordinary logs, retain enough metadata to reproduce a plan, and put a timeout around each external call. I’m not sure a universal “safe” chunk margin exists; the right value depends on your language mix and output contract. Measure it.
The result is pleasantly unglamorous: a request either fits the budget and has a restartable path, or it is declined with a reason the user can understand.
Top comments (0)