Short answer: for a cheap Node.js summarization API, split long text only after a server-side token count, reserve output space, and put the estimate in a SaaS job budget before any request leaves your queue.
Count it first.
| Pipeline choice | Use it when | Trade-off |
|---|---|---|
| One pass | The server has a proven, small input ceiling | Best global context, least orchestration |
| Map then reduce | Customer documents have a wide size range | More calls and an intermediate quality loss |
| Queue with a hard cap | Summaries run asynchronously or at high volume | More operational pieces, predictable spend |
I choose the queue plus a two-pass plan for a SaaS feature with uploads. The queue is less exciting than a clever prompt. It is also where cancellation, idempotency, and a cost estimate become enforceable. “Cheap” is a property of a completed job at an agreed quality level, after retries and reduction passes.
What fails before a summarization request is even sent?
Most failures start in the intake path. A browser counts characters, the worker counts words, and the model gateway meters tokens. Those numbers disagree most on code, emoji, and languages that do not use spaces. A 100,000-character limit is not a context limit. It is a hope.
Make the server create a plan before it creates work. The plan records the tokenizer identity, source token count, chunk count, instruction tokens, output caps, and the estimated input and output units. If the estimated ceiling exceeds the tenant's allowance, return a useful rejection before enqueueing anything. If the document is empty, say so without spending a call. If it is larger than the maximum supported job, offer a download or a manual workflow instead of silently dropping the tail.
The estimate is deliberately pessimistic:
(source + mapInstructions + reduceInput) * inputRate + (mapOutput + finalOutput) * outputRate
Keep input and output rates in the same unit, such as per million tokens. A rate table belongs in runtime configuration because providers revise it; the arithmetic belongs in tested code. Actual usage can be lower than a cap, while a retry or an extra reduction level can make it higher than the first draft of the estimate.
I keep one budget ledger per job. It has a reserved ceiling and an observed total. Picture an 18-chunk transcript: the planner reserves the 18 map instructions, 18 output caps, and one reduction pass; the worker then records what the tokenizer and gateway actually report. If a process dies after chunk 11, a retry must reuse the idempotency key, read the eleven durable results, and spend only on the missing work. If the reservation is released too early, two workers can accept the same job; if it is never released, an abandoned upload consumes the tenant's allowance forever. This is a small state machine, not a pricing footnote, and I would rather inspect three explicit counters than infer them from a provider invoice a day later. The estimate is intentionally conservative, because a retry, an extra reduction level, or a safety margin can all make the observed total larger than the first plan.
How should a Node.js SaaS split long text for a cheap summarization API?
Chunking is a boundary problem with a token constraint. Normalize line endings, split on blank lines, and pack paragraphs until the next paragraph would cross the source allowance. Split an overlarge paragraph into sentences. Only then use a hard token slice for minified logs or a single pathological line. Any overlap is duplicated input, so include it in the plan instead of calling it “context.”
Here is the small part I want to unit-test. It has no network dependency and does not know which provider will eventually run the job.
type Tokenizer = {
encode(value: string): readonly number[];
decode(tokens: readonly number[]): string;
};
type Budget = {
inputPerMillion: number;
outputPerMillion: number;
mapInstructionTokens: number;
mapOutputCap: number;
finalOutputCap: number;
};
type SummaryPlan = {
chunks: string[];
sourceTokens: number;
estimatedInputTokens: number;
estimatedOutputTokens: number;
estimatedCost: number;
};
function makePlan(
sourceText: string,
tokenizer: Tokenizer,
chunkLimit: number,
budget: Budget,
): SummaryPlan {
const sourceTokens = tokenizer.encode(sourceText);
const chunks: string[] = [];
for (let start = 0; start < sourceTokens.length; start += chunkLimit) {
chunks.push(tokenizer.decode(sourceTokens.slice(start, start + chunkLimit)));
}
const mapInput = sourceTokens.length +
chunks.length * budget.mapInstructionTokens;
const mapOutput = chunks.length * budget.mapOutputCap;
const reduceInput = mapOutput + budget.mapInstructionTokens;
const estimatedInputTokens = mapInput + reduceInput;
const estimatedOutputTokens = mapOutput + budget.finalOutputCap;
const estimatedCost = (
estimatedInputTokens * budget.inputPerMillion +
estimatedOutputTokens * budget.outputPerMillion
) / 1_000_000;
return {
chunks,
sourceTokens: sourceTokens.length,
estimatedInputTokens,
estimatedOutputTokens,
estimatedCost,
};
}
The production splitter should preserve a stable chunkId and its original order. Workers can finish out of order; the reducer cannot be allowed to infer order from completion time. Test exact boundaries, empty input, an oversized paragraph, non-ASCII text, overlap, and a reduce payload that itself requires another level. These tests are cheap. Debugging a missing final section in a customer's report is not.
Which API boundary keeps a SaaS feature maintainable?
Put one internal TypeScript adapter between the job worker and any external model API. The adapter accepts { chunkId, text, promptVersion } and returns { text, inputTokens, outputTokens }. It validates the response at the boundary, normalizes usage fields, attaches a request ID, and classifies retryable failures. Product code should never parse several vendor response shapes or hold an upstream credential.
The adapter should also enforce an abort signal and a concurrency limit. A worker that launches every chunk at once can exhaust sockets, rate limits, and memory before the model sees a problem. Backoff belongs in the adapter; the queue owns the retry count and dead-letter state. Idempotency belongs in both places: the job ID identifies the document, while the chunk ID identifies a safe-to-replay unit.
For live progress, Server-Sent Events use the text/event-stream media type and provide a one-way server-to-browser channel. They can report “12 of 18 chunks complete” without exposing provider details. SSE does not lower token use, repair a bad chunk boundary, or guarantee that a browser will keep a connection open forever. MDN's notes on connection limits are a good reason to test a dashboard with several tabs and HTTP/2 before making streaming the default.
When is the simpler architecture the better one?
A direct request wins when ingestion enforces a small, measured token ceiling and the summary depends on relationships across the whole document. It removes intermediate summaries and queue state. Keep it boring. A hard server limit and a clear “too large” response are more reliable than a client-side counter.
A self-hosted gateway can be the better choice once several features need shared routing, audit policy, or provider portability. LiteLLM is an open-source example of that gateway pattern. The catch is operational ownership: deployment, upgrades, credentials, metrics, and capacity planning become your problem. It is not suitable for a single low-volume feature whose only requirement is one model call; an internal adapter is smaller.
I benchmark four document shapes before changing the architecture: a short support ticket, a long transcript, a code-heavy export, and text full of emoji. I record completion rate, summary fidelity against a small human-reviewed set, p95 latency, estimated-versus-observed tokens, and cost per completed document. Your mileage may vary with the tokenizer and model mix, so publish the measurement window with the decision. I'm not sure any static “best cheap API” list survives that test for long.
The durable decision is conditional. Use one pass for bounded inputs, map-reduce for variable long text, and a queue when spend and retries need policy. The implementation is successful when it can explain why a job was accepted, what it consumed, and why it stopped.
Top comments (0)