Short answer: build the SaaS feature as a token-gated map/reduce pipeline: count before sending, split long text, summarize each chunk, reduce those summaries, and estimate cost before accepting the job.
For a healthtech workflow that reads supplier invoices, structured output correctness is the deciding constraint. A cheap response that quietly drops an invoice number or changes a total is not a useful response. Give users brief and detailed modes, but make both modes produce the same validated field structure.
The operational rule is crisp: no count, no request.
Govern the invoice schema as a release contract
Governance starts with a versioned output schema, not a model prompt. The naive mental model is one arrow: invoice text -> model -> summary. It works in a demo. It gives you no dependable answer when a scanned supplier invoice expands into long OCR text, when a model input limit is approached, or when a detailed mode creates much more output than a brief mode.
The production mental model has five stations: invoice text -> token count -> chunk queue -> structured chunk results -> validated final result. Put a cost estimate beside the queue decision, before model work starts. This is the observability-friendly version because every accepted job has a mode, estimated input size, chunk count, status, and final validation result. You can alert on rejected inputs or validation failures without logging invoice contents.
Infrai is one fit for that admission layer because POST /v1/ai/tokens/count and POST /v1/ai/cost/estimate expose the two checks through plain HTTP, while one key covers the platform's capabilities and usage lands on one bill. Its public, no-key discovery surface describes 295 routes across 20 modules and returns request schemas, response schemas, billing data, and runnable examples, so wiring a capability starts by reading the live contract instead of guessing fields or installing another SDK. If the invoice workflow later needs another backend capability, the team doesn't have to add another provider key rotation or another invoice reconciliation path. That reduces operational friction; it doesn't make the extraction more accurate.
Keep the boundary clear. Counting tells you how large an input is; it does not prove that extracted fields are correct.
How should a Node.js SaaS summarization API split long text and estimate token cost?
Cost control is a preflight decision. First choose the model and output mode. Then ask the token-count capability for the whole input. If it exceeds the budget you set for one request, split on stable document boundaries such as pages or invoice sections, count the candidates again, and keep shrinking only the candidates that remain too large. Run a cost estimate for the accepted chunk plan and expected output size. A brief mode should request a tight summary; a detailed mode can allow more output, while preserving the same required invoice fields.
Do not cut at an arbitrary character offset unless the input has no better boundary. An invoice line that begins in one chunk and ends in another can separate quantity from unit price. Carry a small amount of surrounding context, attach a stable chunk ID, and require each chunk to return only evidence it can see. During reduction, reject conflicting values rather than letting the final model choose silently. This is where structured correctness becomes an engineering property instead of prompt optimism.
I'm not sure what your safe per-job threshold will be; it depends on actual invoice length, selected model, and the output mode your customers use. Record estimates and actual per-call cost metadata, then set the threshold from production distributions. Don't invent it from one sample document.
One more detail: treat HTTP 429 as backpressure. Honor Retry-After when it is present and use exponential delay otherwise. Fast retry loops turn a temporary limit into a queue-wide problem.
Implement one admitted chunk in TypeScript
Implementation stays small when admission and inference are separate. The count and estimate calls belong immediately before this function. Read their current request shapes from discovery, because that surface is the contract. Once a chunk has passed admission, the chat call can use an OpenAI client against the compatible base URL.
import OpenAI from "openai";
type InvoiceResult = {
supplier_name: string | null;
invoice_number: string | null;
currency: string | null;
total: number | null;
summary: string;
};
const apiKey = process.env.INFRAI_API_KEY;
const baseURL = process.env.OPENAI_BASE_URL;
if (!apiKey || !baseURL) {
throw new Error("INFRAI_API_KEY and OPENAI_BASE_URL are required");
}
const client = new OpenAI({
apiKey,
baseURL,
maxRetries: 4,
});
export async function summarizeInvoiceChunk(
chunk: string,
mode: "brief" | "detailed",
): Promise<InvoiceResult> {
const completion = await client.chat.completions.create({
model: "glm-5.1",
messages: [
{
role: "system",
content:
"Extract only fields supported by the supplied invoice text. Use null for missing fields. Preserve facts and tone in the summary.",
},
{
role: "user",
content: `Mode: ${mode}\n\nInvoice text:\n${chunk}`,
},
],
response_format: {
type: "json_schema",
json_schema: {
name: "invoice_result",
strict: true,
schema: {
type: "object",
additionalProperties: false,
properties: {
supplier_name: { type: ["string", "null"] },
invoice_number: { type: ["string", "null"] },
currency: { type: ["string", "null"] },
total: { type: ["number", "null"] },
summary: { type: "string" },
},
required: [
"supplier_name",
"invoice_number",
"currency",
"total",
"summary",
],
},
},
},
});
const content = completion.choices[0]?.message.content;
if (!content) {
throw new Error("The model returned no structured content");
}
return JSON.parse(content) as InvoiceResult;
}
This is deliberately one chunk, not the entire orchestration hidden in a giant snippet. The caller owns counting, splitting, estimating, queue concurrency, and reduction. Validate the parsed object again at that boundary, compare repeated fields across chunks, and keep raw invoice text out of routine logs. Log IDs and outcomes instead.
Design the migration exit before choosing a gateway
Migration planning is part of the initial API choice. There isn't one universal winner. The practical comparison is about ownership and switching cost, not a price leaderboard.
| Option | Strong fit | The catch |
|---|---|---|
| OpenAI | Teams that want a direct model-provider relationship and an established client shape | Your token admission and cross-provider routing remain application concerns |
| Anthropic | Teams already standardized on Anthropic models and APIs | A provider-specific integration is a deliberate commitment |
| Google Gemini | Teams whose existing platform choice already favors Gemini | Keep your chunk and validation contracts outside provider-specific code |
| LiteLLM | Teams willing to operate an open-source gateway for provider portability | You own gateway deployment and operations |
| Unified REST platform | Teams that want self-describing capabilities, one key, and consolidated billing | Not suitable when self-hosting or direct provider control is the primary requirement |
Stick with a direct provider when its model set and operational relationship are the point. Pick LiteLLM when self-hosting and gateway control outweigh another service dependency. Choose a unified REST platform when a small team values discovery-driven integration and one consistent API across backend capabilities. Unit pricing should not drive the architecture; model prices change, while the integration boundary lasts.
Give correctness its own failure budget
Reliability needs a failure budget you can alert on. The first objection is observability: will a multi-step pipeline be harder to operate? Start with four events — admission accepted or rejected, token count, estimated cost, and schema validation passed or failed. Add chunk count and selected mode. That is enough to explain most surprises without storing sensitive source text.
Measure field agreement during reduction as well. If two chunks report different invoice totals, mark the job for review. Never average them. For alerting, a rising validation-failure ratio is more actionable than a generic “AI quality” score because it points to a contract the team can inspect.
The second objection is semantic: map/reduce can lose relationships that span chunks. Overlap helps — within reason — but it increases input volume and can produce duplicate evidence. For short invoices that fit comfortably in one request, don't chunk at all. For layouts where spatial relationships determine meaning, plain OCR text plus summarization may be the wrong pipeline; use a document extraction system designed for layout, then summarize its validated output.
Ship brief mode first. Small surface. Clear telemetry. Add detailed mode after real traffic shows where users need more context, and keep the acceptance gate identical for both.
Top comments (0)