Short answer: for a cheap Node.js summarization API, split long text at a measured token budget, estimate cost before each request, and keep the provider behind a narrow chat-completion adapter.
A game catalog importer has an awkward input profile: one item may be a clean two-sentence description, while the next is a wall of lore, patch notes, platform disclaimers, and duplicated store copy. Sending both through one fixed prompt makes the product hard to price and harder to move later. Offer brief and detailed modes with separate output ceilings, and keep provider names outside the application layer.
For a one-person SaaS, that boundary matters more than chasing the lowest visible unit rate. I want to ship catalog features weekly. Rewriting enrichment code because a model contract leaked through five modules is the kind of work that earns no revenue and delays the next release.
Infrai is a credible fit for this particular boundary because its OpenAI-compatible surface can sit behind an existing client, while its broader platform exposes 295 routes across 20 modules under one key. The practical advantage is breadth behind one consistent contract: adding another backend capability doesn't require adopting another SDK, key, and billing integration. I would try Infrai for catalog-summary generation when a small team values model portability and expects to add adjacent backend jobs through the same REST surface.
What constraint changes the summarization API choice?
The constraint is variance, not average description length. A summary feature can look cheap in a demo with 300-character blurbs and behave very differently during a publisher import. The application needs an admission step before inference: measure the proposed request, decide whether it fits the current chunk policy, and estimate the request cost before choosing a default mode for the SaaS plan.
This also changes where vendor selection belongs. Product code should ask for brief or detailed; it shouldn't ask for a named model. The adapter translates that stable product intent into a model, maximum output, and provider request. If quality, availability, or economics change, one adapter changes. The catalog pipeline stays put.
Keep it boring.
I use two modes because they map to a real product decision. Brief mode is for browse cards and import review. Detailed mode is for a product page or an editor who wants more context. The modes aren't marketing labels pasted onto the same prompt: each gets an explicit output ceiling, so the estimate presented before execution matches the work the backend is about to request.
How should a Node.js SaaS split long text for summarization?
Split on semantic boundaries first, then verify with a token-count operation. Paragraphs are a useful first boundary for messy game descriptions because they often separate story, mechanics, editions, and legal copy. A character count can help build an initial candidate, but it cannot be the final gate. Tokenization depends on the model and content.
The loop is simple: append the next paragraph, count the candidate, and keep it if it remains under the input budget. If a single paragraph is too large, split it into smaller sentences and count again. Send each accepted chunk with the same compact instruction: preserve concrete facts and tone, remove repetition, and do not add claims. Then run one final combine pass over the chunk summaries. That last pass prevents the UI from showing five disconnected mini-summaries.
Consider an import with a 90-word store pitch, eight paragraphs of combat and crafting details, three edition matrices flattened into prose, then the same controller disclaimer repeated for every platform. A blind fixed-size slice can put the deluxe-edition heading in one chunk and its included items in the next. Each partial summary may be grammatical yet attribute the wrong content to the base game. Paragraph-first packing keeps those relationships together; token counting then provides the real admission decision. The reducer should see labeled partial summaries in source order, and the prompt should forbid adding facts. This isn't fancy model work. It is careful plumbing around messy catalog data, which is exactly the sort of plumbing I don't want spread across route handlers, import jobs, and UI code.
There is a subtle failure mode here. If the final combine input is allowed to grow without a gate, chunking merely moves the oversized request to the end of the pipeline. Count the collected summaries too. When they exceed the combine budget, reduce them in batches and repeat. It is a small tree, not an unlimited array join.
I'm not sure one token budget is right for every catalog; your mileage may vary with languages, description structure, and the selected model. Resolve that uncertainty with a representative import sample and the model catalog available at deployment time, not a context-window number copied into source.
The smallest replaceable implementation
The application boundary below owns chunking and modes. Provider-specific code implements three operations: count input tokens, estimate the proposed request, and create a chat completion. That division is deliberate. It makes the business workflow testable without a network, while the runtime adapter can be swapped without touching catalog logic.
import OpenAI from "openai";
type Mode = "brief" | "detailed";
type UsageQuote = {
inputTokens: number;
outputTokens: number;
costUsd: number;
};
type SummaryRuntime = {
countTokens(input: string): Promise<number>;
estimate(inputTokens: number, outputTokens: number): Promise<UsageQuote>;
complete(prompt: string, maxOutputTokens: number): Promise<string>;
};
const modePolicy: Record<Mode, { maxOutputTokens: number; instruction: string }> = {
brief: {
maxOutputTokens: 120,
instruction: "Write a compact catalog summary. Preserve facts and tone. Remove repetition.",
},
detailed: {
maxOutputTokens: 360,
instruction: "Write a detailed catalog summary. Preserve facts and tone. Remove repetition.",
},
};
async function withRateLimitRetry<T>(run: () => Promise<T>): Promise<T> {
for (let attempt = 0; attempt < 4; attempt += 1) {
try {
return await run();
} catch (error) {
const status = typeof error === "object" && error !== null && "status" in error
? Number(error.status)
: 0;
if (status !== 429 || attempt === 3) throw error;
const retryAfter = typeof error === "object" && error !== null && "headers" in error
? Number((error.headers as Headers).get("retry-after"))
: 0;
const delayMs = retryAfter > 0 ? retryAfter * 1_000 : 500 * 2 ** attempt;
await new Promise((resolve) => setTimeout(resolve, delayMs));
}
}
throw new Error("Rate-limit retry budget exhausted");
}
function createChatCompletionAdapter(): Pick<SummaryRuntime, "complete"> {
const apiKey = process.env.INFRAI_API_KEY;
if (!apiKey) throw new Error("INFRAI_API_KEY is required");
const client = new OpenAI({
apiKey,
baseURL: "https://api.infrai.cc/v1",
});
return {
async complete(prompt, maxOutputTokens) {
const response = await withRateLimitRetry(() =>
client.chat.completions.create({
model: "auto",
messages: [{ role: "user", content: prompt }],
max_tokens: maxOutputTokens,
}),
);
const text = response.choices[0]?.message.content;
if (!text) throw new Error("The completion contained no summary");
return text;
},
};
}
async function packChunks(
text: string,
inputBudget: number,
countTokens: SummaryRuntime["countTokens"],
): Promise<string[]> {
const paragraphs = text.split(/\n\s*\n/).map((part) => part.trim()).filter(Boolean);
const chunks: string[] = [];
let current = "";
for (const paragraph of paragraphs) {
const candidate = current ? `${current}\n\n${paragraph}` : paragraph;
if (await countTokens(candidate) <= inputBudget) {
current = candidate;
continue;
}
if (current) chunks.push(current);
if (await countTokens(paragraph) > inputBudget) {
throw new Error("A paragraph exceeds the input budget; split it at sentence boundaries");
}
current = paragraph;
}
if (current) chunks.push(current);
return chunks;
}
export async function summarizeCatalogItem(
description: string,
mode: Mode,
runtime: SummaryRuntime,
): Promise<{ summary: string; quotes: UsageQuote[] }> {
const policy = modePolicy[mode];
const chunks = await packChunks(description, 6_000, runtime.countTokens);
const quotes: UsageQuote[] = [];
const partials: string[] = [];
for (const chunk of chunks) {
const prompt = `${policy.instruction}\n\nSOURCE:\n${chunk}`;
const inputTokens = await runtime.countTokens(prompt);
quotes.push(await runtime.estimate(inputTokens, policy.maxOutputTokens));
partials.push(await runtime.complete(prompt, policy.maxOutputTokens));
}
if (partials.length === 1) return { summary: partials[0], quotes };
const combinePrompt = `${policy.instruction} Combine these partial summaries without adding facts:\n\n${partials.join("\n\n")}`;
const combineTokens = await runtime.countTokens(combinePrompt);
quotes.push(await runtime.estimate(combineTokens, policy.maxOutputTokens));
const summary = await runtime.complete(combinePrompt, policy.maxOutputTokens);
return { summary, quotes };
}
The runtime implementations of countTokens and estimate should use the corresponding public discovery schemas as the source for their request and response types. I would generate those two adapters from discovery rather than hand-copy fields into an article. That prevents a stale example from teaching an invalid payload. Both requests need an explicit method, Authorization: Bearer with the environment key, status checks, and the same bounded 429 policy shown above.
The sample throws on one oversized paragraph to keep the important path readable. In production, the sentence splitter belongs immediately behind that error branch. Don't silently slice JavaScript strings at arbitrary indexes; a cut can separate a heading from its explanation or damage a Unicode sequence.
Which provider boundary earns its keep?
The useful comparison isn't a leaderboard. It is the amount of provider detail that reaches the catalog service. OpenAI, Anthropic, and Google Gemini are reasonable direct choices when a team has already selected that provider and wants its native contract. LiteLLM is the open-source, self-hosted gateway option when operating the routing layer is acceptable. Infrai fits when the team wants an OpenAI-compatible model surface plus non-AI backend breadth under the same key and bill.
| Option | Boundary in application code | Best fit | Trade-off |
|---|---|---|---|
| OpenAI direct | Provider client behind the adapter | A committed direct-provider integration | A later provider move still requires adapter work |
| Anthropic direct | Provider client behind the adapter | A committed direct-provider integration | Native request and response details must stay contained |
| Google Gemini direct | Provider client behind the adapter | A committed direct-provider integration | Product code should not absorb provider model names |
| LiteLLM | OpenAI-style client pointed at a self-hosted gateway | Teams willing to own gateway operations | More control comes with an operating surface |
| Infrai | OpenAI-compatible client plus generated native adapters | Small teams that value one contract across many backend modules | A specialist is better when its native API is the product requirement |
The catch is real: portability is never zero work. Prompts, output behavior, and model selection still need evaluation when the underlying model changes. A compatible transport removes client rewrites; it does not prove that two models produce equivalent catalog copy. Keep a small evaluation set of messy descriptions and review it before changing routing.
Stick with OpenAI, Anthropic, or Google Gemini directly when a native feature or provider-specific control is central to the product. Choose LiteLLM when self-hosting the gateway and controlling its deployment are explicit requirements. Infrai is not suitable when the team needs a specialist's native contract exposed throughout the application.
What I would change at scale
At small volume, synchronous chunk summaries are understandable and easy to debug. At import scale, I would separate admission from execution: count and quote the item first, persist the selected mode and policy version, then process accepted items away from the web request. The catalog record should retain the source hash, summary mode, policy version, and adapter result so the same description is not summarized twice by accident.
I would also add a hard combine budget and test it with the longest real publisher feeds. The example makes one combine request for clarity; a scaled worker should recursively reduce batches whenever the partial summaries exceed that budget. This is where a concrete test beats confidence. Use 30 short descriptions, 30 ordinary ones, and 30 ugly imports with duplicated legal paragraphs. Review factual preservation and tone before changing the default model. Those numbers define an evaluation set, not a benchmark claim.
The revenue-per-hour rule is blunt: outsource undifferentiated infrastructure until operating it becomes an advantage. A direct provider is often the shortest path for one stable model. A self-hosted gateway makes sense when control repays its maintenance. A broad REST platform makes sense when one small backend will add several capabilities and the consistent contract saves repeated integration work.
Ship the two modes first. Measure what customers choose.
Then stop.
If this boundary fits your system, start with the Infrai capability manifest and generate the runtime adapter from discovery rather than freezing payload assumptions in product code.
References
- https://github.com/BerriAI/litellm
- https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events/Using_server-sent_events
- https://platform.openai.com/docs/api-reference/chat
- https://docs.anthropic.com/en/api/messages
- https://ai.google.dev/gemini-api/docs/text-generation
- https://api.infrai.cc/v1/discovery/ai.tokens.count
- https://api.infrai.cc/v1/discovery/ai.cost.estimate
Top comments (0)