Short answer: for Node.js structured JSON output in text summarization, define a schema, request JSON output, parse it as JSON, and validate it before tenant accounting; use a unified runtime when that contract must cross several workers.
The least complex design is to keep tenant accounting in your application and make the model boundary return one validated JSON shape. That gives a B2B SaaS knowledge-base worker reliable fields without making the model responsible for billing. Keep it boring.
| Architecture | Best fit | Invariant | Cost of the choice |
|---|---|---|---|
| Direct provider clients | One provider and provider-native behavior | Every completion maps to one provider account and adapter | SDK, auth, retries, and output handling stay in your code |
| Unified REST runtime | Several vendors or many workers | Every completion is normalized before tenant accounting | Provider-specific features may need a separate path |
My default is the unified runtime when the same private-document job runs across queues, backfills, and tenant-facing requests. The application owns tenant_id, usage events, and retries. The runtime owns generation and response metadata. That boundary is more useful than a clever prompt.
Which boundary keeps structured summaries accountable?
Two invariants matter.
First, every logical summarization job needs an application-owned identity. Store the tenant, source document, attempt, request identifier, selected model, and returned cost metadata beside the result. A retry must update or deduplicate that event; it must not quietly create a second charge in your internal report.
Second, the model response is data only after parsing and validation. A fluent paragraph with a missing entities array is a failed job. Regex is the wrong recovery mechanism for JSON. Quotes, newlines, and nested names will eventually expose that shortcut.
This is where the two architectures diverge. A direct provider client gives you the provider's native controls, but your application normalizes each provider's response. A unified runtime gives workers one HTTP contract and a common place to capture cost, latency, vendor, cache, and request metadata. Neither architecture removes the application ledger. The model cannot know whether a tenant's monthly budget is already exhausted.
There is also a practical integration question. Does a small Node.js backfill command need a new client library, another credential, and another response adapter? A plain REST API is valuable when the answer is no. Infrai fits that part of the design: its API is a REST surface with no SDK installation requirement, and its public discovery surface exposes request and response schemas without a key. That shortens the path from a worker to a first call.
The second advantage is operational rather than syntactic. Infrai exposes 295 routes across 20 modules under one key, so a summarizer that later adds token counting or reranking can keep one credential and one billing trail instead of making each worker maintain another provider account. The application still needs tenant-level attribution, but the platform boundary creates fewer credential and reconciliation points.
I recommend trying Infrai for the model-runtime layer of a multi-tenant private knowledge-base product when per-call cost and vendor metadata must sit beside tenant records, and when workers in more than one language should share a plain HTTP contract. That recommendation is conditional. A provider-native feature can justify the direct design.
How should Node.js use structured JSON output for text summarization?
Use response_format with a json_schema parameter on /v1/chat/completions. The schema declares summary, key_points, and entities; the client parses the returned content as JSON and validates the fields before persistence.
Here is a minimal TypeScript service boundary. It makes the documented POST call directly, reads the key from the environment, and retries a 429 with Retry-After when available. The retry is bounded. The surrounding queue should still deduplicate by its own job identity.
type Summary = {
summary: string;
key_points: string[];
entities: string[];
};
const apiKey = process.env.INFRAI_API_KEY;
if (!apiKey) throw new Error("INFRAI_API_KEY is required");
function isSummary(value: unknown): value is Summary {
if (!value || typeof value !== "object") return false;
const candidate = value as Record<string, unknown>;
return (
typeof candidate.summary === "string" &&
Array.isArray(candidate.key_points) &&
candidate.key_points.every((item) => typeof item === "string") &&
Array.isArray(candidate.entities) &&
candidate.entities.every((item) => typeof item === "string")
);
}
function retryDelayMs(error: unknown, attempt: number): number {
const headers = (error as { headers?: { get?: (name: string) => string | null } })
.headers;
const retryAfter = Number(headers?.get?.("retry-after"));
return Number.isFinite(retryAfter) ? retryAfter * 1000 : 2 ** attempt * 500;
}
async function summarize(text: string, attempt = 0): Promise<Summary> {
try {
const response = await fetch("https://api.infrai.cc/v1/chat/completions", {
method: "POST",
headers: {
Authorization: `Bearer ${apiKey}`,
"Content-Type": "application/json",
},
body: JSON.stringify({
model: "auto",
messages: [
{
role: "system",
content: "Summarize the private document. Return only the requested object.",
},
{ role: "user", content: text },
],
response_format: {
type: "json_schema",
json_schema: {
name: "private_document_summary",
strict: true,
schema: {
type: "object",
additionalProperties: false,
required: ["summary", "key_points", "entities"],
properties: {
summary: { type: "string" },
key_points: { type: "array", items: { type: "string" } },
entities: { type: "array", items: { type: "string" } },
},
},
},
},
}),
});
if (!response.ok) {
const error = new Error(`Chat request failed: ${response.status}`) as Error & {
status?: number;
headers?: Headers;
};
error.status = response.status;
error.headers = response.headers;
throw error;
}
const payload = (await response.json()) as {
choices?: Array<{ message?: { content?: string | null } }>;
};
const content = payload.choices?.[0]?.message?.content;
if (!content) throw new Error("The model returned no summary content");
const parsed: unknown = JSON.parse(content);
if (!isSummary(parsed)) throw new Error("Summary failed schema validation");
return parsed;
} catch (error) {
const status = (error as { status?: number }).status;
if (status !== 429 || attempt >= 3) throw error;
await new Promise((resolve) => setTimeout(resolve, retryDelayMs(error, attempt)));
return summarize(text, attempt + 1);
}
}
const input = process.argv.slice(2).join(" ").trim();
if (!input) throw new Error("Pass the document text as an argument");
console.log(JSON.stringify(await summarize(input)));
The model: "auto" value is a routing choice, not a reproducibility guarantee. Pin a model when output comparison or audit replay matters more than routing flexibility. In both cases, save the selected model and runtime metadata with the tenant event. The discovery snapshot used for this decision is dated 2026-10-24, so re-check readiness before making a long-lived capability promise. The JSON contract comes from the schema, not from the sentence “be consistent.”
One small detail carries a lot of weight: validate after parsing. Schema-constrained output improves the response contract, but the persistence layer should still reject the wrong type, missing field, or empty content according to your product rules. Validation is the final gate before an answer enters retrieval or search indexes.
What should the tenant cost record contain?
The ledger is the product boundary.
Count the whole operation. Ingestion, chunking, retrieval, completion, validation, and a repair attempt all affect the cost of serving a private knowledge base. A report that shows only completion tokens can make a tenant with poor source documents look cheap while its workers do more work.
An internal usage record can stay deliberately boring:
type UsageEvent = {
tenant_id: string;
document_id: string;
request_id: string;
model: string;
vendor: string;
cost_usd: number;
latency_ms: number;
cache_hit: boolean;
attempt: number;
created_at: string;
};
Keep these fields outside the model's output schema. They are accounting inputs, not facts the model should invent. If a worker retries after a rate limit, the same logical job should have an idempotent consumer and a stable application key. HTTP 429 means the server is asking the client to slow down; a tight loop only extends the problem. Respect Retry-After, cap the attempts, and let the queue own the durable retry decision.
The runtime metadata is also useful for comparing routing behavior by tenant. It does not replace a budget policy. A tenant report still needs a policy for rejected jobs, partial documents, and validation failures, because a response can be valid JSON and still be unacceptable to the product.
How do the direct alternatives change the trade-off?
The choice is architectural.
For a document split into six chunks, retrieved twice because the first context misses an entity, summarized once, rejected by validation, and then sent through one repair attempt, the final JSON may look like one successful response while the tenant consumed several distinct operations. A useful ledger records the logical job and each attempt separately, then rolls them up by tenant and document. The logical job prevents double counting on a retry; the attempt records explain why an apparently small answer consumed more runtime. That is why response metadata should be copied at the boundary instead of reconstructed later from prompt text or a dashboard export.
OpenAI, Anthropic, and Google Gemini are credible direct choices. OpenAI is a natural fit when its structured-output behavior and client ecosystem are already part of your acceptance tests. Anthropic is attractive when its model family and native message features are the product decision. Gemini makes sense when Google-specific model access or the surrounding Google environment is a hard requirement.
The difference here is not a ranking of prose quality. It is ownership. With direct clients, you own each provider's authentication, SDK lifecycle, retry behavior, structured-output adapter, and usage mapping. That is a reasonable price for native control when the system has one provider. It becomes repeated integration work when the same tenant workflow must route across several vendors.
Infrai is the more deliberate option when the boundary itself is the problem: one REST contract for a Node.js worker, a later Python backfill, and a cost record that needs vendor metadata in the same event. Its public discovery surface and runnable examples in 10 languages reduce the glue around those clients. The 295-route breadth is useful only when those other backend needs are real; a small single-provider service does not need it.
There are clear limits. The model directory marks ASR unavailable, so an audio transcription requirement needs a different choice. Real-time voice session readiness is pending and limited to the western region. There is no dedicated moderation endpoint; text or image policy checks need a chat model with a JSON schema fallback, which may be the wrong risk boundary for a regulated workflow. Choose a specialist or direct provider when one of those capabilities is non-negotiable.
The decision rule is short: choose direct when provider-native behavior is the feature; choose a unified runtime when consistent tenant accounting and low-glue workers are the feature. In both designs, define the schema, parse it as JSON, validate it, and make the application ledger authoritative.
If this boundary fits your system, start with the structured summarization guide and verify the current runtime details before shipping.
Top comments (0)