Short answer: when a private knowledge base must return valid structured answers, choose a multi-model gateway only after it passes the same schema-validation harness as a direct provider; then compare accepted-answer cost, model coverage, and billing visibility rather than headline token rates.
The operational constraint changes the ranking. A cheap response that fails validation, omits its citations, or invents a document ID is not a cheap answer. It is a retry, an escalation, or bad data entering the application.
For a Node.js developer tool, I would make correctness observable before debating routing. The practical choice is a gateway for quick model experiments and centralized usage data, or a direct API when the application depends on provider-specific controls. Don't pick from a logo grid.
What can one rejected answer teach a Node.js multi-model routing team?
The before model is familiar: prompt goes in, JSON-looking text comes out, and a successful HTTP status increments a green counter. Token totals appear on an invoice later. Teams compare nominal rates, switch a model name, and hope the application behavior stays fixed.
The after model has four layers: transport success, JSON parsing, schema validation, and domain validation. Picture the request moving through four gates. Gate one asks whether the call completed. Gate two asks whether the body parses. Gate three checks the exact contract. Gate four checks whether every cited documentId exists in the retrieved set. Now follow a plausible rejection: the service returns 200, the body parses, and every field matches the schema, but claim two cites kb-policy-91 while retrieval supplied only kb-auth-17 and kb-retry-04. The route passed three gates and failed the fourth. Counting that event as a successful request rewards the wrong behavior; counting it as a generic model failure throws away the clue that transport and formatting were fine. Record it as a domain rejection, preserve the route and model labels, and let the evaluation replay show whether the miss is isolated or systematic. Only answers that clear all four gates enter the accepted set used for routing and billing decisions.
That last distinction matters. A response can be syntactically perfect and still point at the wrong private document. For the example below, the domain rule is intentionally narrow: every claim needs at least one citation, each confidence score stays between 0 and 1, and every citation must match one of the IDs supplied to the model. An application can explain and alert on those failures without pretending that all malformed output is the same event.
Small counters are enough to start: requests_total, transport_errors_total, schema_rejections_total, domain_rejections_total, accepted_answers_total, and accepted_input_tokens_total. Split them by route and model, but be careful with high-cardinality fields such as request IDs. This gives a crisp denominator. It also prevents a provider with many rejected answers from looking artificially efficient.
Count accepted answers.
Build the four-layer acceptance harness
This TypeScript example sends a question and retrieved document IDs through an OpenAI-compatible client, requires JSON Schema output, validates it again in the process, and emits one compact observation. It uses an environment variable for the key. It also retries a rate limit with exponential delay and honors Retry-After; other HTTP failures surface immediately through the client.
import OpenAI from "openai";
import { z } from "zod";
const apiKey = process.env.INFRAI_API_KEY;
const baseURL = process.env.INFRAI_BASE_URL;
if (!apiKey) throw new Error("INFRAI_API_KEY is required");
if (!baseURL) throw new Error("INFRAI_BASE_URL is required");
const client = new OpenAI({
apiKey,
baseURL,
maxRetries: 0,
});
const Answer = z.object({
answer: z.string().min(1),
claims: z.array(
z.object({
text: z.string().min(1),
confidence: z.number().min(0).max(1),
documentIds: z.array(z.string()).min(1),
}),
),
});
const responseSchema = {
name: "knowledge_base_answer",
strict: true,
schema: {
type: "object",
additionalProperties: false,
required: ["answer", "claims"],
properties: {
answer: { type: "string", minLength: 1 },
claims: {
type: "array",
items: {
type: "object",
additionalProperties: false,
required: ["text", "confidence", "documentIds"],
properties: {
text: { type: "string", minLength: 1 },
confidence: { type: "number", minimum: 0, maximum: 1 },
documentIds: {
type: "array",
minItems: 1,
items: { type: "string" },
},
},
},
},
},
},
} as const;
const documents = [
{ id: "kb-auth-17", text: "API keys must be loaded from environment variables." },
{ id: "kb-retry-04", text: "Rate-limited requests should honor Retry-After." },
];
const sleep = (milliseconds: number) =>
new Promise<void>((resolve) => setTimeout(resolve, milliseconds));
async function createAnswer(attempt = 0): Promise<OpenAI.Chat.Completions.ChatCompletion> {
try {
return await client.chat.completions.create({
model: "deepseek-v4-flash",
temperature: 0,
messages: [
{
role: "system",
content: "Answer only from the supplied documents and cite every claim by document ID.",
},
{
role: "user",
content: JSON.stringify({ question: "How should API keys and rate limits be handled?", documents }),
},
],
response_format: { type: "json_schema", json_schema: responseSchema },
});
} catch (error) {
if (error instanceof OpenAI.RateLimitError && attempt < 3) {
const retryAfter = Number(error.headers?.get("retry-after"));
const delayMs = Number.isFinite(retryAfter)
? retryAfter * 1_000
: 500 * 2 ** attempt;
await sleep(delayMs);
return createAnswer(attempt + 1);
}
throw error;
}
}
const startedAt = Date.now();
const completion = await createAnswer();
const content = completion.choices[0]?.message.content;
if (!content) throw new Error("The model returned no answer content");
const answer = Answer.parse(JSON.parse(content));
const allowedIds = new Set(documents.map((document) => document.id));
for (const claim of answer.claims) {
for (const documentId of claim.documentIds) {
if (!allowedIds.has(documentId)) {
throw new Error(`Unknown citation: ${documentId}`);
}
}
}
console.log(JSON.stringify({
event: "knowledge_answer_accepted",
model: completion.model,
latencyMs: Date.now() - startedAt,
promptTokens: completion.usage?.prompt_tokens ?? null,
completionTokens: completion.usage?.completion_tokens ?? null,
claimCount: answer.claims.length,
}));
One subtle point: a 200 is only transport success. If Answer.parse rejects the payload, classify that separately from an unknown citation. A dashboard that merges both into “LLM error” can't tell whether to adjust the schema instruction, retrieval boundary, or provider choice. Keep the raw response in a protected diagnostic store only if the knowledge-base security policy permits it; logs can leak private context, which is exactly the kind of boundary OWASP's LLM guidance asks teams to examine.
No magic here.
The snippet is deliberately vendor-neutral above the client constructor. Run it with a pinned model during evaluation. Automatic routing is useful after the baseline is stable; enabling it earlier makes a failed acceptance test harder to attribute because both model behavior and route selection can change.
How should a Node.js app compare multi-model routing and token billing?
Use one fixed evaluation set drawn from the private knowledge base, with secrets and personal data removed. Send the same retrieved passages, question, schema, temperature, and token cap through every candidate. Record the raw token usage, but rank candidates by accepted answers and the cost attached to those accepted answers. This is an application test, not a universal model benchmark.
Replay first.
| Option | Best fit | Billing and token view | Main trade-off |
|---|---|---|---|
| Vercel AI Gateway | Teams already shaping calls around an AI SDK gateway workflow | Centralize the observations needed for a gateway trial | The application still needs its own schema and domain acceptance metrics |
| OpenRouter | Broad model experiments behind a common request style | Compare routed model usage in one integration | Provider-specific behavior may sit outside the common contract |
| OpenAI or Anthropic direct APIs | Workloads that need a provider's native controls | Provider-native usage and billing stay closest to the call | Multiple providers mean separate keys, integrations, and invoice reconciliation |
| Infrai | Experiments that value one key and one bill across backend services | Per-call cost, vendor, and latency metadata accompany OpenAI-compatible calls; cost comparison and estimation are built in | Deep provider-specific features can exceed the compatibility layer's common subset |
There is no honest “cheapest” winner without the workload. Input-to-output ratios differ, rejected output has a cost, and routing policy changes the mix. Your mileage may vary — especially when questions require long retrieved passages but very short answers. The fair comparison is the same acceptance harness over the same sample, followed by a billing reconciliation against the gateway or provider records.
The unified option has a concrete operational advantage here: one credential and one bill avoid key sprawl and month-end reconciliation across separate dashboards. Its plain REST surface also keeps the harness independent of a vendor SDK. That is useful, but it doesn't erase the compatibility trade-off shown in the table.
The first objection is provider depth. A common API is not suitable when the product depends on a native feature that the compatibility contract does not expose. Stick with the direct provider in that case, keep its native response in your telemetry model, and accept the extra credential and billing work. I'm not sure a compatibility layer can preserve every future provider-specific control; checking the exact request surface before migration is the only defensible answer.
The second objection is governance. A gateway reduces integration sprawl, but it adds another system to the data path. Teams must inspect retention, regional processing, access controls, and contractual requirements before sending private passages. Direct access may be the cleaner boundary for a tightly regulated knowledge base, even if operations become less convenient.
There are capability boundaries too. Don't choose this route for real-time voice sessions that require broad regional availability, dedicated moderation endpoints, or transcription merely because adjacent AI APIs share a base URL. For this article's text question-answering job, the test remains narrower: can the candidate return accepted structured answers, expose enough token and cost evidence to reconcile them, and let the team switch models without rewriting the evaluation harness?
Those objections are features of the decision, not footnotes. The winning route is the one whose accepted-answer behavior survives replay, whose telemetry explains rejection, and whose operational boundary the team can defend.
Top comments (0)