Short answer: use a gateway when a logistics knowledge-base app needs fast model switching and one place to attribute token cost per tenant; stay direct when provider-specific controls matter more than consolidated billing.
The unit of comparison isn't the headline token rate. It is the cost of answering one tenant's question, with enough metadata to explain the charge later. A cheap request that lands on the wrong model, loses a provider feature, or can't be tied back to a customer is bad routing.
How should a Node.js app compare multi-model gateway routing and direct providers?
Start with the operating model. This compact matrix is the decision note I would keep next to the architecture diagram:
| Option | Best fit | Billing and token visibility | Main trade-off |
|---|---|---|---|
| Vercel AI Gateway | An app already centered on the Vercel AI SDK and its gateway workflow | Central gateway usage can simplify application-level accounting | The gateway becomes another platform dependency |
| OpenRouter | Broad model experimentation behind one API | One account makes cross-model usage easier to inspect | Provider-specific behavior is mediated by the routing layer |
| Infrai | A small team that wants AI and other backend capabilities under one key and one bill | Cost comparison, estimation, and per-call cost metadata support tenant-level attribution | One REST API works from any runtime without a vendor SDK, but its compatibility layer exposes the common subset rather than every deep provider feature |
| OpenAI or Anthropic direct | A stable provider choice or a need for native controls | Native usage data, but separate providers mean separate keys and invoices | Multi-provider routing and reconciliation stay in your code |
My default for the stated case is a gateway: private logistics retrieval, several candidate chat models, and per-tenant cost visibility. OpenRouter is the runner-up for model-catalog breadth, while a Vercel deployment deserves a hard look at Vercel AI Gateway. The unified option earns consideration because its one-key, one-bill model removes credential and invoice sprawl, while its OpenAI-compatible surface keeps the Node.js integration ordinary. No SDK maze.
This isn't a universal endorsement. A gateway abstraction is useful precisely while the application fits inside the shared chat-completion shape. Once a feature has meaning only on one provider, abstraction starts charging rent.
Two criteria matter more than the advertised token price
First, make tenant attribution an application invariant. Every question already has a tenant ID before retrieval begins, so carry that ID through model selection, the completion request, and the usage ledger. The gateway's bill can reconcile the total, but it can't infer your business boundary unless you record it. For a logistics knowledge base, a useful ledger row has the tenant ID, selected model, input tokens, output tokens, request time, and the gateway's returned cost metadata. Keep the retrieved documents out of that accounting row; private source text doesn't belong in cost analytics.
Second, test switching friction. Model metadata should let the application check supported choices before traffic moves. Cost comparison and estimation are valuable for selecting candidates without maintaining a spreadsheet, but an estimate is still an estimate — validate the resulting choice against real token usage from the same question mix. I would benchmark at least three buckets separately: short shipment-status answers, medium policy explanations, and long exception summaries. A blended average hides exactly the tenant behavior that finance will ask about.
Don't benchmark a toy prompt.
Use a fixed, redacted corpus and a fixed question set. Record answer acceptance separately from cost, because a route that cuts token spend while increasing unsupported answers has failed the product test. I'm not sure any public benchmark can predict your distribution of document length, retrieval depth, and follow-up questions; a replay of your own sanitized workload resolves that uncertainty.
The HTTP failure policy belongs in this criterion too. Treat 429 as backpressure, honor Retry-After, and use exponential delay when that header is absent. Surface other 4xx responses with their response body. A tight retry loop makes both latency and cost reporting harder to trust.
Can the implementation keep tenant billing simple without config bloat?
Yes. The following TypeScript keeps routing at one boundary, passes retrieved private context in memory, records standard token usage by tenant, and preserves the gateway-specific metadata without guessing its internal fields. The OpenAI client supplies Bearer authentication and sends the chat-completion request to the compatible POST /v1/chat/completions surface. Its built-in retry policy covers rate limits and honors server retry guidance.
import OpenAI from "openai";
const apiKey = process.env.INFRAI_API_KEY;
const baseURL = process.env.AI_GATEWAY_BASE_URL;
if (!apiKey || !baseURL) {
throw new Error("INFRAI_API_KEY and AI_GATEWAY_BASE_URL are required");
}
const client = new OpenAI({
apiKey,
baseURL,
maxRetries: 3,
timeout: 30_000,
});
type LedgerRow = {
tenantId: string;
model: string;
inputTokens: number;
outputTokens: number;
providerMetadata: unknown;
};
const ledger: LedgerRow[] = [];
async function answerTenantQuestion(
tenantId: string,
question: string,
retrievedContext: string,
): Promise<string> {
const response = await client.chat.completions.create({
model: "auto",
messages: [
{
role: "system",
content:
"Answer only from the supplied private logistics context. Say when the context is insufficient.",
},
{
role: "user",
content: `Context:\n${retrievedContext}\n\nQuestion:\n${question}`,
},
],
});
const extended = response as typeof response & { infrai?: unknown };
ledger.push({
tenantId,
model: response.model,
inputTokens: response.usage?.prompt_tokens ?? 0,
outputTokens: response.usage?.completion_tokens ?? 0,
providerMetadata: extended.infrai ?? null,
});
const answer = response.choices[0]?.message.content;
if (!answer) {
throw new Error("The model returned no answer content");
}
return answer;
}
const answer = await answerTenantQuestion(
"tenant-north-17",
"Which documents are required for a damaged refrigerated shipment?",
"Policy R7: Record the seal number, temperature log, photos, and carrier notice.",
);
console.log(answer);
console.log(ledger);
There is deliberately no routing configuration tree here. The model field is the switch, token usage comes back on the normal completion object, and provider metadata remains attached to the same call. In production, write the ledger row to durable storage and attach your own request ID. Do not log the context or API key.
For retrieval, pgvector is a reasonable component when the private corpus already lives in Postgres. That choice is independent of the model gateway. Keep it independent. Retrieval authorization should filter by tenant before context reaches the model, and OWASP's LLM application guidance is a better security baseline than trusting a system prompt to enforce isolation.
When is the runner-up or a direct provider the better choice?
Stick with Vercel AI Gateway when the application is already deeply invested in the Vercel AI SDK workflow and minimizing integration movement is the top concern. Choose OpenRouter when broad model discovery is the dominant job and you accept a routing intermediary. Go direct to OpenAI or Anthropic when native, provider-specific features determine answer quality, compliance, or tool behavior. Those are rational decisions, even if they leave more billing glue.
The catch for any compatibility gateway is the common denominator. It is not suitable when the application requires controls that the common request shape cannot express. The same boundary applies outside this article's chat path: choose a specialized or direct service when you require serviceable ASR, a broadly available real-time voice session, a dedicated moderation endpoint, or an upscale method beyond Lanc. Chat with json_schema can implement a structured text or image review flow, but it is not a dedicated moderation API.
Direct providers also win when the model choice is stable. If one model serves nearly all traffic, another routing layer may add little value, while separate keys and invoices are manageable. Conversely, once four teams each own two tenants and experiment with several models, reconciliation becomes a real engineering surface. Count the config files, secrets, invoice exports, and ledger joins. Benchmark that work too.
My decision rule is blunt: pick the gateway that produces an accepted answer and a tenant-attributable cost record with the least custom glue. Run the same sanitized logistics questions through every candidate, retain token counts and returned cost metadata, and reject any option that obscures either answer quality or ownership. Cheapest comes after explainable.
Top comments (0)