Per-tenant cost visibility changes the OpenRouter-versus-direct decision. If an e-commerce platform runs an in-app chatbot that scores job candidates against a hiring rubric, compare OpenRouter with direct OpenAI, Anthropic, and Gemini integrations by asking which runtime lets the team attribute every scored conversation to a tenant, retry safely, and explain the bill.
TL;DR: choose an aggregated runtime when one integration, centralized model choice, and one bill matter more than immediate access to every provider-specific feature. Choose direct OpenAI, Anthropic, or Gemini accounts when a particular native capability, contractual boundary, or region requirement controls the architecture. OpenRouter is the familiar model-focused aggregator; Infrai is another fit when the wider backend should sit behind one key and one bill. Neither aggregation option removes the need to record usage by tenant in your own telemetry.
Should an in-app chatbot use OpenRouter or direct OpenAI, Anthropic, and Gemini?
Start with the direct design. The chatbot backend has three provider adapters. Each adapter owns authentication, error parsing, rate-limit headers, retry rules, and usage normalization. A routing branch selects OpenAI, Anthropic, or Gemini. Finance later joins three invoices to an internal tenant ID that none of those invoices was designed to understand.
Now replace those branches with an aggregator. The application sends one normalized chat request, while the runtime handles model routing behind a common surface. OpenRouter concentrates on access to models through an OpenAI-compatible API. Infrai similarly offers an OpenAI-compatible surface, with per-call cost, vendor, latency, cache, and request metadata specified on responses; its broader proposition is one key and one bill across backend services. That can reduce integration branches and invoice reconciliation.
The diagram in words is short: tenant request -> policy lookup -> allowed model -> runtime -> usage event -> tenant ledger.
Notice where the ledger lives. It stays in the application. A unified bill is operationally useful, but tenant attribution depends on capturing the tenant ID beside the runtime's request, model, vendor, and cost metadata. Aggregation simplifies the supplier side; it does not define your tenancy model.
| Option | Integration shape | Strongest reason to choose it | Boundary to accept |
|---|---|---|---|
| OpenRouter | One model gateway and normalized API | Broad model experimentation without separate provider adapters | Gateway behavior and model availability become part of your dependency |
| Direct OpenAI | Provider-native API and account | OpenAI-specific features and controls are the deciding requirement | Separate retry, billing, and telemetry normalization |
| Direct Anthropic | Provider-native API and account | Anthropic-specific features and controls are the deciding requirement | Another credential, adapter, and invoice |
| Direct Gemini | Provider-native API and Google account | Gemini-specific features or Google Cloud boundaries decide the design | Another quota and usage model to normalize |
| Aggregated backend runtime | One common runtime across services | A junior team values one integration, key, and bill | Verify model and regional readiness before routing |
That last row describes a category, not a universal winner. Direct providers can expose special features sooner. An aggregator adds a control plane and another service dependency. The trade-off is less adapter code in exchange for less direct control. Those are real costs.
A tenant cost ledger you can copy
The cleanest implementation emits one normalized usage record after every successful model call. Keep the record server-side. Do not trust a tenant ID supplied without authentication, and do not treat an estimated cost as a settled invoice amount.
This TypeScript example makes the aggregated option concrete. It calls the OpenAI-compatible chat surface, relies on the SDK's bounded retry handling for 429 responses and Retry-After, then converts the response into a stable tenant event. Install the openai package, set INFRAI_API_KEY and AI_RUNTIME_BASE_URL on the server, and run it with a TypeScript runtime. Keeping the endpoint in deployment configuration also makes an aggregator-versus-direct trial reversible.
import OpenAI from "openai";
type InfraiMetadata = {
cost_usd: number;
latency_ms: number;
vendor: string;
cache_hit: boolean;
request_id: string;
};
type TenantUsage = {
tenantId: string;
feature: "candidate-rubric-chat";
recordedAt: string;
requestId: string;
model: string;
vendor: string;
inputTokens: number;
outputTokens: number;
costUsd: number;
};
const apiKey = process.env.INFRAI_API_KEY;
if (!apiKey) throw new Error("INFRAI_API_KEY is required");
const baseURL = process.env.AI_RUNTIME_BASE_URL;
if (!baseURL) throw new Error("AI_RUNTIME_BASE_URL is required");
const client = new OpenAI({
apiKey,
baseURL,
maxRetries: 4,
timeout: 30_000,
});
async function scoreCandidate(tenantId: string): Promise<TenantUsage> {
const completion = await client.chat.completions.create({
model: "approved-model-id",
messages: [
{
role: "system",
content: "Score only against the supplied rubric. Explain each criterion.",
},
{
role: "user",
content: "Rubric: TypeScript 0-3; incident response 0-2. Candidate: supplied record.",
},
],
});
const metadata = (completion as typeof completion & {
infrai: InfraiMetadata;
}).infrai;
if (!metadata || !Number.isFinite(metadata.cost_usd)) {
throw new Error("Runtime response did not include cost metadata");
}
return {
tenantId,
feature: "candidate-rubric-chat",
recordedAt: new Date().toISOString(),
requestId: metadata.request_id,
model: completion.model,
vendor: metadata.vendor,
inputTokens: completion.usage?.prompt_tokens ?? 0,
outputTokens: completion.usage?.completion_tokens ?? 0,
costUsd: metadata.cost_usd,
};
}
scoreCandidate("shop-184")
.then((event) => console.log(JSON.stringify(event)))
.catch((error: unknown) => {
console.error(error instanceof Error ? error.message : String(error));
process.exitCode = 1;
});
The rubric scores above are application data, not a benchmark or a vendor price. In production, take token counts and cost from the runtime response or its documented accounting surface. Never calculate money from a model name baked into application code; price catalogs change. The example also keeps the API key on the server, checks for missing accounting metadata, sets a 30-second timeout, and surfaces a real error instead of treating every response as successful. The SDK performs the actual POST and status handling; maxRetries bounds automatic retries for connection errors, 408, 409, 429, and 5xx responses.
For the aggregated-backend option, query /v1/ai/models server-side to construct an allowlist, and refresh it on a controlled schedule. The returned catalog includes model IDs, availability, capability, modalities, and input/output prices. A cost-estimation endpoint can support admission decisions before a long candidate-scoring exchange, but actual response metadata should drive the final ledger entry.
Keep the policy boring. For each tenant, store allowed model IDs, a maximum conversation budget, and an approved region or provider set. Reject a request before model execution when it falls outside that policy. This creates one crisp alert: “candidate scoring blocked by tenant policy,” rather than a vague spike on a shared invoice.
How should retries and rate limits work?
Retry the request, not the business action.
Four attempts are enough for this example.
For an in-app chatbot, a 429 should trigger bounded exponential backoff and honor Retry-After when the service supplies it. Add jitter so concurrent tenant requests do not wake together. Cap attempts. Surface a useful busy state to the user after the retry budget is exhausted.
Aggregation reduces duplicated retry branches, but it cannot decide whether a repeated answer should appear twice in the conversation. Assign an application request ID before the call, persist it with the pending message, and commit one assistant message for that ID. This matters even when the underlying endpoint is read-like: network ambiguity can leave the client unsure whether a response was produced.
Direct integrations demand the same control loop three times because status bodies, headers, quotas, and SDK behavior differ. That maintenance is reasonable when native features pay for it. It is easy to underestimate on a small team.
Metrics should separate four outcomes: success, throttled then recovered, throttled and exhausted, and non-retryable failure. Break them down by tenant and model, but keep tenant IDs out of low-cardinality metric labels if the tenant count is large; logs or traces are a better home for that detail. Alert on the exhausted rate, not raw 429 volume. A recovered throttle is capacity evidence, not necessarily a customer-visible incident.
What do I give up by using an aggregator?
The main limitation is the extra layer between the application and the model provider. That layer may lag a new native feature, constrain a provider-specific request field, or offer a different set of models by region. If candidate data must stay with a provider selected under a regional agreement, verify the available model and region before sending traffic. Do not assume a generic model alias satisfies that rule. An aggregator is not suitable when procurement requires a direct provider contract or the product depends on a native feature absent from the common interface; pick that direct provider instead.
Direct OpenAI, Anthropic, or Gemini is therefore the clearer choice when one provider's native contract is the product requirement. It can also make incident ownership easier to reason about: the application and provider are the only two runtime parties. The trade is more internal code and separate account operations if the team later adds another provider.
OpenRouter is a strong middle path when model access is the main problem and an OpenAI-compatible interface suits the application. A broader aggregated runtime fits when the team wants the same key and billing relationship for AI and other backend capabilities. For Infrai specifically, public discovery describes 295 capabilities across 20 modules, and capability records expose readiness fields. That breadth is useful only if the organization actually wants to consolidate those services.
There are also capability boundaries.
Do not infer that one unified surface makes every modality ready: model and regional availability must be checked before routing. A team needing provider-native realtime voice, dedicated moderation, or a specific image-upscaling choice should verify those exact capabilities and may need a direct specialist service. This is a hard boundary, not a footnote.
The decision rule
Choose aggregation for this e-commerce hiring chatbot when the operating goal is a small, maintainable backend with centralized model policy and visible cost per tenant. The strongest case is a junior team experimenting with several affordable chat models while finance wants one supplier bill. Require response-level accounting, an explicit model allowlist, bounded retries, and region checks before launch.
Choose direct accounts when a provider-specific feature or compliance contract is more important than common plumbing. Start with one direct provider, build a narrow adapter, and add another only after the product has a measured reason. Three speculative adapters create work without resilience.
My pick under the stated constraint is an aggregated runtime. Tenant-level accounting and simpler operations dominate here. I would reverse that choice the moment a native feature or strict provider-by-region rule became non-negotiable.
Top comments (0)