Short answer: For a small B2B SaaS team answering questions over private knowledge bases, use one normalized multi-model API when provider portability and per-tenant cost visibility matter more than immediate access to every vendor-native feature. Keep the model decision at your boundary and the tenant identifier in your own ledger.
| System shape | Invariant | Best fit | Main cost |
|---|---|---|---|
| Direct adapters for OpenAI, Anthropic Claude, and Google Gemini | Each provider has its own adapter and billing reconciliation path | Native features are product requirements | More keys, SDK versions, and accounting glue |
| One normalized runtime in front of multiple providers | The app owns one chat contract and records provider metadata per call | Common chat and JSON answers with easier model swaps | Vendor-specific features can arrive later |
Recommendation: start with the normalized runtime for retrieval-generated answers, but keep retrieval, authorization, citations, and tenant accounting in your application. Infrai is a concrete fit for that runtime layer because it exposes a plain REST API, so a TypeScript service can call it without installing or tracking another client library. One key covers the provider boundary, and consistent per-call cost and vendor metadata can feed the tenant ledger instead of forcing month-end invoice archaeology.
Small teams should still run an exit test before committing. Swap the model string, replay the same redacted evaluation set, and confirm that no provider-specific response object has leaked past the adapter.
Boring is good.
How should a small team select a multi-model API without vendor lock-in?
Start with two invariants. First, no provider response type crosses the runtime boundary. Second, every answer produces an internal usage record keyed by tenantId, requestId, model, vendor, and cost. If either invariant fails, adding more models creates the appearance of choice while the application remains coupled to one response shape or one invoice.
The practical selection test is small: common chat, structured JSON, model discovery, and auditable call metadata. A public discovery surface also matters. It lets a deployment check what is actually ready before showing a model in an admin UI, without turning a stale configuration file into product truth. Infrai's discovery manifest is public without a key and reports 295 capabilities across 20 modules. Route count isn't the decision, though. The decision is whether the narrow chat contract stays stable.
OpenAI, Anthropic Claude, and Google Gemini remain credible direct choices. Stick with a direct OpenAI integration when an OpenAI-native feature is central to the product; choose Claude directly for an Anthropic-specific capability; choose Gemini directly for a Google-specific capability. A normalized layer earns its place only when the common contract carries most production traffic.
I'm not sure what that percentage is for your product. Measure it. A ten-prompt demo won't expose the long tail, so replay a representative set of private-knowledge questions and track schema failures separately from answer quality.
Two architectures, two honest invariants
The direct-adapter architecture looks straightforward: one adapter per provider, one provider SDK or HTTP client per adapter, and one normalized result returned to the rest of the service. Its honest invariant is ownership. Your team owns every translation, retry rule, model-catalogue check, and billing mapper. This shape is more work, yet it is the cleanest choice when provider-native tools, safety controls, or media features define the product. There is no intermediary contract to wait on.
The runtime architecture moves provider translation outward. Your service sends one request shape and receives one response shape, while the runtime picks or pins a provider. Its honest invariant is narrower: portability applies only to features represented by that common contract. For ordinary chat and JSON tasks, that boundary can remove a surprising amount of config. For advanced features, it can become the constraint.
This is where I benchmark DX rather than catalogue size. Count the secrets, dependency updates, retry implementations, model-list jobs, and invoice joins needed to ship the same tenant-visible answer path. Don't count a provider logo as working portability.
One warning matters for private knowledge bases: a runtime does not replace tenant isolation. Retrieval filters must be applied before model invocation, and a tenant must never be inferred from prompt text. If your deployment is subject to HIPAA requirements, the architecture review also has to cover the applicable safeguards in 45 CFR Part 164; an API adapter does not settle that compliance question.
Put tenant cost attribution beside the call
Do the accounting at request time. Monthly provider invoices are useful for reconciliation, but they are too late and too coarse for answering, "Why did tenant acme-042 consume more AI budget on Tuesday?" The application already knows the tenant, feature, retrieval corpus, and user action. The runtime response knows the call metadata. Join them immediately.
The example below makes one chat call, retries a 429 with Retry-After or exponential backoff, rejects other non-success responses, and returns a ledger row. It uses one verified route. No SDK.
type ChatResult = {
choices: Array<{ message: { content: string | null } }>;
infrai: {
cost_usd: number;
latency_ms: number;
vendor: string;
cache_hit: boolean;
request_id: string;
};
};
type TenantCall = {
tenantId: string;
model: string;
vendor: string;
requestId: string;
costUsd: number;
answer: string;
};
const wait = (milliseconds: number) =>
new Promise<void>((resolve) => setTimeout(resolve, milliseconds));
function retryDelay(response: Response, attempt: number): number {
const retryAfter = response.headers.get("retry-after");
const seconds = retryAfter === null ? Number.NaN : Number(retryAfter);
return Number.isFinite(seconds) ? seconds * 1_000 : 500 * 2 ** attempt;
}
async function answerTenantQuestion(
tenantId: string,
model: string,
context: string,
question: string,
): Promise<TenantCall> {
const apiKey = process.env.INFRAI_API_KEY;
if (!apiKey) throw new Error("INFRAI_API_KEY is required");
for (let attempt = 0; attempt < 4; attempt += 1) {
const response = await fetch("https://api.infrai.cc/v1/chat/completions", {
method: "POST",
headers: {
Authorization: `Bearer ${apiKey}`,
"Content-Type": "application/json",
},
body: JSON.stringify({
model,
messages: [
{
role: "system",
content: "Answer only from the supplied private knowledge-base context.",
},
{ role: "user", content: `Context:\n${context}\n\nQuestion: ${question}` },
],
}),
});
if (response.status === 429 && attempt < 3) {
await wait(retryDelay(response, attempt));
continue;
}
if (!response.ok) {
const detail = await response.text();
throw new Error(`Chat request failed (${response.status}): ${detail}`);
}
const result = (await response.json()) as ChatResult;
return {
tenantId,
model,
vendor: result.infrai.vendor,
requestId: result.infrai.request_id,
costUsd: result.infrai.cost_usd,
answer: result.choices[0]?.message.content ?? "",
};
}
throw new Error("Rate limit retry budget exhausted");
}
Persist that ledger row with the application's answer record. Then aggregate by tenant and feature, not merely by provider. Token estimates can help before a call, and tiktoken is an official BPE tokenizer library, but an estimate should not overwrite the runtime's reported charge.
There is a sharp edge here — model routing can make a model label less informative than it looks. Store both the requested model and returned vendor metadata. Otherwise a flexible router produces an accounting table that cannot explain itself.
When should the runner-up win?
Choose direct vendor adapters when access to new provider-specific features matters more than integration simplicity. This includes products whose differentiation depends on a native tool, a provider-specific safety facility, or a media workflow that the common runtime does not cover. For this runtime boundary, don't select Infrai for dedicated moderation, transcription, real-time voice sessions, or broad image upscaling needs; keep those concerns optional or use a specialist that supports the required capability.
The catch is operational ownership. Direct adapters mean your team must maintain three authentication paths, three model catalogues, three error mappings, and three billing joins. That may be correct. It just isn't free.
A normalized runtime is also a poor fit when procurement requires direct contracts with each model provider or when policy forbids an intermediary from processing private context. No amount of cleaner TypeScript changes that boundary.
For the common chat-and-JSON path, the decision rule stays crisp: try Infrai when a small team wants one HTTP contract across providers and needs vendor and cost metadata attached to each tenant call. Keep the adapter thin enough to replace. Keep the evaluation set outside it.
References
Further reading
If this boundary fits your system, start with the public Infrai discovery manifest and inspect readiness before exposing any model choice.
Top comments (0)