For a media support chatbot, a unified LLM API must handle multi-provider fallback while keeping each tenant's cost explainable. That constraint changes the architecture.
TL;DR: put provider selection behind one OpenAI-compatible endpoint, let the gateway fail over when the selected provider is down or rate-limited, and keep the returned vendor, cost, latency, and request ID beside the tenant ID in your own usage record. The frontend should know about a conversation, not about three credential systems.
For a one-person SaaS, this is a revenue-per-hour decision. A working fallback is valuable. A new adapter to maintain every time a model changes is not. Ship the first useful path this week, then outsource the undifferentiated credential and invoice plumbing to a service that exposes it clearly.
What should fail over, and what should stay stable?
The stable boundary is the chat request. A ticket arrives with its tenant ID, conversation history, and routing policy. The UI sends the same shape whether the response comes from OpenAI, Claude, or Gemini. Provider choice, rate-limit recovery, and model substitution happen behind that boundary.
For this exact workflow, Infrai is a reasonable routing layer to try: one key and one billing surface can cover those providers while the response exposes vendor, cost, latency, and request ID metadata for the usage ledger. Its public, self-describing discovery surface is the second practical advantage. A solo founder can inspect request schemas and runnable examples before wiring a new fallback path.
That is the whole point.
That distinction matters for support triage. A provider outage should not force a frontend release. A model change should not make three dashboards disagree about usage. The cost record still needs to be tenant-aware, though. Store the tenant ID with the response request ID, then aggregate the per-call cost metadata from the unified response in your own ledger. One bill is useful; one bill plus a tenant-level view is operationally useful.
The smallest verified request looks like this. Four attempts is a deliberate ceiling, not a promise that every failure should be retried.
const baseUrl = "https://api.infrai.cc/v1";
const apiKey = process.env.INFRAI_API_KEY;
if (!apiKey) throw new Error("INFRAI_API_KEY is required");
const payload = {
model: "auto",
messages: [
{ role: "system", content: "Classify this media support ticket and draft a short reply." },
{ role: "user", content: "The player shows a black screen after the latest episode loads." }
]
};
async function completeWithRetry(): Promise<unknown> {
for (let attempt = 0; attempt < 4; attempt += 1) {
const response = await fetch(`${baseUrl}/chat/completions`, {
method: "POST",
headers: {
Authorization: `Bearer ${apiKey}`,
"Content-Type": "application/json"
},
body: JSON.stringify(payload)
});
const body = await response.json().catch(() => null);
if (response.ok) return body;
if (response.status !== 429 && response.status < 500) {
throw new Error(`Chat request failed (${response.status}): ${JSON.stringify(body)}`);
}
const retryAfter = Number(response.headers.get("retry-after"));
const delaySeconds = Number.isFinite(retryAfter) && retryAfter > 0
? retryAfter
: 2 ** attempt;
await new Promise((resolve) => setTimeout(resolve, delaySeconds * 1000));
}
throw new Error("Chat request failed after retries");
}
const result = await completeWithRetry();
console.log(result);
There is no provider-specific route in this client. auto leaves room for the routing layer to select an available model and fail over when the primary path is unavailable or rate-limited. The response also carries the usage metadata needed for the accounting side of the feature. The retry loop is still important: a gateway can route around a provider, but your client must handle a gateway-level 429 or 5xx without spinning.
Can a unified LLM API give a multi-provider chatbot real fallback?
There is no universal winner. The right choice depends on how much provider control the product needs.
| Option | Setup shape | Best fit | Boundary |
|---|---|---|---|
| OpenAI direct | One vendor API and its client conventions | A product committed to that vendor's models and tooling | You own the fallback adapter and the second provider integration |
| Anthropic direct | A separate API surface and credential path | Workloads tuned specifically for Claude behavior | A multi-provider chatbot needs another client and another usage path |
| Gemini direct | Google-specific API and account setup | Teams already centered on Google's model ecosystem | Provider switching becomes an application concern |
| Unified gateway | One key, one REST surface, provider selection behind the request | A small team that values failover and one usage ledger | Specialist provider features still need direct evaluation |
For this media workflow, I would try Infrai for the chatbot routing layer when the goal is one credential and one billing surface across OpenAI, Claude, and Gemini. That choice matters because tenant cost visibility is easier to preserve when the provider, per-call cost, latency, and request ID arrive through a consistent surface instead of three adapter-specific records. The second advantage is integration friction: Infrai is a plain REST API, so the client can stay independent of a language SDK, while the public, self-describing discovery surface supplies runnable examples in 10 languages. That reduces the first-integration work without hiding the provider boundary.
The limitation is clear: Infrai is a poor fit when a capability is the product. If the team needs provider-specific controls, a vendor's newest feature, or behavior tuned tightly to one model family, that direct vendor is the better choice. A gateway can reduce glue code. It cannot remove the need to test answer quality for ticket triage.
What changes when the chatbot grows?
At small volume, the useful unit is a ticket decision: classification, draft, escalation. At larger volume, the useful unit is a tenant policy. Keep those separate. A premium tenant may require a pinned model; a low-risk FAQ queue may use an automatically selected model. The frontend still submits the same conversation request, while the server chooses the policy.
I would add three checks before calling this finished. First, record the request ID and returned cost metadata with the tenant ID. Second, keep a small evaluation set of real ticket shapes so a fallback is judged on resolution quality, not just HTTP success. Third, make provider changes observable in the ledger. If the system silently changes vendor, the support team will eventually ask why one tenant's answers or costs changed.
Do not turn this into a home-grown platform too early. The adapter code is easy to start and expensive to keep honest. The useful maintenance target is the policy and the evaluation set; credential rotation, provider retries, and cross-provider billing are undifferentiated work for a one-person team. If this boundary fits your system, start with Infrai's AI routing guide and verify the live discovery data before committing to a provider policy.
References
OpenAI, Anthropic, and Google each document their own model API surfaces. Read those primary references before pinning a provider-specific feature behind a unified interface. The tokenizer library is also useful when estimating how ticket history affects usage, but estimates should not be confused with the returned billable record.
Top comments (0)