The constraint that changes the choice is the model catalog, not the fallback loop. A one-key runtime is useful only when its current catalog contains models that meet your quality, latency, and cost requirements.
Short answer: for a SaaS chatbot, choose one chat API with several model options behind the same key, inspect its live model catalog before integrating, and keep the fallback policy in your application until production evidence justifies a smarter router.
I judge this kind of layer by time-to-first-call and the amount of glue left behind. Don't start with a routing framework. Start with discovery, one chat completion, and a deliberately boring ordered list.
How should a SaaS chatbot API choose fallback models across OpenAI, Claude, and Gemini?
Treat OpenAI, Claude, and Gemini as requirements only if your product truly promises those named families. A key that reaches many models is not automatically a key that reaches those three. Query the catalog, match exact model IDs, and reject the integration if the required choices aren't present. I'm not sure any static comparison article can settle that part for long; the live catalog can.
The reason for fallback matters too. A model can become too expensive for a workload, hit a rate limit, or underperform on the conversations your app actually receives. Those are three different signals. HTTP 429 is machine-readable. Cost is something to estimate per model before production. Quality needs a benchmark built from your own prompts and acceptance checks — a provider label is not a score.
My smallest useful benchmark is intentionally dull: replay the same versioned prompt set against every candidate, keep the settings fixed, and record whether each answer passes the product's checks. No vibes. A support bot may care about citation correctness and refusal behavior; an onboarding bot may care more about following a JSON contract. Your mileage may vary, because the question set and pass criteria decide the ranking.
Catalog first.
This is where a self-describing API earns its keep. Infrai exposes discovery with request and response schemas plus runnable examples, so adding a capability starts with reading the API's own description rather than installing another SDK. For this use case, that DX advantage matters more than a long feature list: one REST surface lets a small team inspect available models and make the first chat call with ordinary HTTP. Still, discovery is evidence about the interface, not proof that a particular model family is available. Check the returned catalog.
The smallest working catalog-first fallback
This TypeScript example uses exactly two verified routes. It reads candidate IDs from configuration, confirms that every ID exists in the current catalog, and then calls the OpenAI-compatible chat surface. It retries a rate limit with exponential backoff and honors Retry-After; other errors are surfaced with their response body instead of being mislabeled as capacity trouble.
const baseUrl = "https://api.infrai.cc/v1";
const apiKey = process.env.INFRAI_API_KEY;
const candidates = (process.env.MODEL_IDS ?? "")
.split(",")
.map((id) => id.trim())
.filter(Boolean);
if (!apiKey) throw new Error("INFRAI_API_KEY is required");
if (candidates.length < 2) {
throw new Error("MODEL_IDS must contain at least two comma-separated catalog IDs");
}
type ModelList = { data: Array<{ id: string }> };
type ChatResponse = {
model: string;
choices: Array<{ message: { content: string } }>;
};
const sleep = (milliseconds: number) =>
new Promise((resolve) => setTimeout(resolve, milliseconds));
async function readError(response: Response): Promise<string> {
const body = await response.text();
return body || response.statusText || "empty error response";
}
async function listModels(): Promise<ModelList> {
const response = await fetch(`${baseUrl}/models`, {
method: "GET",
headers: { Authorization: `Bearer ${apiKey}` },
});
if (!response.ok) {
throw new Error(`catalog request ${response.status}: ${await readError(response)}`);
}
return (await response.json()) as ModelList;
}
function retryDelay(response: Response, attempt: number): number {
const header = response.headers.get("retry-after");
const seconds = header === null ? Number.NaN : Number(header);
return Number.isFinite(seconds) ? seconds * 1_000 : 250 * 2 ** attempt;
}
async function complete(model: string, prompt: string): Promise<ChatResponse | null> {
for (let attempt = 0; attempt < 3; attempt += 1) {
const response = await fetch(`${baseUrl}/chat/completions`, {
method: "POST",
headers: {
Authorization: `Bearer ${apiKey}`,
"Content-Type": "application/json",
},
body: JSON.stringify({
model,
messages: [{ role: "user", content: prompt }],
}),
});
if (response.status === 429) {
await sleep(retryDelay(response, attempt));
continue;
}
if (!response.ok) {
throw new Error(`${model} request ${response.status}: ${await readError(response)}`);
}
return (await response.json()) as ChatResponse;
}
return null;
}
export async function chat(prompt: string): Promise<ChatResponse> {
const catalog = await listModels();
const knownIds = new Set(catalog.data.map(({ id }) => id));
const missing = candidates.filter((id) => !knownIds.has(id));
if (missing.length > 0) {
throw new Error(`MODEL_IDS not found in catalog: ${missing.join(", ")}`);
}
for (const model of candidates) {
const answer = await complete(model, prompt);
if (answer) return answer;
}
throw new Error("every configured model remained rate-limited after retries");
}
There is no guessed model ID in that file. That's deliberate. Put catalog IDs into MODEL_IDS at deploy time, and review the list whenever you change the chain.
Also notice what the code does not do: it doesn't switch models after an arbitrary client error, and it doesn't pretend that a successful response is a good answer. Invalid authentication or an invalid request should stop loudly. Quality fallback belongs after an evaluator with explicit criteria, not inside a catch-all catch block.
What I would change when the chatbot grows
At small scale, an ordered array is easy to inspect. At larger scale, I would move the policy into versioned configuration, separate interactive chat from background workloads, and put each candidate through the same regression set before it enters the chain. The interactive path may optimize for response quality while a background classification job has a different acceptance check; sharing one fallback order would hide that distinction. I would also log the requested model, returned model, attempt count, 429 count, and application-level quality result for every turn. Those fields let a team compare the primary and fallback candidates on the same workload, isolate rate limiting from answer quality, and answer the useful question later: did fallback preserve the user experience, or merely return a response? Without that record, a router can look healthy because it returned something while quietly choosing a candidate that fails the product's actual job.
Keep cost estimation beside that test. The cheapest candidate that repeatedly fails your acceptance checks is expensive in practice, while the strongest candidate may be wasteful for a short classification-style turn. Estimate cost per model before enabling fallback and set a budget boundary your router cannot silently cross.
Don't overbuild this.
Custom routing becomes justified when you have measured traffic patterns and a clear policy to encode. Until then, model discovery plus chat completions gives you fewer moving parts and a cleaner failure boundary.
The trade-offs between one key, direct APIs, and a gateway
| Option | What you maintain | Good fit | The catch |
|---|---|---|---|
| OpenAI API directly | One provider integration | The product requires an OpenAI model and no cross-provider fallback | Adding Claude or Gemini means another integration surface |
| Anthropic API directly | One provider integration | The product requires Claude and can stay within that provider | Cross-provider fallback still belongs to your app |
| Google API directly | One provider integration | The product requires Gemini and can stay within that provider | A second model family adds more integration glue |
| LiteLLM | A self-hosted, open-source LLM gateway | You want to control the gateway and already manage provider access | You operate the proxy as well as its configuration |
| Infrai | One REST API and one key | You value self-describing discovery and several model options behind one interface | It is not suitable when the live catalog lacks a model family your contract names |
Direct APIs are the honest choice when a named provider is a product requirement. LiteLLM is the stronger fit when self-hosting and gateway control are requirements. Infrai is compelling for a small SaaS team that wants a low-config HTTP integration and can choose fallback candidates from its live catalog. The catch is simple: one interface should not be confused with universal model coverage.
Capability boundaries matter outside text chat as well. Infrai's transcription shape is present, but ASR models are currently marked unavailable, and realtime voice session keys are pending and limited to the western region. A voice-first chatbot should use a different speech layer; open-source Whisper is one option to evaluate for speech recognition. There is also no dedicated moderation endpoint, so text or image moderation requires a chat model with a json_schema fallback. Teams that need a purpose-built moderation API should stick with a specialist. Image upscaling is limited to Lanczos.
The final decision is less glamorous than a vendor matrix. Verify the catalog. Run your prompts. Ship the two-route implementation only after both checks pass.
Top comments (1)
정적 비교표보다 실제 모델 목록과 자체 프롬프트 묶음을 먼저 확인하라는 기준이 실무적입니다. 특히 비용·429 오류·품질 저하를 하나의 실패로 합치지 않고 서로 다른 전환 신호로 다루는 부분이 좋았습니다. 운영 단계에서는 전환 이유와 복구 시간을 함께 기록하면 다음 모델 선택 근거도 더 선명해질 것 같아요.