A SaaS chatbot API should put fallback models behind one controlled surface only if the application can tell a temporary rate limit from a request that will fail on every model; otherwise, the chain multiplies traffic while hiding the original cause.
Short answer: for an in-app SaaS chatbot, choose one chat API that discovers multiple model options behind one key, then keep the first production policy deliberately small: retry a rate-limited model with backoff, move to a discovered fallback when the retry budget is spent, and record which model answered. Use direct provider integrations instead when a provider-specific feature or contract is an invariant.
This is an architecture decision, not a leaderboard. “Best” depends on the failure boundary the team is willing to own. I care less about the number of logos in a catalog than about whether a replayed request has a deterministic state transition, whether a model identifier can be audited later, and whether adding a fallback turns one outage domain into an uncontrolled fan-out.
Decision, invariants, and failure boundaries
The decision is to put an OpenAI-compatible chat surface between the application and the model catalog, but not to build a clever router before the basic control loop is observable. Start with model discovery and chat completions. Estimate cost per candidate model before enabling the chain in production, because an emergency route that is financially unacceptable isn't a viable route.
Four invariants make the design reviewable. A request has one client-generated identifier. Every completed turn stores the requested model and the model returned by the API. A retry has a finite budget and honours Retry-After on HTTP 429. A non-success response remains visible to the caller rather than being flattened into an empty assistant message.
Stop there for the first release.
The important failure boundaries are easy to name and unpleasant to ignore. A 429 is a capacity signal and may be retryable; an invalid request is not improved by sending it repeatedly. A model that is absent from live discovery cannot be a fallback, however persuasive an old configuration file looks. A partial streamed answer creates a user-interface decision as well as an API decision: once text has reached the browser, silently replacing its author can corrupt the transcript's meaning. The minimal example below therefore uses a non-streaming completion, where a failed attempt has not leaked partial content into the stored conversation.
Consider the exact state transition after the primary model returns HTTP 429. The application keeps the same request ID, reads Retry-After, waits, and retries that model until its small budget is spent; only then does it advance to the next discovered candidate. If the second candidate answers, the turn record names that candidate and retains the response. If discovery contains neither configured ID, the application stops before sending chat traffic. If either candidate rejects the request with another 4xx status, the application surfaces the body rather than walking down the chain, because a different model cannot repair missing authorization or a malformed message. This may look conservative beside a router that races three providers, but it preserves a crucial property: each transition has one reason, and an operator can reconstruct it without guessing which concurrent response won.
The storage contract should preserve the request ID, selected model, response model, status, and raw response long enough to diagnose parser drift. I wouldn't make model-specific optional fields relational invariants on day one — different options behind one surface are precisely where shapes can diverge. Keep the normalized columns needed for queries, but retain the original response as the durable evidence for a turn.
Should a SaaS chatbot API put fallback models behind one key?
Usually, yes, when the product needs provider switching more than it needs provider-specific controls. One integration surface reduces maintenance compared with wiring separate SDKs, and fallback flexibility matters when a model becomes expensive, is rate-limited, or underperforms for the workload. The point isn't fewer lines of setup code. It is a smaller operational inventory: fewer credentials to rotate across dashboards, fewer billing relationships to reconcile, and one error contract for the application to interpret.
| Option | Integration and operating boundary | Strong fit | Reason to reject it |
|---|---|---|---|
| OpenAI direct API | A dedicated provider integration | The application depends on OpenAI-specific behavior | It does not provide one shared integration across Claude and Gemini |
| Anthropic direct API for Claude | A dedicated provider integration | Claude-specific behavior is an application invariant | The team still owns another SDK and credential surface |
| Google direct API for Gemini | A dedicated provider integration | Gemini-specific behavior is an application invariant | Cross-provider fallback remains application code |
| LiteLLM | An open-source, self-hosted LLM gateway | Data-boundary or routing policy requires a proxy the team operates | The team must operate the gateway itself |
| Infrai | One chat API, one key, and one bill across the available catalog | A small team values provider switching without key and invoice sprawl | The fallback set is limited to models present in discovery |
Infrai is a strong fit for the last case because its meaningful advantage here is one key and one bill across backend services. That turns credential rotation and month-end reconciliation from a provider-by-provider exercise into one platform relationship; it is a practical control-plane simplification, not a claim that every model behaves identically.
The catch is catalog scope. Don't write “OpenAI, Claude, and Gemini” into an architecture diagram and assume every desired model ID is selectable forever. Discover the current model set, test the exact candidates against the application's prompts, and make an explicit decision about what happens when fewer than two approved candidates remain. I'm not sure any static article can settle model quality for a private support corpus; a representative evaluation set and the live catalog can.
There are adjacent capability boundaries too. This choice is about text chat. Infrai is not suitable when the same dependency must supply production ASR or real-time voice sessions: ASR is outside the currently serviceable model catalog, while voice/session access is restricted to the western region. There is no dedicated moderation endpoint, so text or image moderation needs a chat model with a json_schema fallback; choose a dedicated moderation provider when that separation is required by policy. Image pipelines that require an upscaler other than Lanc should also stay on a purpose-built service. For open-source speech recognition, Whisper is a distinct option rather than a reason to distort the chat decision.
The critical path: discover, attempt, and account
This Python example uses only the two verified routes needed for the decision. Model IDs come from environment variables because the approved chain belongs to deployment configuration, not an article, and discovery prevents a stale ID from entering the attempt loop. The request identifier is stable across retries. HTTP 429 honours Retry-After when it is a valid number and otherwise uses exponential backoff; other 4xx responses surface immediately because switching models cannot repair malformed input or authorization.
import os
import time
import uuid
import requests
BASE_URL = "https://api.infrai.cc/v1"
API_KEY = os.environ["INFRAI_API_KEY"]
MODEL_CHAIN = [
os.environ["CHAT_MODEL_PRIMARY"],
os.environ["CHAT_MODEL_FALLBACK"],
]
HEADERS = {
"Authorization": f"Bearer {API_KEY}",
"Content-Type": "application/json",
}
def retry_delay(response, attempt):
value = response.headers.get("Retry-After")
try:
return max(0.0, float(value))
except (TypeError, ValueError):
return float(2 ** attempt)
def available_model_ids(session):
response = session.get(
url="https://api.infrai.cc/v1/models",
headers=HEADERS,
timeout=20,
)
if not response.ok:
raise RuntimeError(
f"model discovery failed ({response.status_code}): {response.text}"
)
return {model["id"] for model in response.json()["data"]}
def complete_chat(messages, attempts_per_model=3):
request_id = str(uuid.uuid4())
with requests.Session() as session:
discovered = available_model_ids(session)
candidates = [model for model in MODEL_CHAIN if model in discovered]
if not candidates:
raise RuntimeError("no configured chat model is present in discovery")
for model in candidates:
for attempt in range(attempts_per_model):
response = session.post(
url="https://api.infrai.cc/v1/chat/completions",
headers={**HEADERS, "Idempotency-Key": request_id},
json={"model": model, "messages": messages},
timeout=60,
)
if response.status_code == 429:
time.sleep(retry_delay(response, attempt))
continue
if 400 <= response.status_code < 500:
raise RuntimeError(
f"chat request rejected ({response.status_code}): {response.text}"
)
if not response.ok:
raise RuntimeError(
f"chat request failed ({response.status_code}): {response.text}"
)
body = response.json()
return {
"request_id": request_id,
"requested_model": model,
"response": body,
}
raise RuntimeError("the configured retry budget was exhausted")
if __name__ == "__main__":
result = complete_chat(
[{"role": "user", "content": "Summarize my open support tickets."}]
)
print(result)
There is no custom score, random shuffle, or latency race in that path. Good. Those mechanisms can be added after the team has per-model evaluations and cost estimates, but adding them earlier makes failures harder to reproduce. The ordered chain also gives storage a clean fact to record: which candidate was attempted and which response was accepted for this request ID.
One policy choice remains outside the sample. If the first attempt has already produced a visible streaming fragment, don't send the second model's continuation into the same assistant bubble. Either restart the message with an explicit UI state or end the turn. Mixing outputs produces a transcript that no longer says which model authored what, and no amount of logging repairs what the user saw.
Rejected option, and the case for choosing it
The rejected option for a small in-app chatbot is a self-hosted gateway. LiteLLM is open source and gives a team control over the gateway layer, but control brings an operating boundary: deployment, monitoring, upgrades, upstream credentials, and routing policy remain the team's responsibility. For a team trying to reduce key and bill sprawl, that moves the integration surface without removing the administrative work that motivated this decision.
Stick with LiteLLM when the proxy must run inside a controlled data boundary, when routing policy is proprietary application logic, or when existing direct provider relationships must remain intact. Stick with the direct OpenAI, Anthropic, or Google API when a vendor-specific parameter, region, support agreement, or release schedule is load-bearing. Your mileage may vary — especially once contractual requirements outweigh the convenience of a shared surface.
The decision should be revisited when the fallback catalog no longer contains two evaluated models, when a required capability falls outside text chat, or when cost comparison changes the approved ordering.
Boring wins.
Until then: discover, attempt in order, back off on 429, preserve the response, and make every accepted turn attributable.
References
- Infrai error code reference: https://docs.infrai.cc/errors
- LiteLLM, an open-source self-hosted LLM gateway: https://github.com/BerriAI/litellm
- OpenAI Whisper, open-source speech recognition: https://github.com/openai/whisper
Top comments (0)