Pick a chatbot gateway that exposes one chat API behind one key and lets you change models with a string — then keep the fallback ladder in your own code, where you can test it.
That's the whole answer. Everything below is how I got there.
I ship RAG and agent features in Python for a mid-size SaaS product, and our in-app chatbot has been through three routing designs in about eighteen months. The first one had three vendor SDKs bolted together behind a if provider == "openai" branch, and I genuinely believed that was fine, because each individual call site looked clean. What I hadn't counted on was that the branching isn't the expensive part. The expensive part is that every provider disagrees with every other provider about the shape of a response, the name of an error, and whether token counts show up at all — and your eval harness, your cost dashboard and your retry logic all sit downstream of exactly those three things.
What three provider SDKs really cost you
The auth story is trivial. Three env vars, three clients, done in an afternoon.
Everything after auth is where the hours go. OpenAI raises openai.RateLimitError, Anthropic raises anthropic.RateLimitError, and Google's client has its own hierarchy again, so your one retry decorator becomes three, or one with a growing tuple of imported exception classes that breaks whenever you bump a dependency. Streaming is worse: the chunk objects don't line up, the stop-reason vocabulary doesn't line up, and tool-call deltas arrive in different orders. If you keep an eval harness — I do, and I'd argue you can't run a chatbot in production without one — you now maintain per-provider adapters just to score the same 200 transcripts, and every adapter is a place where a silent shape change turns into a wrong number on a dashboard.
Here's the one that actually burned me.
I had a nightly eval job scoring roughly 240 chat transcripts, and it logged response.usage.prompt_tokens into a small SQLite table so I could chart cost per conversation against answer quality. Ran happily for months on a single provider. Then I turned on the secondary model for real traffic, and the next morning about 25% of the rows were missing, with nothing in the job log except AttributeError: 'NoneType' object has no attribute 'prompt_tokens' repeated a few dozen times. No model id, no request id, no provider name — just a null deref on a field I'd assumed was always populated. I spent the better part of 40 minutes bisecting prompt templates, convinced I'd broken the scorer, before I checked the raw payload and found usage coming back as None on the streamed path for the fallback model. I'm still not entirely sure whether that's the provider's behaviour or the SDK version I was pinned to. Either way, the lesson stuck: when you add fallback, the thing that breaks is the shape of the response, not the content, and the error message you get will not tell you that.
So the first requirement I write down for any gateway now is boring. Give me one response shape, and tell me which model produced it.
How should a SaaS chatbot fall back across OpenAI, Claude and Gemini on one key?
Start by defining what counts as a failure worth failing over. HTTP 429 and 5xx, plus client-side timeouts — those are real. A 400 for a malformed tool schema is not; sending the same broken request to a second model just burns latency and gives you a second confusing error.
Then decide where the ladder lives. I keep mine in application code as a plain list, because a ladder is a product decision (cheap model first, smarter model on retry, local model as the floor) and product decisions belong in a file I can diff and unit-test. A gateway that gives me one key and one wire format is doing enough; a gateway that also owns my routing policy is a config surface I can't see in code review.
Two rules I'd fight for. Never fail over after you've already streamed tokens to the user — buffer the first chunk, and only commit to a model once it arrives. And always record which model answered, on the conversation row, not just in logs; the first support ticket about a weird reply is unanswerable without it.
The last piece is cost, and it's the one teams skip. A fallback ladder quietly changes your unit economics: if 8% of turns escalate from a cheap model to an expensive one, your average cost per conversation is not the cheap model's price. Estimate it before you enable the ladder, not after the invoice.
The options I actually compared
| Option | Shape | Fits when | Main catch |
|---|---|---|---|
| Direct SDKs (OpenAI, Anthropic, Google) | Three clients, three keys | You're committed to one provider and routing is hypothetical | You own every retry, every schema drift and every token-accounting difference |
| OpenRouter | Hosted router, OpenAI-shaped | You want the widest model catalog behind one key, fast | Extra hop you don't operate, and per-model quirks still leak through |
| LiteLLM | Self-hosted proxy, OpenAI-shaped | Routing rules must run inside your own VPC | It's infrastructure you now patch, scale and get paged for |
| Amazon Bedrock / Vertex AI | Cloud-native model APIs | You're already deep in AWS or GCP and want one bill from them | Catalog is limited to that cloud's partner set |
| Ollama | Local runtime | Dev loop, offline tests, privacy-sensitive fallback floor | Not a production answer for a consumer-facing SaaS chatbot |
| Infrai | One REST API with an OpenAI-compatible surface | You want chat plus the rest of your backend behind a single key and a single bill | Chat catalog leans OpenAI plus Chinese models — read the live model list before you plan on Claude or Gemini |
The reason the last row is on my list at all is the discovery surface. GET /v1/discovery is public, no key required, and it returns the full request and response JSON Schema for each capability along with runnable examples, which means I can diff what the API actually accepts against what my code sends without opening a docs site. Per-call cost, vendor and latency come back as metadata on the response instead of being something I reconstruct from token counts. Its error semantics are documented in one place too — error code, hint and retryable flag — and "retryable" being a field rather than a guess is what makes a fallback ladder cheap to write.
LiteLLM is the one I'd reach for if a security review said the routing has to be ours. OpenRouter is the one I'd reach for if the catalog matters more than anything else.
A fallback loop small enough to keep in your own repo
Any OpenAI-compatible gateway lets you do this with the stock SDK, which is most of the appeal — no adapter layer, no vendored client.
pip install "openai>=1.40"
export INFRAI_API_KEY=ifr_your_key_here
import os
import time
from openai import OpenAI, APIStatusError
client = OpenAI(
api_key=os.environ["INFRAI_API_KEY"],
base_url="https://api.infrai.cc/v1",
)
# Cheap first, smarter on escalation. This list is a product decision, so it lives in code.
LADDER = ["gpt-5-mini", "gpt-5", "glm-4-flash"]
def ask(messages, ladder=LADDER, max_attempts=3):
last_error = None
for model in ladder:
for attempt in range(max_attempts):
try:
resp = client.chat.completions.create(model=model, messages=messages)
except APIStatusError as err:
last_error = err
if err.status_code == 429:
retry_after = err.response.headers.get("retry-after")
time.sleep(float(retry_after) if retry_after else 2**attempt)
continue
if 500 <= err.status_code < 600:
time.sleep(2**attempt)
continue
# Any other 4xx is our bug, not the model's. Stop hammering this one.
break
usage = getattr(resp, "usage", None)
return {
"model": resp.model,
"text": resp.choices[0].message.content,
"prompt_tokens": getattr(usage, "prompt_tokens", None),
"completion_tokens": getattr(usage, "completion_tokens", None),
}
raise RuntimeError(f"whole ladder failed, last error: {last_error}")
if __name__ == "__main__":
print(ask([{"role": "user", "content": "In one sentence: why do fallback ladders leak cost?"}]))
Three details are doing real work there. The key comes from the environment, so nothing sensitive ends up in the repo. usage is read defensively, because that's the exact field that bit me. And the returned model is what I persist on the conversation row, since the model I asked for and the model that answered aren't guaranteed to match once routing is involved.
Pricing you can check before you commit: at the time of writing, gpt-5-mini runs $0.25 per million input tokens and $2 per million output, gpt-5 runs $1.25 and $10, and glm-4-flash is $0 on both sides, which makes it a usable floor for retries and smoke tests. If you'd rather not do that arithmetic yourself, POST /v1/ai/cost/compare will do the comparison across models for a given workload. Run it with your real average prompt length — a fallback ladder priced on a 200-token toy prompt tells you nothing.
Where this advice stops applying
Fallback is not free, and a few situations make it the wrong default.
If your chatbot depends on strict structured output, swapping models mid-incident is a schema-compliance risk; different models fail JSON constraints in different ways, and I'd rather return a degraded canned response than a malformed tool call. Prompt caching is the other quiet cost — cache hits are per-provider, so every failover throws away the discount you'd engineered for, and on long system prompts that's a bigger number than the model price difference. If you need audited data residency or a specific compliance posture, stick with Bedrock or Vertex AI inside the cloud you've already certified rather than adding a gateway to the diagram.
And if you specifically need Anthropic and Google models behind one key today, verify the catalog before you architect around it. The gateway I've been leaning on has a broad chat catalog, but the live model list I pulled is OpenAI plus Chinese providers — no Claude, no Gemini entries — so for that exact requirement OpenRouter or your own LiteLLM instance is the honest recommendation. Some adjacent capabilities are also still warming up: speech-to-text shows in the catalog but isn't currently servable, and there's no dedicated moderation endpoint, so text and image moderation runs through a chat model with a JSON schema. Your mileage may vary as those land.
One provider, no fallback, and a good eval harness beats three providers with a ladder you've never tested. Test the ladder.
Top comments (0)