Every LLM API fails. Not "might fail." Fails.
OpenAI returns 429s during peak hours. Anthropic has regional outages. Google's Gemini API occasionally decides your key isn't authorized for the model you've been calling all week. A provider you depend on goes down for 45 minutes and your agent pipeline — the one you told your team was "production-ready" — sits there dead, mid-task, with no idea what happened.
You already know retry logic. This article is about what comes after retry: how to build a multi-model fallback system that keeps your application running when the primary model is unreachable, and how to avoid the mistakes that make fallback worse than just failing.
I'll show real Python patterns, the gotchas that break them in practice, and how to test your fallback without waiting for an actual outage.
Disclosure up front: I work on daoxe, a multi-model API gateway. The patterns in this article work against any provider — official APIs, third-party gateways, or self-hosted endpoints. Use them wherever your key works.
1. Why retry alone isn't enough
The simplest thing that sometimes works:
import time
import openai
def call_with_retry(model: str, messages: list, max_retries: int = 3):
for attempt in range(max_retries):
try:
return openai.chat.completions.create(
model=model, messages=messages
)
except openai.APIError as e:
if attempt == max_retries - 1:
raise
time.sleep(2 ** attempt)
This handles transient network blips and brief rate limiting. It does not handle:
- Sustained outages. If the provider is down for 30 minutes, exponential backoff just means your caller eventually raises an exception after 14 seconds instead of immediately. Your agent is still dead.
- Quota exhaustion. Your key hits its monthly limit. Retrying won't fix it. It'll just burn your error budget faster.
-
Model deprecation. The provider sunsets the model version you pinned. Every call fails with
model_not_found. Retry spins uselessly. - Regional blocks. The endpoint is up — just not from where your server lives. The client sees a timeout and retries into the same wall.
Retry is the first layer. Fallback is the second.
2. The model-level fallback pattern (simple, with sharp edges)
The basic idea: if model A fails, try model B.
import os
from openai import OpenAI
FALLBACK_CHAIN = [
{"model": "gpt-5.5", "base_url": "https://api.openai.com/v1", "key_env": "OPENAI_API_KEY"},
{"model": "claude-sonnet-4-6", "base_url": "https://api.anthropic.com/v1/messages",
"key_env": "ANTHROPIC_API_KEY", "protocol": "anthropic"},
{"model": "gemini-2.5-pro", "base_url": "https://generativelanguage.googleapis.com/v1beta",
"key_env": "GEMINI_API_KEY", "protocol": "gemini"},
]
def call_with_fallback(messages: list, chain: list = FALLBACK_CHAIN):
last_error = None
for cfg in chain:
try:
client = OpenAI(
base_url=cfg["base_url"],
api_key=os.environ[cfg["key_env"]],
timeout=30.0,
)
return client.chat.completions.create(
model=cfg["model"], messages=messages
)
except Exception as e:
last_error = e
continue
raise RuntimeError(
f"All {len(chain)} models failed. Last error: {last_error}"
)
This works — until it doesn't. Here's where it breaks.
Gotcha 1: Protocol incompatibility
Anthropic's API uses the Messages protocol, not Chat Completions. Calling it through an OpenAI-shaped client with base_url=https://api.anthropic.com/v1/messages produces a 404 or a garbled error. You need a protocol adapter, or you need to use a gateway that translates between protocols.
The honest fix: either maintain separate client code paths per provider, or use a gateway that presents every model behind a single protocol. Both approaches work. The single-protocol path means fewer branches to test.
Gotcha 2: The system prompt disappears
Some models handle system prompts differently. Claude native uses a top-level system parameter. OpenAI uses a system role in the messages array. Gemini puts system instructions in a systemInstruction config field. When you fall back across providers, your carefully crafted system prompt might arrive as a user message — or not at all.
Test this explicitly. Send a message that asks the model to quote the system prompt verbatim. If it can't, your fallback is silently dropping instructions.
Gotcha 3: Tool calling breaks between families
Anthropic and OpenAI have different tool calling schemas. The function definitions that work on GPT may produce tool_use blocks on Claude that your parser doesn't recognize. If your agent depends on structured tool calls, cross-provider fallback is risky without a normalization layer.
For agent pipelines that use tool calling heavily, fallback within the same model family first — GPT family before Claude family before Gemini — and only cross families when the task doesn't require structured output.
3. Routing by capability, not just availability
A smarter fallback system doesn't blindly try models in a fixed order. It selects the fallback based on what the task actually needs.
CAPABILITY_PROFILES = {
"reasoning": ["claude-sonnet-4-6", "gpt-5.5", "gemini-2.5-pro"],
"coding": ["claude-sonnet-4-6", "gpt-5.5", "deepseek-v4-pro"],
"fast_chat": ["claude-haiku-4-5", "gpt-5.4", "gemini-3.5-flash"],
"long_context": ["gemini-2.5-pro", "claude-sonnet-4-6", "gpt-5.5"],
}
MODEL_CONFIGS = {
"claude-sonnet-4-6": {"base_url": "...", "key_env": "...", "protocol": "anthropic"},
"gpt-5.5": {"base_url": "...", "key_env": "...", "protocol": "openai"},
# ... etc
}
def call_by_capability(messages: list, capability: str):
candidates = CAPABILITY_PROFILES.get(capability, [])
if not candidates:
raise ValueError(f"Unknown capability: {capability}")
last_error = None
for model_id in candidates:
cfg = MODEL_CONFIGS[model_id]
try:
client = OpenAI(base_url=cfg["base_url"], api_key=os.environ[cfg["key_env"]])
return client.chat.completions.create(model=model_id, messages=messages)
except Exception as e:
last_error = e
continue
raise RuntimeError(f"All {capability} models failed: {last_error}")
Now your code declares intent — "I need long-context reasoning" — and the dispatcher picks appropriate models. This is more resilient than a fixed order, and it makes your code self-documenting about what each call actually requires.
4. The gateway pattern: one endpoint, many models
Managing separate API keys, base URLs, and protocol adapters for a half-dozen providers is operational friction. The alternative is a single endpoint that presents many models behind one API surface:
# One base URL. One key. All models available.
client = OpenAI(
base_url="https://api.daoxe.com/v1",
api_key=os.environ["DAOXE_API_KEY"],
)
FALLBACK_CHAIN = [
"claude-sonnet-4-6", # primary
"gpt-5.5", # fallback 1
"gemini-2.5-pro", # fallback 2
]
def call_single_endpoint(messages: list, chain: list = FALLBACK_CHAIN):
last_error = None
for model in chain:
try:
return client.chat.completions.create(
model=model, messages=messages, timeout=30.0
)
except Exception as e:
last_error = e
continue
raise RuntimeError(f"All models in chain failed: {last_error}")
The advantage is not having to manage per-provider credentials, protocol quirks, or billing relationships. The trade-off is that the gateway itself becomes a single point of failure — if the gateway goes down, all models behind it become unreachable. (Mitigations: monitor the gateway's health separately, and keep a direct-to-provider key as an escape hatch for critical paths.)
5. Circuit breaker: when even fallback should stop
Blindly cycling through fallback models under load can make things worse. If your primary model is returning 429s because you're over your rate limit, hammering the fallback models immediately is antisocial — and it might exhaust your quota on all providers in a cascade.
A circuit breaker stops calling a model after N consecutive failures and only re-enables it after a cooldown.
import time
from collections import defaultdict
class CircuitBreaker:
def __init__(self, failure_threshold: int = 5, cooldown_seconds: float = 60.0):
self.failure_threshold = failure_threshold
self.cooldown_seconds = cooldown_seconds
self._failures: dict[str, int] = defaultdict(int)
self._open_until: dict[str, float] = {}
def allow(self, model: str) -> bool:
if model in self._open_until and time.time() < self._open_until[model]:
return False
return True
def record_failure(self, model: str):
self._failures[model] += 1
if self._failures[model] >= self.failure_threshold:
self._open_until[model] = time.time() + self.cooldown_seconds
def record_success(self, model: str):
self._failures[model] = 0
self._open_until.pop(model, None)
Wire it into the fallback dispatcher:
breaker = CircuitBreaker(failure_threshold=3, cooldown_seconds=30.0)
def call_with_breaker(messages: list, chain: list):
for model in chain:
if not breaker.allow(model):
continue # circuit open, skip
try:
result = client.chat.completions.create(model=model, messages=messages)
breaker.record_success(model)
return result
except Exception as e:
breaker.record_failure(model)
continue
raise RuntimeError("All models unavailable (circuit open or failed)")
Now when a model goes down repeatedly, the system stops trying it for 30 seconds, giving it time to recover without wasting your remaining quota on guaranteed failures.
6. Observability: you can't fix what you can't see
A fallback system without instrumentation is a debugging nightmare. When a call succeeds on the third model in the chain, you need to know that it happened and why the first two failed.
At minimum, log:
import logging
import json
logger = logging.getLogger("llm_fallback")
def call_instrumented(messages: list, chain: list):
for i, model in enumerate(chain):
try:
start = time.monotonic()
result = client.chat.completions.create(model=model, messages=messages)
elapsed = time.monotonic() - start
if i > 0:
logger.warning(
"fallback_used",
extra={
"primary": chain[0],
"actual": model,
"fallback_depth": i,
"latency_s": round(elapsed, 3),
"prompt_tokens": result.usage.prompt_tokens,
"completion_tokens": result.usage.completion_tokens,
},
)
return result
except Exception as e:
logger.error(
"model_call_failed",
extra={"model": model, "attempt": i, "error": str(e)},
)
continue
logger.critical("all_models_exhausted", extra={"chain": chain})
raise RuntimeError("All models in chain failed")
Critical metric to track over time: fallback rate (what percentage of calls hit a fallback model). If this creeps above 1-2%, your primary provider has a reliability problem. If it spikes suddenly, something is broken. Either way, you want to know before your users do.
7. Testing your fallback without waiting for an outage
The worst time to discover your fallback system is broken is during an actual outage. Test it artificially.
Method 1: The unreachable endpoint
Point your primary model at a dead URL and verify the fallback fires:
# Set primary to a port where nothing listens
PRIMARY_BASE_URL="http://localhost:19999/v1" python test_fallback.py
Method 2: The error-injection proxy
Write a tiny proxy that returns errors for a percentage of requests:
# error_proxy.py — run with: python error_proxy.py
# Proxies requests to your real endpoint, but returns 503 for ~30% of them.
from http.server import HTTPServer, BaseHTTPRequestHandler
import random
class ErrorProxy(BaseHTTPRequestHandler):
def do_POST(self):
if random.random() < 0.3:
self.send_response(503)
self.end_headers()
self.wfile.write(b'{"error": {"message": "injected failure", "type": "server_error"}}')
return
# ... proxy to real endpoint
Run your application against this proxy. If the fallback doesn't fire within a few requests, something is misconfigured.
Method 3: The key rotation test
Rotate your primary API key to an invalid value and verify the system fails over to the next model. This catches the case where your fallback is configured but the credentials are wrong — a failure mode that's silent until the primary actually goes down.
8. The honest limitations
No fallback system is transparent to the caller. Here's what you actually trade off:
Latency. Each fallback attempt adds connection + inference time. A fast call that normally takes 800ms can take 3+ seconds if the first two models fail. For real-time applications, set a total timeout and surface to the user that a fallback is in progress.
Cost unpredictability. Fallback models may have different pricing. Your monthly bill can spike if your primary model has a bad week and all traffic shifts to a more expensive fallback. Track spend per model and set alerts when fallback-model spend exceeds a threshold.
Behavioral drift. Different models produce different outputs. A system prompt that Claude handles elegantly might produce aggressive refusals on Gemini. A chain-of-thought instruction that works on GPT may be ignored by DeepSeek. Test your fallback models with your actual prompts before depending on them.
Determinism is gone. Even with temperature=0, different models produce different outputs. If your application depends on deterministic or near-deterministic behavior, cross-model fallback fundamentally changes the semantics of the call. Your fallback is keeping the system running, not keeping it identical.
There is no zero-dependency option. A fallback system that spans multiple providers means you depend on all of them — their uptime, their pricing changes, their model deprecations, their API migrations. Your operational surface area multiplies.
9. What to do right now
If you take nothing else from this article, do these three things today:
Identify which LLM calls in your codebase are single-model and single-provider. Anywhere you hardcode
model="gpt-5.5"with no alternative path is a single point of failure. Count them.Pick one non-critical path and add a two-model fallback chain. Not your payment processing. Not your auth flow. A summarization step, a classification call, a "suggest alternative phrasing" feature. Ship it, instrument it, watch the logs for a week. Get comfortable with the pattern before you apply it to critical paths.
Add a health-check endpoint that probes your primary model every 60 seconds and exposes the result. A simple
/health/llmthat calls your primary model with a trivial prompt and returns{"status": "ok", "latency_ms": 420}or{"status": "degraded", "error": "timeout"}. Wire this into whatever monitoring you already use. You want to know about provider degradation before your users do — and before your fallback system has to prove itself under fire.
The goal of a fallback system is not perfection. It's to turn a hard failure — your agent crashes, your pipeline stops, your user sees an error — into a soft degradation the caller might not even notice. That's worth the complexity.
If you liked this, you might also want llm-honesty-probe (github.com/seven7763/llm-honesty-probe), an open-source tool that checks whether your API endpoint is actually serving the model it claims. Fallback + verification = you're covered on both reliability and trust.
Top comments (0)