"Just add a fallback model" is the kind of advice that sounds complete until you've actually shipped it. A single backup model handles one failure mode (the primary is down) and quietly ignores the other four that show up in production: slow responses, malformed output, rate limits, and a fallback model that doesn't accept the same context shape as your primary.
Here's the pattern breakdown that actually holds up.
The five patterns, and when each one is the right call
| Pattern | What it does | Best for | Main risk |
|---|---|---|---|
| Retry | Same model, again, after a delay | Timeouts, brief 5xx, short rate-limit windows | Too many retries just adds latency |
| Fallback routing | Different model or endpoint | Primary is down or unhealthy | Output can differ meaningfully from the primary |
| Load balancing | Spreads traffic across healthy paths | High-volume traffic | Behavior varies slightly by which path you land on |
| Caching | Reuses a prior response | Repeated FAQs, deterministic tasks | Stale or wrong cache hits if governance is loose |
| Semantic routing | Routes by meaning/intent, not just a static model name | Agents, multi-domain apps | Misclassification routes the request badly |
None of these replace the others. A production system usually needs some combination: check cache, check model health, classify the request, pick a route, validate the output, log the decision. Skip the validation step and you'll eventually route a mangled response straight into a downstream tool call.
The part that actually bites: model context
This is the failure mode that doesn't show up until you've already shipped the "add a fallback" version. Model context, meaning system prompt, conversation history, retrieved documents, tool schemas, output format rules, isn't guaranteed to survive a switch between models cleanly. Things that quietly differ:
- Context length limits
- How system prompts get interpreted
- Tool/function schema expectations
- JSON formatting strictness
- Streaming behavior
OpenAI-style and Anthropic-style APIs use genuinely different request shapes (OpenAI reference, Anthropic Messages API). An "OpenAI-compatible" gateway smooths over the request/response format, it does not guarantee the fallback model behaves the same way given the same input. That distinction matters and is easy to skip past when you're just trying to get failover shipped.
For agents specifically, this compounds
A single user task can trigger planning, tool selection, summarization, and a final response, each a separate model call. If any one of those fails without a sane retry or fallback path, the whole task can collapse. Worth building in from the start:
- Retry limits with backoff and jitter (not unlimited retries)
- Short fallback chains, long chains just add latency without adding reliability
- Health checks tracking timeouts, 5xx, and 429 patterns specifically
- Output validation before anything downstream (a tool call, a database write) touches the response
- Logging per route: which model, latency, token usage, whether a fallback fired
Where a router fits into this
An AI gateway or model router centralizes the routing/retry/cache logic instead of every service reimplementing its own version. GonkaRouter is one example, an OpenAI- and Anthropic-compatible endpoint currently routing to MiniMax-M2.7, Kimi-K2.6, and GLM-5.2, built on the Gonka decentralized compute network. Worth being precise here: "compatible" means the request/response format matches, not that every model behaves identically to what you'd get from OpenAI or Anthropic directly, that distinction is exactly the model-context problem above, and no gateway makes it disappear on its own.
If you're evaluating something like this, the actual test is: change only the endpoint in an existing integration, run real prompts through each supported model, and measure latency, output quality, and failure behavior before you decide where your app-level retry and fallback logic needs to live.
Quick checklist before shipping failover
- [ ] Defined behavior for timeout, 5xx, 429, and invalid JSON, not just "the request failed"
- [ ] Retry limits with backoff, not unbounded retries
- [ ] Fallback chains kept short
- [ ] Context transformation handled explicitly between models, not assumed
- [ ] Caching scoped with real invalidation rules, not just "cache everything"
- [ ] Every route logged: model, latency, tokens, fallback fired or not
- [ ] Output validated before it reaches a tool call or downstream write
Curious what others here are using for the context-transformation step specifically, that's the part I've seen bite people hardest when a fallback model quietly handles system prompts differently than the primary.
Top comments (1)
Strong breakdown, especially the distinction between API compatibility and behavioral compatibility. Normalizing an OpenAI- or Anthropic-shaped request does not make prompts, tool schemas, or structured outputs semantically portable across models.
I would add three production concerns to the pattern set: circuit breakers, end-to-end deadline propagation, and idempotency. If the primary is already unhealthy, retrying it on every request before falling back only adds latency and can create a retry storm. Likewise, retrying an agent step after a timeout can duplicate a downstream side effect unless tool executions carry idempotency keys and persisted state.
For context transformation, we treat the provider-neutral representation as more than messages: normalized roles and content blocks, tool schemas, completed tool-call IDs, prompt/schema versions, remaining latency and cost budgets, plus replayability metadata for each step. Provider-specific adapters are generated only at the edge.
That does not make models behaviorally identical, but it makes failover observable, testable, and much less likely to repeat work.