ny application that depends on a single large language model provider inherits that provider's bad days. Providers have outages, impose rate limits during traffic spikes, return transient server errors, and occasionally slow to a crawl under load. If your architecture has exactly one path to a model, every one of those events becomes your outage too. High availability for LLM applications is the discipline of making sure that a problem with one model or provider does not become a problem for your users.
The tools for this are retries, fallbacks, health detection, and ordered failover chains. They are often lumped together under "resilience," but they are distinct mechanisms that solve distinct problems, and using them well means understanding where each one applies. This article walks through those mechanisms and assembles them into a reference failover design. We use the configuration model of an LLM gateway as the running example, since a gateway is the natural place to implement failover once rather than in every application.
Retries and fallbacks are not the same thing
The first distinction to get right is between a retry and a fallback, because they respond to different kinds of failure.
A retry re-sends the request to the same target. It is the right response to a transient failure, the kind that is likely to succeed if you simply try again a moment later: a brief rate-limit, a momentary server hiccup, a connection that dropped. The target itself is fundamentally fine; the request just happened to hit a bad instant.
A fallback stops retrying the current target and sends the request to a different target. It is the right response to a failure that retrying will not fix: the target is down, is persistently rejecting the request, or is returning an error that indicates a real problem rather than a transient one. Hammering a dead target with retries only adds latency before the inevitable; a fallback moves on to something that can actually answer.
A well-designed system uses both in sequence. It retries a target a bounded number of times to ride out transient trouble, and if the target still will not cooperate, it falls back to the next target. The configuration below shows exactly this pattern, which we will return to throughout the article.
A reference failover configuration
routing_config:
type: priority-based-routing
load_balance_targets:
- target: azure/gpt-4o
priority: 0
retry_config:
attempts: 3
delay: 200
on_status_codes: ["429", "500", "503"]
fallback_status_codes: ["429", "500", "502", "503"]
fallback_candidate: true
- target: openai/gpt-4o
priority: 1
retry_config:
attempts: 2
delay: 100
fallback_status_codes: ["429"]
fallback_candidate: true
- target: anthropic/claude-sonnet
priority: 2
fallback_candidate: false
This single block encodes a complete failover policy across three providers. Read top to bottom, it says: prefer the Azure deployment; retry it up to three times on specific transient codes; if it still fails on a fallback-worthy code, move to the OpenAI deployment; retry that twice; and finally, if needed, fall through to the Anthropic deployment. The rest of the article explains each piece.
Retry configuration in detail
Retries are configured per target through retry_config, and the defaults are deliberately cautious. By default a target gets zero retry attempts, with a 100-millisecond delay between attempts, retrying on the status codes 429, 500, 502, and 503. Zero attempts as a default means retries are opt-in: you turn them on where you want them, rather than having the system silently retry everywhere.
Each part of the configuration controls one aspect of retry behavior. The attempts field sets how many times to retry the same target. The delay field sets the pause between attempts in milliseconds, which gives a struggling target a brief moment to recover rather than slamming it instantly. The on_status_codes field restricts retries to the response codes where retrying makes sense, typically rate-limit and server-error codes.
In the reference configuration, the Azure target retries up to three times with a 200-millisecond delay, but only on 429, 500, and 503. That last restriction matters: there is no point retrying a 404, because the resource will still be missing on the second attempt. Choosing which codes to retry is really a judgment about which failures are transient, and encoding that judgment in on_status_codes keeps the system from wasting time on errors that retrying cannot fix.
Fallback configuration in detail
Fallbacks are governed by two settings. The first, fallback_status_codes, defines the response codes that should stop retries on the current target and trigger a move to a different target. Its defaults are 401, 403, 404, 408, 429, 500, 502, and 503, a broad set covering authentication and authorization failures, missing resources, timeouts, rate limits, and server errors. When a target returns one of these codes, the gateway stops working on it and fails over.
The second setting, fallback_candidate, controls whether a target is allowed to receive fallback traffic from other targets. It defaults to true, meaning a target can serve as a backup for others. Setting it to false, as the reference configuration does for the Anthropic target, is worth pausing on. In that configuration the Anthropic target has fallback_candidate: false, which means it is the lowest-priority target and will be reached through the ordinary priority chain, but it will not be pulled in as a fallback destination outside that chain. This lets you keep a target in the lineup while controlling precisely when it is used.
Together these two settings separate two questions that are easy to conflate: when should I give up on a target, answered by fallback_status_codes, and which targets am I willing to fail over to, answered by fallback_candidate.
Health detection and automatic recovery
Retries and fallbacks handle individual failed requests. Health detection handles the pattern across many requests, so the gateway can stop sending traffic to a target that is reliably failing rather than discovering the failure one request at a time.
The rule is concrete: a target is marked unhealthy after it accumulates at least two failures within a two-minute window. Once unhealthy, the target is moved to the end of the line and is used only if the healthy targets fail first. It is not removed entirely, because it might recover, but it stops being a preferred destination. This shields users from a degraded target: instead of a stream of requests each independently discovering the target is down, the gateway learns the lesson once and routes around it.
Recovery is automatic. You do not have to intervene or flip a switch to bring a target back; as it starts succeeding again, the gateway restores it to normal rotation. This matters operationally because provider problems are usually temporary, and a system that required manual re-enablement would leave you either babysitting it or running permanently degraded after every blip.
SLA cutoffs: failing over on slowness
Not every failure is an error code. A target can accept your request, return a valid response, and still fail you by being far too slow. Priority-based routing addresses this with SLA cutoffs, which let you mark a target unhealthy when its speed crosses a threshold you set on time-per-output-token or time-to-first-token.
This extends the notion of health from "is it returning errors" to "is it meeting my latency requirements." For an interactive application, a model that takes many seconds to produce its first token may be effectively unusable even though it never errors. An SLA cutoff lets the gateway treat that slowness as a trigger to fail over to a faster target, rather than forcing users to wait on a technically-successful but practically-useless response.
Assembling the reference design
Putting the pieces together, a robust failover design built on priority-based routing works like this. Requests go to the healthy target with the lowest priority number, where 0 is highest. That target retries on transient codes to absorb brief trouble. If it returns a fallback-worthy code, the gateway stops and moves to the next priority. Targets that repeatedly fail, or that violate an SLA cutoff, are marked unhealthy, pushed to the end of the line, and recovered automatically when they improve. The fallback_candidate flag lets you decide which targets may serve as backups beyond their place in the chain.
The result is layered defense. Transient failures are absorbed by retries without the user noticing. Persistent single-target failures are handled by fallback to the next provider. Recurring failures are handled by health detection, which stops wasting requests on a known-bad target. And slowness, not just errors, is handled by SLA cutoffs. No single provider's bad day reaches your users, because at every layer there is a defined next step. Designing for high availability, in the end, is just making sure there is always a next step and that the system takes it without you.
Key takeaways
A retry re-sends the request to the same target for transient failures; a fallback moves to a different target when retrying will not help. Use both in sequence.
Retries are configured per target with retry_config; defaults are cautious (0 attempts, 100ms delay, retrying on 429/500/502/503), so retries are opt-in and limited to codes worth retrying via on_status_codes.
Fallbacks use fallback_status_codes (defaults 401/403/404/408/429/500/502/503) to decide when to abandon a target, and fallback_candidate (default true) to decide which targets may receive fallback traffic.
Health detection marks a target unhealthy after at least two failures within two minutes, moves it to the end of the line, uses it only if healthy targets fail, and recovers it automatically.
SLA cutoffs extend health to speed, marking a target unhealthy when time-per-output-token or time-to-first-token crosses a threshold, so you can fail over on slowness, not just errors.
A priority-based chain combining retries, fallbacks, health detection, and SLA cutoffs gives layered defense so that no single provider's outage reaches your users.
Top comments (0)