Anyone running an LLM feature for more than a few months has a fallback path: a second provider, a router, or a forty-line wrapper that tries the next model when the first one returns 529 or 503. The design work usually stops at "if the primary fails, use the secondary." That is the easy half.
The hard half is what happens to the secondary at the exact moment the primary trips.
Failover is a synchronised event
When the primary starts returning overload errors, every client fails over within the same few seconds. The secondary does not receive a trickle; it receives its normal traffic plus the entire primary's traffic, arriving as one burst, with retry loops attached.
If the secondary was sized for its own load plus a small margin, it was sized for a number that never occurs during an incident. The result is a familiar pattern: the primary degrades, the fallback immediately degrades, and the incident looks like "both providers are down" when in fact the second one is being DDoS'd by its own customer.
Two numbers make this concrete:
burst = normal_secondary_load + primary_load x failover_fraction
peak = burst x (1 + retry_multiplier)
With a secondary at 30% of primary capacity and a 2x retry multiplier, the fallback sees roughly 2.6x what it handles on a good day, in the first minute.
Three things that actually help
1. Treat Retry-After as authoritative, and jitter everything else. 503 and 529 carry the header inconsistently, so read it when present, and when absent use exponential backoff with jitter. Unjittered backoff turns a synchronised burst into a synchronised second burst, one interval later.
2. Reserve headroom on the secondary instead of treating it as free capacity. Either keep it deliberately under-subscribed, or give fallback traffic a lower concurrency ceiling than primary traffic so it cannot consume the whole pool. A fallback you can oversubscribe is a shared-fate design, not a resilience one.
3. Instrument failover rate per provider, not just error rate. Error rate tells you what is failing now. Failover rate tells you how much of your traffic has already moved, which is the number that predicts whether the fallback is about to become the incident. In my experience it leads the user-visible symptom by minutes, which is the only window where you can act.
The uncomfortable implication
A fallback is not a spare tire. It is a second system that has to absorb a correlated spike at exactly the moment its sibling is unhealthy. Designing it as identical-but-idle guarantees it will be the first thing to fall over, and it will do so while every dashboard still shows the primary as the problem.
The cheap version of all this is a single line in the runbook: when the primary trips, cap fallback concurrency before you touch anything else. The expensive version is discovering it during the incident.
Related, since these are the same bill from different angles: the three numbers that decide an agent bill (retry rate, the cap, merge-slot yield) and the KV cache transfer cost of splitting prefill and decode.
Top comments (0)