Every guide on running an LLM app in production gives the same advice: add a fallback. If the main model times out or the vendor has an outage, send the request to another model so your users still get an answer. Good advice. What the guides rarely say is that the fallback isn't free insurance. While it's active, it bills at its own price list, and it may not even accept the requests your main model handles.
I ran the numbers on five common pairs, and the result is simple: pick a fallback by its price ratio and its context window, not by "the strongest model we have".
A few words first
A token is a chunk of text the model reads as one piece; APIs bill per million of them. Input tokens are what you send, output tokens are what the model writes back, and output costs more. The context window is the largest request, in tokens, a model will accept; anything bigger is rejected. The primary model is the one you normally call. A fallback is the model your code calls when the primary fails.
The setup
A typical call here is 1,500 input tokens and 300 output tokens, a chatbot reply with some history. The app makes 10,000 calls a day, 30 days a month. Prices are official list prices from the catalog of the LLM gateway I work on, snapshot of 16 November 2026, with no discounts.
| Primary → fallback | Cost per call | Fallback multiplier | Window |
|---|---|---|---|
| Claude Haiku 5.5 → GPT-5.6 mini | $0.00030 → $0.00066 | 2.2x | 1M → 400K |
| Claude Haiku 5.5 → GPT-5.6 | $0.00030 → $0.00660 | 22x | 1M → 400K |
| GPT-5.6 mini → Claude Haiku 5.5 | $0.00066 → $0.00030 | 0.45x | 400K → 1M |
| Claude Sonnet 5.5 → GPT-5.6 | $0.00600 → $0.00660 | 1.1x | 1M → 400K |
| GPT-5.6 → Claude Sonnet 5.5 | $0.00660 → $0.00600 | 0.91x | 400K → 1M |
The second row is the one people actually write. The primary is a small, cheap model because the task is easy, and the fallback is the big model "to be safe". On a normal day that app costs $3. On a day the primary is down, the same traffic costs $66.
What an outage does to the month
A month on Claude Haiku 5.5 alone costs $90. One full day on the GPT-5.6 fallback turns that into $153, which is 1.70x. You didn't ship anything or gain a single user; the bill grew because of somebody else's outage.
It doesn't take a full day, either. Timeouts and rate-limit errors send individual requests to the fallback all the time. If 1% of calls fall back, the month grows by 21%. If 4.8% fall back, the bill doubles. That share is easy to reach without noticing, because a fallback that works is silent: the user gets an answer and nothing shows up in your error logs.
Compare the first row. Same primary, but the fallback is the other vendor's small model. One outage day there costs $6.60 instead of $3, and 5% of calls on the fallback add 6% to the month. Pairs of similar size from different vendors (rows 4 and 5) barely move the bill at all, within 10% in either direction.
The window problem
The price is the part you can see on a pricing page. The window is the part that fails quietly. Claude Haiku 5.5 and Claude Sonnet 5.5 accept up to 1M tokens; GPT-5.6 and its mini version accept 400K. Any request between those two sizes works on the primary and is rejected by the fallback. A 600,000-token request on Claude Sonnet 5.5 costs $1.20 and succeeds; during an outage, it has nowhere to go.
These are exactly the requests you least want to lose: long documents, long agent runs, big code reviews. Going the other direction is fine. A fallback with a bigger window accepts everything the primary did.
Checking both numbers in code
The fix is to make the fallback decision explicit. Before rerouting, check that the request fits the fallback's window and that the call won't cost more than a ratio you chose in advance. If neither candidate passes, fail with an error you can see.
MODELS = { # $ per 1M tokens (input, output), context window
"claude-haiku-5.5": (0.10, 0.50, 1_000_000),
"gpt-5.6-mini": (0.20, 1.20, 400_000),
"gpt-5.6": (2.00, 12.00, 400_000),
}
def cost(model, tokens_in, tokens_out):
p_in, p_out, _ = MODELS[model]
return (tokens_in * p_in + tokens_out * p_out) / 1e6
def pick_fallback(primary, candidates, tokens_in, tokens_out, max_ratio=3):
budget = cost(primary, tokens_in, tokens_out) * max_ratio
for m in candidates:
fits = tokens_in + tokens_out <= MODELS[m][2]
if fits and cost(m, tokens_in, tokens_out) <= budget:
return m
return None # fail loudly instead of paying any price
print(pick_fallback("claude-haiku-5.5", ["gpt-5.6", "gpt-5.6-mini"], 1_500, 300))
print(pick_fallback("claude-haiku-5.5", ["gpt-5.6", "gpt-5.6-mini"], 600_000, 300))
The first call prints gpt-5.6-mini: GPT-5.6 costs 22x the primary, more than the 3x limit, so it's skipped. The second prints None, because a 600K request fits neither fallback. Then count how often the fallback fires, the same way you count errors. That number is part of your bill.
What this doesn't solve
A cheaper fallback isn't the same model. Answers will differ in style and sometimes in quality, and a prompt tuned for one vendor can behave differently on another. I didn't measure quality here, only price and window.
Token counts aren't portable either. Each vendor has its own tokenizer, so the same text is a different number of tokens on Claude and on GPT. The window check in the code is an estimate; leave a margin.
The 1,500/300 call and the outage shares are my assumptions, not measurements. Your ratio of input to output changes the multipliers a little, though not the order of the rows.
Finally, a gateway can route around an outage automatically, but it can't decide how much an outage is worth to you. The ratio limit in that function is a business decision, and you have to make it yourself.
Top comments (0)