OpenAI returns a 503. Every LLM call in your app fails with it, and your users stare at a spinner while the status page catches up.
Model fallback is the fix everyone reaches for: if the primary model fails, send the call to a backup model on another provider. It sounds like a try/catch. In practice there are four details that decide whether it helps or makes things worse.
This post walks through those details and shows how to set up LLM fallback in TypeScript with VernLLM, using GPT-6.2 Sol as the primary and Gemini 4 Neon as the backup. No gateway, no proxy: it runs in your process.
Why a try/catch fallback is not enough
try {
return await openai.chat.completions.create({ model: 'gpt-6.2-sol', messages });
} catch {
return await gemini.models.generateContent({ model: 'gemini-4-neon', contents });
}
This is where most fallback code starts. It has four problems:
- It gives up on the primary too early. A single 503 is often a blip. You want a few retries with backoff on the primary before you switch providers.
- It falls over on everything. If the model returned JSON that fails your schema, or the user cancelled the request, a second provider will not fix it. You just pay twice.
- It breaks streaming. If the stream already sent half an answer, falling over splices a second model's output into text the user is reading.
- It hides who answered. Your metrics, costs and debugging all need to know which provider and model produced the response.
LLM fallback with VernLLM
npm i vern-llm openai @google/genai
import OpenAI from 'openai';
import { GoogleGenAI } from '@google/genai';
import { VernLLM, metaRef } from 'vern-llm';
import { fromOpenAI, fromGemini } from 'vern-llm/adapters';
const llm = new VernLLM({
client: fromOpenAI(new OpenAI({ apiKey: process.env.OPENAI_API_KEY })),
model: 'gpt-6.2-sol',
name: 'openai',
maxRetries: 2,
fallback: [
{
client: fromGemini(new GoogleGenAI({ apiKey: process.env.GEMINI_API_KEY })),
model: 'gemini-4-neon',
name: 'gemini',
},
],
onEvent: (event) => {
if (event.kind === 'fallback') console.warn(`fallback ${event.from} -> ${event.to}`);
},
});
const meta = metaRef();
const summary = await llm.call({ userContent: 'Summarize the earnings call.', meta });
fallback is an ordered list. The order is the whole policy: VernLLM never reorders or scores targets behind your back, so the primary is always tried first.
Each target is independent. It gets its own retry loop, its own timeout, and its own circuit breaker and rate limiter if you configure them. Retry and timeout options a target leaves out inherit from the primary.
What a failover looks like
I ran this setup with a primary that always returns 503 and maxRetries: 2. The primary got 3 attempts, then the fallback event fired:
fallback openai -> gemini
and meta.current reported who answered:
{
provider: 'gemini',
model: 'gemini-4-neon',
fallbackIndex: 0, // -1 when the primary answers
usedFallback: true,
attempts: 1,
position: 1
}
Your code gets a normal result. The outage shows up in your logs and metrics, not in front of your users.
When to fall back, and when to stop
After a target's own retries run out, VernLLM asks a policy whether to move to the next target. The default, defaultFallbackOn, moves on for failures a different provider can actually fix and stops for the rest:
| Error | Decision | Why |
|---|---|---|
| 5xx, timeout, network error | next | The provider is sick, another one may not be |
| circuit open | next | This target is known to be down, skip it right away |
| unsupported capability | next | Another adapter may support it |
| invalid JSON or schema validation | stop | Another model given the same prompt likely fails the same way |
| caller aborted | stop | The user cancelled, respect it |
| model refused | stop | Opt in with your own policy if you want another provider to try |
In my run I also pointed the primary at a model that returned { "summary": 42 } against a Zod schema that wanted a string. The call stopped with a validation error and the backup received zero requests.
You can wrap the default instead of replacing it:
import { defaultFallbackOn } from 'vern-llm';
const llm = new VernLLM({
// ...
fallbackOn: (error, context) => {
if (error.type === 'circuit_open' && context.isLastTarget) {
alerting.page('every configured provider is unavailable');
}
return defaultFallbackOn(error, context);
},
});
Streaming: fall back before the first chunk only
With stream: true, fallback applies when a stream fails to open: a 5xx or a timeout before the first chunk. Once a chunk has reached your user, a failure is final for that call. VernLLM will not splice a second model's answer into one the user already started reading.
Track fallback usage
onUsage reports provider and usedFallback on every successful call, so you can see how often you are running on the backup:
onUsage: (usage) => {
metrics.increment('llm.call', { provider: usage.provider });
if (usage.usedFallback) metrics.increment('llm.fallback_used', { provider: usage.provider });
},
Pick the order per call
targets chooses which targets to try for one call, and in what order:
await llm.call({ userContent: '...', targets: ['gemini', 'openai'] });
If you want the order chosen from health, latency or cost, the router extension does that as middleware. Core only runs the list it is given.
Wrap up
Model fallback is more than catching an error. Retry the primary first, fall over only for failures another provider can fix, never fall over mid stream, and always record who answered.
npm i vern-llm
Docs: Multi provider fallback
Top comments (0)