DEV Community

Cover image for LLM Model Fallback in TypeScript
LakBud
LakBud

Posted on AI-assisted

LLM Model Fallback in TypeScript

OpenAI returns a 503. Every LLM call in your app fails with it, and your users stare at a spinner while the status page catches up.

Model fallback is the fix everyone reaches for: if the primary model fails, send the call to a backup model on another provider. It sounds like a try/catch. In practice there are four details that decide whether it helps or makes things worse.

This post walks through those details and shows how to set up LLM fallback in TypeScript with VernLLM, using GPT-6.2 Sol as the primary and Gemini 4 Neon as the backup. No gateway, no proxy: it runs in your process.

Why a try/catch fallback is not enough

try {
  return await openai.chat.completions.create({ model: 'gpt-6.2-sol', messages });
} catch {
  return await gemini.models.generateContent({ model: 'gemini-4-neon', contents });
}
Enter fullscreen mode Exit fullscreen mode

This is where most fallback code starts. It has four problems:

  1. It gives up on the primary too early. A single 503 is often a blip. You want a few retries with backoff on the primary before you switch providers.
  2. It falls over on everything. If the model returned JSON that fails your schema, or the user cancelled the request, a second provider will not fix it. You just pay twice.
  3. It breaks streaming. If the stream already sent half an answer, falling over splices a second model's output into text the user is reading.
  4. It hides who answered. Your metrics, costs and debugging all need to know which provider and model produced the response.

LLM fallback with VernLLM

npm i vern-llm openai @google/genai
Enter fullscreen mode Exit fullscreen mode
import OpenAI from 'openai';
import { GoogleGenAI } from '@google/genai';
import { VernLLM, metaRef } from 'vern-llm';
import { fromOpenAI, fromGemini } from 'vern-llm/adapters';

const llm = new VernLLM({
  client: fromOpenAI(new OpenAI({ apiKey: process.env.OPENAI_API_KEY })),
  model: 'gpt-6.2-sol',
  name: 'openai',
  maxRetries: 2,

  fallback: [
    {
      client: fromGemini(new GoogleGenAI({ apiKey: process.env.GEMINI_API_KEY })),
      model: 'gemini-4-neon',
      name: 'gemini',
    },
  ],

  onEvent: (event) => {
    if (event.kind === 'fallback') console.warn(`fallback ${event.from} -> ${event.to}`);
  },
});

const meta = metaRef();
const summary = await llm.call({ userContent: 'Summarize the earnings call.', meta });
Enter fullscreen mode Exit fullscreen mode

fallback is an ordered list. The order is the whole policy: VernLLM never reorders or scores targets behind your back, so the primary is always tried first.

Each target is independent. It gets its own retry loop, its own timeout, and its own circuit breaker and rate limiter if you configure them. Retry and timeout options a target leaves out inherit from the primary.

What a failover looks like

I ran this setup with a primary that always returns 503 and maxRetries: 2. The primary got 3 attempts, then the fallback event fired:

fallback openai -> gemini
Enter fullscreen mode Exit fullscreen mode

and meta.current reported who answered:

{
  provider: 'gemini',
  model: 'gemini-4-neon',
  fallbackIndex: 0,     // -1 when the primary answers
  usedFallback: true,
  attempts: 1,
  position: 1
}
Enter fullscreen mode Exit fullscreen mode

Your code gets a normal result. The outage shows up in your logs and metrics, not in front of your users.

When to fall back, and when to stop

After a target's own retries run out, VernLLM asks a policy whether to move to the next target. The default, defaultFallbackOn, moves on for failures a different provider can actually fix and stops for the rest:

Error Decision Why
5xx, timeout, network error next The provider is sick, another one may not be
circuit open next This target is known to be down, skip it right away
unsupported capability next Another adapter may support it
invalid JSON or schema validation stop Another model given the same prompt likely fails the same way
caller aborted stop The user cancelled, respect it
model refused stop Opt in with your own policy if you want another provider to try

In my run I also pointed the primary at a model that returned { "summary": 42 } against a Zod schema that wanted a string. The call stopped with a validation error and the backup received zero requests.

You can wrap the default instead of replacing it:

import { defaultFallbackOn } from 'vern-llm';

const llm = new VernLLM({
  // ...
  fallbackOn: (error, context) => {
    if (error.type === 'circuit_open' && context.isLastTarget) {
      alerting.page('every configured provider is unavailable');
    }
    return defaultFallbackOn(error, context);
  },
});
Enter fullscreen mode Exit fullscreen mode

Streaming: fall back before the first chunk only

With stream: true, fallback applies when a stream fails to open: a 5xx or a timeout before the first chunk. Once a chunk has reached your user, a failure is final for that call. VernLLM will not splice a second model's answer into one the user already started reading.

Track fallback usage

onUsage reports provider and usedFallback on every successful call, so you can see how often you are running on the backup:

onUsage: (usage) => {
  metrics.increment('llm.call', { provider: usage.provider });
  if (usage.usedFallback) metrics.increment('llm.fallback_used', { provider: usage.provider });
},
Enter fullscreen mode Exit fullscreen mode

Pick the order per call

targets chooses which targets to try for one call, and in what order:

await llm.call({ userContent: '...', targets: ['gemini', 'openai'] });
Enter fullscreen mode Exit fullscreen mode

If you want the order chosen from health, latency or cost, the router extension does that as middleware. Core only runs the list it is given.

Wrap up

Model fallback is more than catching an error. Retry the primary first, fall over only for failures another provider can fix, never fall over mid stream, and always record who answered.

npm i vern-llm
Enter fullscreen mode Exit fullscreen mode

Docs: Multi provider fallback

Top comments (0)