Add Multi-Model Fallback to an OpenAI App Without Rewriting It

Most OpenAI integrations start with a single model:

const response = await client.chat.completions.create({
  model: "gpt-5.4-mini",
  messages,
});

That is perfectly reasonable until the model becomes temporarily unavailable, reaches a rate limit, or takes too long to respond.

At that point, many applications either return an error to the user or grow a collection of provider-specific SDKs and adapters.

There is a simpler option when the models are available through the same OpenAI-compatible API: keep the application interface unchanged and move the fallback decision into a small wrapper.

In this tutorial, we will use TokenBay, where I work as a Developer Advocate. TokenBay exposes GPT, Claude, Gemini, Qwen, DeepSeek, and other models through one API key and one OpenAI-compatible Base URL.

The fallback logic itself stays in our application. We are not going to hide provider behavior or blindly retry every error.

What we are building

The wrapper will:

1. Send the request to a primary model.
2. Apply a timeout to that attempt.
3. Move to the next model only when the failure is retryable.
4. Stop immediately for authentication, permission, or invalid-request errors.
5. Return the model that actually completed the request.

Our example fallback order will be:

gpt-5.4-mini
      ↓
gemini-2.5-flash
      ↓
claude-haiku-4.5

All three calls use the same SDK client, Base URL, API key, and message format.

Set up the project

Create a new Node.js project:

mkdir multi-model-fallback
cd multi-model-fallback
npm init -y
npm install openai dotenv

Add the following field to package.json:

{
  "type": "module"
}

Create a .env file:

TOKENBAY_API_KEY=your_api_key_here
TOKENBAY_BASE_URL=https://api.tokenbay.com/v1

Do not commit this file. Add it to .gitignore:

.env
node_modules

Create one OpenAI-compatible client

Create fallback.js:

import "dotenv/config";
import OpenAI from "openai";

if (!process.env.TOKENBAY_API_KEY) {
  throw new Error("TOKENBAY_API_KEY is not configured");
}

const client = new OpenAI({
  apiKey: process.env.TOKENBAY_API_KEY,
  baseURL:
    process.env.TOKENBAY_BASE_URL ??
    "https://api.tokenbay.com/v1",
});

const DEFAULT_MODELS = [
  "gpt-5.4-mini",
  "gemini-2.5-flash",
  "claude-haiku-4.5",
];

The important part is that the rest of the application still talks to an OpenAI-style chat.completions.create() interface.

Changing models does not require importing another provider SDK.

Decide which failures should trigger fallback

Not every error should move to another model.

A 429 rate-limit response, 503 unavailable response, or upstream timeout may be resolved by trying another model.

A 401 authentication failure will not. Neither will a malformed request.

Add the following functions:

const RETRYABLE_STATUS_CODES = new Set([
  429,
  500,
  502,
  503,
  504,
]);

function getStatus(error) {
  return typeof error?.status === "number"
    ? error.status
    : undefined;
}

function isTimeout(error) {
  return (
    error?.name === "AbortError" ||
    error?.code === "ETIMEDOUT"
  );
}

function isRetryable(error) {
  const status = getStatus(error);

  return (
    isTimeout(error) ||
    RETRYABLE_STATUS_CODES.has(status)
  );
}

function describeError(error) {
  return {
    status: getStatus(error) ?? null,
    code: error?.code ?? null,
    message: error?.message ?? "Unknown error",
  };
}

This distinction matters.

If the request contains an unsupported parameter, sending the same invalid payload to two more models only creates more failed requests. If the API key is invalid, fallback cannot repair it.

Implement the fallback wrapper

Now add the main function:

export async function createChatCompletion({
  messages,
  models = DEFAULT_MODELS,
  timeoutMs = 30_000,
  temperature = 0.2,
}) {
  if (!Array.isArray(messages) || messages.length === 0) {
    throw new TypeError(
      "messages must be a non-empty array"
    );
  }

  if (!Array.isArray(models) || models.length === 0) {
    throw new TypeError(
      "models must be a non-empty array"
    );
  }

  const attempts = [];

  for (const model of models) {
    const startedAt = Date.now();

    try {
      const response =
        await client.chat.completions.create(
          {
            model,
            messages,
            temperature,
          },
          {
            signal: AbortSignal.timeout(timeoutMs),
          }
        );

      attempts.push({
        model,
        ok: true,
        durationMs: Date.now() - startedAt,
      });

      return {
        response,
        model,
        attempts,
      };
    } catch (error) {
      const retryable = isRetryable(error);

      attempts.push({
        model,
        ok: false,
        retryable,
        durationMs: Date.now() - startedAt,
        error: describeError(error),
      });

      console.warn(
        JSON.stringify({
          event: "llm_attempt_failed",
          model,
          retryable,
          ...describeError(error),
        })
      );

      if (!retryable) {
        const failure = new Error(
          `Non-retryable LLM request failure on ${model}`
        );

        failure.cause = error;
        failure.attempts = attempts;

        throw failure;
      }
    }
  }

  const failure = new Error(
    `All ${models.length} models failed`
  );

  failure.attempts = attempts;

  throw failure;
}

Each model gets one bounded attempt.

The wrapper records:

• which model was called;
• whether it succeeded;
• how long it took;
• whether the failure was retryable;
• the status, code, and message returned by the API.

It also returns the model that produced the final response. That makes fallback visible to the rest of the application instead of silently pretending the primary model handled the request.

Call it from the application

Create app.js:

import { createChatCompletion } from "./fallback.js";

const messages = [
  {
    role: "system",
    content:
      "Answer clearly and keep the response under 100 words.",
  },
  {
    role: "user",
    content:
      "Explain why idempotency matters in API retries.",
  },
];

try {
  const result = await createChatCompletion({
    messages,
    models: [
      "gpt-5.4-mini",
      "gemini-2.5-flash",
      "claude-haiku-4.5",
    ],
    timeoutMs: 30_000,
  });

  console.log("Model used:", result.model);
  console.log(
    "Answer:",
    result.response.choices[0].message.content
  );
  console.log("Attempts:", result.attempts);
} catch (error) {
  console.error(error.message);
  console.error(error.attempts ?? []);
  process.exitCode = 1;
}

Run it with:

node app.js

A successful primary request might produce:

Model used: gpt-5.4-mini
Attempts: [
  {
    model: "gpt-5.4-mini",
    ok: true,
    durationMs: 1842
  }
]

If the first model is temporarily unavailable, the result could look like:

Model used: gemini-2.5-flash
Attempts: [
  {
    model: "gpt-5.4-mini",
    ok: false,
    retryable: true,
    durationMs: 301,
    error: {
      status: 503,
      code: null,
      message: "Service unavailable"
    }
  },
  {
    model: "gemini-2.5-flash",
    ok: true,
    durationMs: 973
  }
]

The user still gets an answer, while the application retains evidence that fallback occurred.

Why this is not just retry logic

Retrying the same model and switching models solve different problems.

A retry can help with a brief network interruption. A model fallback can help when a particular route is rate-limited, overloaded, or unavailable for longer than the application can wait.

In production, I usually separate the two decisions:

Request
  │
  ├─ temporary network failure
  │      └─ retry the same model once
  │
  ├─ rate limit or model unavailable
  │      └─ try the next compatible model
  │
  └─ invalid request or authentication failure
         └─ stop and surface the error

The example above keeps things intentionally small by giving every model one attempt. A production version can add one same-model retry before moving to the next model, but the total attempt count should remain bounded.

Compatibility still needs testing

An OpenAI-compatible API standardizes the request interface. It does not make every model behaviorally identical.

Before placing models in the same fallback chain, test the features your application relies on:

• tool calling;
• structured JSON output;
• image input;
• system-message handling;
• maximum context length;
• streaming events;
• token limits;
• safety behavior.

For a plain text-generation request, switching models may be straightforward.

For an agent that depends on a particular tool schema, the fallback model must pass the same contract tests as the primary model. Availability is not useful if the replacement model cannot complete the job correctly.

Do not replay side effects blindly

Fallback becomes more dangerous when the model can trigger external actions.

Imagine that the first attempt sends a customer email but times out before your application receives the response. Repeating the complete workflow with another model could send the email twice.

For tool-using applications:

• separate planning from execution;
• assign an operation ID to every user action;
• store tool results before starting another model attempt;
• use idempotency keys where the downstream API supports them;
• never replay completed side effects automatically.

The model request may be safe to repeat. The real-world action may not be.

Measure fallback instead of hiding it

A fallback that happens once a month may be useful resilience.

A fallback that handles half of production traffic is evidence that the primary-model choice, timeout, rate limit, or request design needs attention.

At minimum, track:

llm_attempt_total{model, status}
llm_attempt_duration_ms{model}
llm_fallback_total{from_model, to_model}
llm_request_failure_total{status}

Also record the model used with each application result. Otherwise, debugging a quality change becomes difficult when two users receive answers from different models.

The practical tradeoff

Multi-model fallback does not make an LLM application failure-proof.

It gives the application another controlled option before returning an error, without requiring a separate SDK integration for every provider.

Using one OpenAI-compatible endpoint makes the mechanical part small:

same client
same API key
same message format
different model ID

The engineering work is deciding:

• which failures justify fallback;
• which models are actually compatible with the task;
• how many attempts the user can wait for;
• how fallback affects cost and output quality;
• which actions are unsafe to repeat.

That policy belongs in application code, where it can be tested, logged, and changed deliberately.

If you want to try the example with GPT, Claude, Gemini, or other supported models, create a key and check the current model IDs in the TokenBay model catalog.