Designing Failover for LLM Agents: When GPT Goes Down, Claude Shouldn't Notice

Your AI agent just called GPT-4o. OpenAI has an outage. Your agent crashes. Customers notice.

Here's the failover pattern we use in production:

# Simple failover: try provider A, fall back to B
import requests

MODELS = [
    {"provider": "openai", "model": "gpt-4o"},
    {"provider": "anthropic", "model": "claude-3-5-sonnet"},
]

def call_with_failover(messages):
    for provider in MODELS:
        try:
            return call_model(provider, messages)
        except (requests.Timeout, requests.ConnectionError, Exception):
            log.warning(f"{provider} failed, trying next")
            continue
    raise Exception("All providers down")

Why not just retry?

Retries don't help when the entire upstream is down. You need circuit breaking: if OpenAI has failed 3 times in a row, stop calling it for 60 seconds and route traffic to Claude/Gemini instead.

Where to put the logic

You can build this into your app... or you can put it in a gateway in front of your models. Same OpenAI SDK, same code, but the gateway handles:

• Health checks on each upstream
• Automatic routing to a healthy fallback
• Retries with exponential backoff
• Per-request logs so you know which model actually responded

from openai import OpenAI

client = OpenAI(
    api_key="your-gateway-key",
    base_url="https://api.agentoken.co/v1"  # gateway handles failover
)

# If gpt-4o is down, gateway auto-routes to claude
resp = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": "Hi"}]
)

Key lesson

Don't build failover into every agent you ship. Put it once in a gateway, and every agent you build after that inherits it for free.

Agent Token runs this kind of routing + failover out of the box, so you don't have to babysit model outages.