Relying on a single LLM provider for a production application is risky. API rate limits, temporary outages, and unexpected latency spikes can bring down your application without warning. When your primary provider goes offline, your users shouldn't be stuck staring at a loading spinner. You need a system that automatically routes requests to backup providers like Claude or Gemini, without breaking your agent's state or changing your output format.
Here is how to build a lightweight, model-agnostic fallback handler in Python to keep your agents running smoothly.
Before starting, make sure you have:
- Python 3.10 or higher installed.
- API keys for OpenAI, Anthropic, and Google Gemini.
- The provider SDKs installed (
openai,anthropic,google-genai).
If you are building complex systems, setting up robust AI agent development patterns early saves you dozens of hours in maintenance down the road.
Step 1: Normalize Prompts and Outputs
Different providers expect different payload formats. OpenAI uses a messages list with role and content. Anthropic uses a similar system but handles system prompts as a top-level parameter. Gemini uses its own content structure.
Create a single standard format for your application.
from dataclasses import dataclass
from typing import List, Dict, Optional
@dataclass
class ChatMessage:
role: str # "system", "user", or "assistant"
content: str
@dataclass
class LLMResponse:
content: str
provider: str
model: str
Step 2: Build Generic Provider Wrappers
Wrap each client in a standard interface. Each function should accept your ChatMessage list and return your LLMResponse.
import os
from openai import OpenAI
from anthropic import Anthropic
import google.genai as genai
class ProviderAdapter:
def __init__(self):
self.openai_client = OpenAI(api_key=os.getenv("OPENAI_API_KEY"))
self.anthropic_client = Anthropic(api_key=os.getenv("ANTHROPIC_API_KEY"))
self.gemini_client = genai.Client(api_key=os.getenv("GEMINI_API_KEY"))
def call_openai(self, messages: List[ChatMessage], model="gpt-4o") -> LLMResponse:
formatted = [{"role": m.role, "content": m.content} for m in messages]
res = self.openai_client.chat.completions.create(model=model, messages=formatted)
return LLMResponse(
content=res.choices[0].message.content,
provider="openai",
model=model
)
def call_claude(self, messages: List[ChatMessage], model="claude-3-5-sonnet-20241022") -> LLMResponse:
system_prompt = ""
user_messages = []
for m in messages:
if m.role == "system":
system_prompt += m.content + "\n"
else:
user_messages.append({"role": m.role, "content": m.content})
res = self.anthropic_client.messages.create(
model=model,
max_tokens=1024,
system=system_prompt.strip(),
messages=user_messages
)
return LLMResponse(
content=res.content[0].text,
provider="anthropic",
model=model
)
def call_gemini(self, messages: List[ChatMessage], model="gemini-2.5-flash") -> LLMResponse:
# Flatten system and user messages into a simple text prompt for basic compatibility
prompt_text = "\n".join([f"{m.role}: {m.content}" for m in messages])
res = self.gemini_client.models.generate_content(
model=model,
contents=prompt_text
)
return LLMResponse(
content=res.text,
provider="gemini",
model=model
)
Step 3: Write the Fallback Router Loop
Now build the fallback router. It loops through your preferred order of models. If one fails due to an API error, timeout, or rate limit, it catches the exception, logs a warning, and moves to the next model immediately.
import logging
logging.basicConfig(level=logging.INFO)
class ResilientAgent:
def __init__(self):
self.adapter = ProviderAdapter()
# Define fallback hierarchy: (provider_name, model_name)
self.fallback_chain = [
("openai", "gpt-4o"),
("anthropic", "claude-3-5-sonnet-20241022"),
("gemini", "gemini-2.5-flash")
]
def execute(self, messages: List[ChatMessage]) -> LLMResponse:
for provider, model in self.fallback_chain:
try:
logging.info(f"Attempting execution with {provider} ({model})...")
if provider == "openai":
return self.adapter.call_openai(messages, model=model)
elif provider == "anthropic":
return self.adapter.call_claude(messages, model=model)
elif provider == "gemini":
return self.adapter.call_gemini(messages, model=model)
except Exception as e:
logging.warning(f"Provider {provider} failed with error: {e}. Trying fallback...")
raise RuntimeError("All LLM providers failed. Check API status and keys.")
This pattern is useful when building mission critical system integration and workflow automation, where uptime is mandatory for daily business operations.
Step 4: Test Your Fallback Flow
You can run a quick test script to make sure the fallback triggers when the primary provider encounters an issue.
if __name__ == "__main__":
agent = ResilientAgent()
conversation = [
ChatMessage(role="system", content="You are a helpful assistant."),
ChatMessage(role="user", content="Explain runtime fallbacks in one sentence.")
]
# Simulating a failure by supplying a bogus key for OpenAI in env variables
# will force the agent to skip OpenAI and hit Claude or Gemini automatically.
try:
response = agent.execute(conversation)
print("\n--- Success ---")
print(f"Provider Used: {response.provider}")
print(f"Model Used: {response.model}")
print(f"Response: {response.content}")
except Exception as err:
print(f"Execution failed completely: {err}")
Expected Outcomes
When you run this architecture, your logs will show the system attempting your primary provider first. If OpenAI hits a 500 server error or a 429 rate limit, the loop catches the exception in milliseconds and switches to Claude or Gemini.
The calling application receives a standard LLMResponse regardless of which vendor actually processed the request. Your downstream data parsing remains identical, keeping your application stable even during major cloud vendor outages.
If you need help scaling this pattern across multi-agent systems or enterprise infrastructure, the team at Gaper provides senior engineers who specialize in production ready AI systems.
Top comments (0)