How I Switched Between 5 LLM Providers Without Changing My Code

rolltok — Sat, 10 Oct 2026 11:27:47 +0000

Last Tuesday, I was debugging a production issue at 2pm when I got a 503 error from my LLM provider. Again.

This was the third time that week. My app was down for 15 minutes while I manually switched to a backup provider, updated environment variables, and redeployed. By the time I finished, I'd lost three customers.

I thought: there has to be a better way.

The Problem

Most LLM providers have their own API format. OpenAI uses one structure, Anthropic uses another, Google has yet another. If you want to switch providers (say, when one goes down), you need to rewrite your entire request logic.

Here's what I mean:

# OpenAI format
response = openai.ChatCompletion.create(
    model="gpt-4",
    messages=[{"role": "user", "content": "Hello"}]
)

# Anthropic format
response = anthropic.messages.create(
    model="claude-3-opus-20240229",
    messages=[{"role": "user", "content": "Hello"}]
)

Different SDKs. Different request structures. Different response formats. If you want to support multiple providers, you're writing adapter layers for each one.

What I Tried First

I started by building my own abstraction layer:

class LLMProvider:
    def __init__(self, provider_name):
        self.provider = provider_name

    def chat(self, message):
        if self.provider == "openai":
            return self._call_openai(message)
        elif self.provider == "anthropic":
            return self._call_anthropic(message)
        # ... 500 lines of adapter code later

This worked, but it was fragile. Every time a provider updated their API, I had to update my adapter. I was spending more time maintaining adapters than building features.

The Unified API Approach

Then I found a different approach: use a single API endpoint that's compatible with the OpenAI format, but routes to different providers behind the scenes.

Here's what the code looks like:

import os
import requests

api_key = os.getenv("LLM_API_KEY")

def chat(model, message):
    response = requests.post(
        "https://api.example.com/v1/chat/completions",
        headers={
            "Authorization": f"Bearer {api_key}",
            "Content-Type": "application/json"
        },
        json={
            "model": model,  # Just change this to switch providers
            "messages": [
                {"role": "user", "content": message}
            ]
        }
    )
    return response.json()["choices"][0]["message"]["content"]

# Use Qwen
print(chat("qwen3.7-plus", "Hello!"))

# Switch to DeepSeek - no code changes
print(chat("deepseek-chat", "Hello!"))

That's it. No adapter layers. No SDK switching. Just change the model name.

Handling Errors (Because APIs Fail)

Here's where things get tricky. APIs fail. A lot.

Last Thursday, I was testing this approach and got a 429 error (rate limit exceeded). My first thought was to just retry immediately. Bad idea. I got another 429. Then another. I was stuck in a retry loop.

Here's what actually worked:

import time

def chat_with_retry(model, message, max_retries=3):
    for attempt in range(max_retries):
        try:
            response = requests.post(
                "https://api.example.com/v1/chat/completions",
                headers={
                    "Authorization": f"Bearer {api_key}",
                    "Content-Type": "application/json"
                },
                json={
                    "model": model,
                    "messages": [{"role": "user", "content": message}]
                }
            )
            response.raise_for_status()
            return response.json()["choices"][0]["message"]["content"]

        except requests.exceptions.HTTPError as e:
            if response.status_code == 429:
                wait_time = 2 ** attempt  # Exponential backoff: 1s, 2s, 4s
                print(f"Rate limit hit. Waiting {wait_time}s...")
                time.sleep(wait_time)
            elif response.status_code == 401:
                print("Invalid API key. Check your credentials.")
                return None
            else:
                print(f"HTTP error: {e}")
                return None

    print("Max retries exceeded")
    return None

The key is exponential backoff. Don't retry immediately. Wait 1 second, then 2 seconds, then 4 seconds. This gives the API time to recover.

Streaming Responses

If you want real-time output (like ChatGPT), you need streaming. Here's how I did it:

def chat_stream(model, message):
    response = requests.post(
        "https://api.example.com/v1/chat/completions",
        headers={
            "Authorization": f"Bearer {api_key}",
            "Content-Type": "application/json"
        },
        json={
            "model": model,
            "messages": [{"role": "user", "content": message}],
            "stream": True
        },
        stream=True
    )

    for line in response.iter_lines():
        if line:
            # Parse Server-Sent Events format
            # Format: data: {"choices": [{"delta": {"content": "..."}}]}
            decoded = line.decode("utf-8")
            if decoded.startswith("data: "):
                data = decoded[6:]  # Remove "data: " prefix
                if data != "[DONE]":
                    import json
                    chunk = json.loads(data)
                    content = chunk["choices"][0]["delta"].get("content", "")
                    print(content, end="", flush=True)

This prints each token as it arrives, giving you that typewriter effect.

What I Learned

After two weeks of using this approach in production, here's what I've learned:

Unified APIs save time. I spent 3 days building my own adapter layer. I could have spent 3 hours integrating a unified API.
Error handling is critical. Don't just retry immediately. Use exponential backoff. Log errors. Monitor rate limits.
Streaming is worth it. Users prefer seeing tokens appear in real-time rather than waiting 10 seconds for a full response.
Model switching should be trivial. If you need to change 50 lines of code to switch providers, your abstraction is too complex.

Full Disclosure

I'm now working on a project called RollTok, which is a unified API gateway for multiple LLM providers. The code examples above are based on what I learned while building it. If you want to try it out, you can get an API key at rolltok.com.

But honestly, the approach works with any unified API. The specific service doesn't matter as much as the pattern.

What's Next

I'm planning to write about:

How to track token usage across multiple providers
How to implement request/response logging
How to set up connection pooling for better performance

If you have questions or suggestions, drop them in the comments.

DEV Community: rolltok