How I Switched Between 5 LLM Providers Without Changing My Code

Last Tuesday, I was debugging a production issue at 2pm when I got a 503 error from my LLM provider. Again.

This was the third time that week. My app was down for 15 minutes while I manually switched to a backup provider, updated environment variables, and redeployed. By the time I finished, I'd lost three customers.

I thought: there has to be a better way.

The Problem

Most LLM providers have their own API format. OpenAI uses one structure, Anthropic uses another, Google has yet another. If you want to switch providers (say, when one goes down), you need to rewrite your entire request logic.

Here's what I mean:

# OpenAI format
response = openai.ChatCompletion.create(
    model="gpt-4",
    messages=[{"role": "user", "content": "Hello"}]
)

# Anthropic format
response = anthropic.messages.create(
    model="claude-3-opus-20240229",
    messages=[{"role": "user", "content": "Hello"}]
)

Different SDKs. Different request structures. Different response formats. If you want to support multiple providers, you're writing adapter layers for each one.

What I Tried First

I started by building my own abstraction layer:

class LLMProvider:
    def __init__(self, provider_name):
        self.provider = provider_name

    def chat(self, message):
        if self.provider == "openai":
            return self._call_openai(message)
        elif self.provider == "anthropic":
            return self._call_anthropic(message)
        # ... 500 lines of adapter code later

This worked, but it was fragile. Every time a provider updated their API, I had to update my adapter. I was spending more time maintaining adapters than building features.

The Unified API Approach

Then I found a different approach: use a single API endpoint that's compatible with the OpenAI format, but routes to different providers behind the scenes.

Here's what the code looks like:

import os
import requests

api_key = os.getenv("LLM_API_KEY")

def chat(model, message):
    response = requests.post(
        "https://api.example.com/v1/chat/completions",
        headers={
            "Authorization": f"Bearer {api_key}",
            "Content-Type": "application/json"
        },
        json={
            "model": model,  # Just change this to switch providers
            "messages": [
                {"role": "user", "content": message}
            ]
        }
    )
    return response.json()["choices"][0]["message"]["content"]

# Use Qwen
print(chat("qwen3.7-plus", "Hello!"))

# Switch to DeepSeek - no code changes
print(chat("deepseek-chat", "Hello!"))

That's it. No adapter layers. No SDK switching. Just change the model name.

Handling Errors (Because APIs Fail)

Here's where things get tricky. APIs fail. A lot.

Last Thursday, I was testing this approach and got a 429 error (rate limit exceeded). My first thought was to just retry immediately. Bad idea. I got another 429. Then another. I was stuck in a retry loop.

Here's what actually worked:

import time

def chat_with_retry(model, message, max_retries=3):
    for attempt in range(max_retries):
        try:
            response = requests.post(
                "https://api.example.com/v1/chat/completions",
                headers={
                    "Authorization": f"Bearer {api_key}",
                    "Content-Type": "application/json"
                },
                json={
                    "model": model,
                    "messages": [{"role": "user", "content": message}]
                }
            )
            response.raise_for_status()
            return response.json()["choices"][0]["message"]["content"]

        except requests.exceptions.HTTPError as e:
            if response.status_code == 429:
                wait_time = 2 ** attempt  # Exponential backoff: 1s, 2s, 4s
                print(f"Rate limit hit. Waiting {wait_time}s...")
                time.sleep(wait_time)
            elif response.status_code == 401:
                print("Invalid API key. Check your credentials.")
                return None
            else:
                print(f"HTTP error: {e}")
                return None

    print("Max retries exceeded")
    return None

The key is exponential backoff. Don't retry immediately. Wait 1 second, then 2 seconds, then 4 seconds. This gives the API time to recover.

Streaming Responses

If you want real-time output (like ChatGPT), you need streaming. Here's how I did it:

def chat_stream(model, message):
    response = requests.post(
        "https://api.example.com/v1/chat/completions",
        headers={
            "Authorization": f"Bearer {api_key}",
            "Content-Type": "application/json"
        },
        json={
            "model": model,
            "messages": [{"role": "user", "content": message}],
            "stream": True
        },
        stream=True
    )

    for line in response.iter_lines():
        if line:
            # Parse Server-Sent Events format
            # Format: data: {"choices": [{"delta": {"content": "..."}}]}
            decoded = line.decode("utf-8")
            if decoded.startswith("data: "):
                data = decoded[6:]  # Remove "data: " prefix
                if data != "[DONE]":
                    import json
                    chunk = json.loads(data)
                    content = chunk["choices"][0]["delta"].get("content", "")
                    print(content, end="", flush=True)

This prints each token as it arrives, giving you that typewriter effect.

What I Learned

After two weeks of using this approach in production, here's what I've learned:

1. Unified APIs save time. I spent 3 days building my own adapter layer. I could have spent 3 hours integrating a unified API.

2. Error handling is critical. Don't just retry immediately. Use exponential backoff. Log errors. Monitor rate limits.

3. Streaming is worth it. Users prefer seeing tokens appear in real-time rather than waiting 10 seconds for a full response.

4. Model switching should be trivial. If you need to change 50 lines of code to switch providers, your abstraction is too complex.

Full Disclosure

I'm now working on a project called RollTok, which is a unified API gateway for multiple LLM providers. The code examples above are based on what I learned while building it. If you want to try it out, you can get an API key at rolltok.com.

But honestly, the approach works with any unified API. The specific service doesn't matter as much as the pattern.

What's Next

I'm planning to write about:

• How to track token usage across multiple providers
• How to implement request/response logging
• How to set up connection pooling for better performance

If you have questions or suggestions, drop them in the comments.

Comments

[Rin Katsuragi]

This is insightful and interesting. I had the same issue with my partner with custom function bundles for robotics, where we had to swap AI models, but without tinkering with the function or schemas or adapters. For us Skillware worked, but it required quite some adjustments from our side, mostly cause we wanted our own custom skills for specific arms. Now this is an interesting take that reminds me of Skillware for APIs. I will try it.

[rolltok]

Hey Rin, thanks for this! Yeah, that's exactly the frustration - you just want to swap models without touching anything else. I haven't actually heard of Skillware before (I might be wrong, but is it more for robotics-specific use cases?). Sounds like you went through a ton of customization work for different arms.

Curious - what made you and your partner decide to build your own setup instead of using something like this? Always trying to understand how others solve this problem.

[Rin Katsuragi]

Skillware is not pop, it's like a niche ai agent toolkit that goes beyond simple agents skills that are just markdown files. It packages scripts, prompts, schemas, and everything a function call needs to execute deterministic code, which is what we need for robotics exactly. My partner found this framework cause they plan to expand to robotic but haven't, so we took their framework and adjusted it for robotics to use for our project, eg. udnerstanding vision, weights of objects and more without relying on the llm, but on hardcoded skill bundles.

The API approach is close or similar to MCPs I guess, but I am making connections already how this can be used for specific use cases.

[rolltok]

Thanks for the clarification — that makes way more sense now. I was thinking of skills as just metadata wrappers, but bundling the actual execution logic (scripts + schemas + deterministic code) is a completely different model.

The robotics angle is interesting. You're essentially pre-compiling the "how" so the LLM only handles the "when/why" routing, right? Vision processing, weight calculations, movement primitives — all deterministic, all packaged. The LLM becomes the orchestrator, not the implementer.

I can see why you'd fork it. Robotics can't afford the latency or unpredictability of "let me generate code for this," especially for real-time physical actions. You need guardrails built into the skill bundles themselves,

Curious: how are you handling the boundary between "skills the robot can do" and "skills it thinks it can do"? Like, does the bundle include capability constraints (weight limits, reach boundaries) that the LLM can query before attempting an action, or are you catching failures after the fact?

And yeah, the MCP parallel is there, but MCPs feel more like "expose tools to the LLM." What you're describing sounds more like "expose verified, constrained behaviors." Subtle but important difference.

What's the hardest part so far — integrating the vision stack, or defining the skill interface in a way that's flexible enough for new behaviors but constrained enough to be safe?

[Rin Katsuragi]

So hard the hardest part is writing the python scripts for our robot arms :D but you got Skillware right. Once you write the bundles, then they can be reused with any model, essentially swapping the brain instead of rewiring motor functions.

The robot has a specific set of skills and it cannot invent new skills. Ensuring these skills are deterministic gives 0 chance for hallucination, so the skill either works either doesn't. there is no grey zone. We had cases where context was cluttered if you have like 20+ skills with full context, so we use the ctx from Skillware which only appends descriptions of what the skill does and when to use it, then based on which skill is triggered, more context is unlocked, also serving as a smart memory management service.

On top of it, you can use a harness like aura, which has a native sequencer that helps us modify it and we know exactly what our robots do, when, why, how, where did the reasoning came from, eg. was it the model who thought of it? was it picked form an internet post? was it from the skillware bundle? etc.

In vision, the hardest part is lighting and unpredecitable background noize, eg. people walking around, hands in the scene, etc. even if the robot does everything perfectly sometimes strict guardrails can make it freeze, cause eg. it saw a hand in the scene and didn't calculate the distance, or only calculated a frozen snapshot, thinking if it moves it might damage the human hand, so it rather freezes than move.

[rolltok]

That's the real constraint right there — Python scripting is where theory meets "wait, does this actually work on hardware?"

The deterministic-or-nothing model is elegant. No hallucinations because there's no room for interpretation. Either the skill executes or it fails cleanly. That's actually safer than an LLM trying to be creative with a robot arm.

The context windowing thing is smart. I get it — 20+ skills all exposed at once just creates decision thrashing. Lazy-loading the context based on triggered skills is exactly what you'd want. The robot narrows its own problem space instead of having to parse everything upfront. And yeah, that doubles as memory management.

The traceability angle with Aura's sequencer is what I didn't expect but totally get now. You can actually audit the reasoning chain — was this a model decision or a skill-driven one? That's genuinely useful for debugging when something goes wrong in production.

But the vision problem you described hits different. That's not a skill failing — that's your guardrails working too well. The robot sees a hand, can't calculate safe distance, so it freezes rather than risk it. That's technically correct but operationally useless.

How are you handling that? Is it: (a) tighter hand detection + distance calculation in the vision bundle itself, (b) a "defer to human" state instead of freezing, or (c) something else? Because I imagine real-world robotics can't just freeze every time the scene gets messy.

[Rin Katsuragi]

Exactly. In our case, we are experimenting with different models and risk levels to see what happens. First it needs classificatio to seperate real hands with fake. Initially we used a fake arm and it actually stopped, then a thermal sensor helps the classifier tell whether the hand is real and organic and not an object that looks like hand. Then we reversed our thinking in a loop, as we had to also consider other robot hands. Haven't cracked that 100% but I think it is solvable with good sensors, spatial understanding and reasoning, and wider context, not just snapshots, eg. did this hand just wave, or standing there still? did the bearer of the hand passed by or in interfering distance etc.

[rolltok]

Okay that's a legitimately hard problem. Not just "is it a hand" but "is it a real hand, a fake hand, or another robot's hand" — and the answer changes what's safe to do.

The thermal sensor step makes sense for organic vs fake, but then you hit the robot-hand edge case. That's where pure sensor fusion hits its limit and you need temporal reasoning. A single frame can't tell you intent or trajectory.

The motion context you're describing is exactly the right direction. "Did this hand just wave or is it static?" and "Is the person passing through or standing still?" are different risk profiles. You're essentially building a scene-level state machine on top of the skill system.

Have you considered fusing the temporal data into the skill bundle itself, or are you keeping that in the harness layer? Like, does the "detect hand" skill output not just "hand detected" but "hand detected + motion vector + proximity trend," so downstream skills can make smarter decisions?

And the robot-hand problem is interesting. You could tag your own robots' hands with a marker the vision system recognizes (IR reflector, specific color pattern), but that only works in controlled environments. In the wild, you'd need something like "classify hand → if metallic/jointed → assume robot → different safety threshold."

[Rin Katsuragi]

the detect layers are essentially custom skill bundles we created on top of skillware, the harness only explain what is done in what sequence, and stops things before bad decisions are taken. we use a classifier yes. are you also working in robotics?