We finally learned to switch LLM providers with 3 code changes instead of rewriting half the app

I thought switching LLM providers would be a model problem.

It turned out to be an architecture problem.

We weren’t blocked by prompts.
We weren’t blocked by evals.
We weren’t even blocked by output quality.

We were blocked because our app had slowly grown a second codebase made of provider quirks.

One path handled OpenAI function calling.
Another handled Anthropic tool_use blocks.
Then somebody added a Gemini branch because pricing and latency got weird and we wanted options.

That was the moment it clicked:

we had built the app as if the model was the app.

That’s backward.

If you want to switch between OpenAI, Anthropic, Gemini, or anything else without a painful rewrite, the fix is boring but effective:

• keep one internal model interface
• normalize tool schemas
• isolate retries and fallbacks
• use an OpenAI-compatible endpoint where possible
• stop leaking provider details into business logic

And yes, for some cases, Gemini really can get surprisingly close with just 3 code changes.

The 3 changes that make Gemini work with OpenAI client code

If your app is doing simple chat completion, Google’s OpenAI-compatible endpoint is the cleanest example of what good migration ergonomics look like.

You can often change:

1. API key
2. base URL
3. model name

from openai import OpenAI

client = OpenAI(
    api_key="GEMINI_API_KEY",
    base_url="https://generativelanguage.googleapis.com/v1beta/openai/"
)

response = client.chat.completions.create(
    model="gemini-3.8-flash",
    messages=[
        {"role": "user", "content": "Explain how AI works"}
    ]
)

print(response.choices[0].message.content)

That feels magical the first time you do it.

But only if your app is basically messages in -> text out.

Once you depend on tools, structured output, retries, fallbacks, or workflow automation, the easy demo stops being the real story.

Where provider migration actually breaks

Not in plain text generation.

That part is usually fine.

The pain shows up in all the stuff that makes the app useful.

1. Tool calling is where the migration pain lives

This is the biggest trap.

OpenAI-style APIs usually expect function tools with JSON Schema.
Anthropic uses tool_use and tool_result blocks with input_schema.
Gemini can look OpenAI-ish through compatibility layers, but the semantics still aren’t always identical.

If your agent loop assumes one provider’s exact tool shape, provider migration is not a config change.

It’s surgery.

Bad pattern: provider syntax inside business logic

if provider == "openai":
    tools = [{
        "type": "function",
        "function": {
            "name": "lookup_customer",
            "parameters": customer_schema,
        }
    }]
elif provider == "anthropic":
    tools = [{
        "name": "lookup_customer",
        "input_schema": customer_schema,
    }]

That works at first.

Then six months later you’re tracing branches across API handlers, background jobs, n8n workflows, Zapier steps, and random utility files.

Better pattern: one internal schema

Define tools once.
Compile them per provider.

customer_lookup_tool = {
    "name": "lookup_customer",
    "description": "Find customer by email",
    "schema": {
        "type": "object",
        "properties": {
            "email": {"type": "string"}
        },
        "required": ["email"]
    }
}

def to_openai_tool(tool):
    return {
        "type": "function",
        "function": {
            "name": tool["name"],
            "description": tool["description"],
            "parameters": tool["schema"]
        }
    }

def to_anthropic_tool(tool):
    return {
        "name": tool["name"],
        "description": tool["description"],
        "input_schema": tool["schema"]
    }

That one separation removes a shocking amount of future pain.

2. Structured output is the second trap

A lot of teams say they have structured output.

What they actually have is:

• ask model for JSON
• hope it returns valid JSON
• regex away markdown fences
• retry when parsing fails
• pretend this is fine

That’s not structured output.
That’s a ritual.

Your app should ask for something like CustomerIntent or InvoiceFields.
It should not care whether the provider implements that through native schema support, tool calling, or some compatibility layer.

Bad pattern

import json

raw = llm_response_text.replace("```

json", "").replace("

```", "")
data = json.loads(raw)

Better pattern

from pydantic import BaseModel

class CustomerIntent(BaseModel):
    intent: str
    urgency: str
    needs_human: bool

Then your harness handles provider-specific structured output mechanics.

That’s not just cleaner.
It’s migration insurance.

3. Retries and fallbacks should not live in app code

This one still drives me nuts.

I keep seeing production systems where business logic knows about:

• rate limits
• context window errors
• backup providers
• regional failover
• API key rotation

That does not belong in your refund workflow.
It does not belong in your support bot.
It definitely does not belong in the branch of an n8n flow that updates Salesforce.

Bad pattern

try:
    result = call_openai(messages)
except RateLimitError:
    try:
        result = call_anthropic(messages)
    except Exception:
        result = call_gemini(messages)

This looks pragmatic.

It also guarantees that provider changes leak into product code forever.

Better pattern

Put retries, routing, and fallback behavior into a harness layer.

That can be your own abstraction.
Or something like LiteLLM or OpenRouter.
Or an OpenAI-compatible edge that hides the routing behind one endpoint.

The point is the same:

business logic should ask for a model capability, not manage provider drama.

The alias trick more teams should use

One of the best patterns here is using a stable internal alias.

Your app calls primary-agent-model.
Infra decides what that actually means today.

For example:

model_list:
  - model_name: primary-agent-model
    litellm_params:
      model: azure/gpt-4o-eu
      api_base: https://my-endpoint-europe.openai.azure.com/
      api_key: os.environ/AZURE_API_KEY_EU
      rpm: 6

  - model_name: primary-agent-model
    litellm_params:
      model: azure/gpt-4o-ca
      api_base: https://my-endpoint-canada.openai.azure.com/
      api_key: os.environ/AZURE_API_KEY_CA
      rpm: 6

Now the app does not need to know about regions, deployments, or failover.

That is how you stop provider changes from turning into product work.

OpenAI-compatible APIs are useful, but not magic

This is the part people oversimplify.

OpenAI-compatible APIs are real.
They save time.
They reduce migration cost.

They are also not identical.

Option	What you gain and what bites you later
OpenAI native API	Strong ecosystem, solid tooling, structured output support, but provider-specific assumptions can spread through your code fast
Gemini OpenAI compatibility endpoint	Fastest migration path for simple chat flows; often just key, base URL, and model swap; still has Gemini-specific behavior and capability differences
LiteLLM or OpenRouter abstraction	Better routing, fallback, normalization, and provider flexibility; adds another layer, but it usually pays for itself the first time pricing or reliability changes
OpenAI-compatible edge like Standard Compute	Lets existing OpenAI SDKs and HTTP clients keep working while routing across multiple models behind one endpoint; especially useful for agents and automations that need predictable cost and less provider lock-in

The mistake is assuming “compatible” means “interchangeable.”

It doesn’t.

Anthropic still has native concepts that don’t map perfectly.
Gemini still has its own quirks.
OpenAI still ships features others imitate later.

Use compatibility layers.
Just don’t confuse them with a universal standard.

Why this matters more for n8n, Make, Zapier, and agent workflows

In app code, a provider swap is annoying.

In automation systems, it gets worse.

Because now provider assumptions are buried inside:

• n8n nodes
• Make scenarios
• Zapier steps
• OpenClaw agents
• custom webhook handlers
• cron jobs nobody wants to touch

If every workflow is wired directly to one provider’s exact API shape, migration becomes a scavenger hunt.

That’s why an OpenAI-compatible endpoint matters so much in automation land.

If the workflow already knows how to talk to an OpenAI-style API, you can often keep the workflow and swap the endpoint underneath it.

That’s the practical win.
Not theory.
Not “future-proofing.”
Just less rebuilding.

The boring fix that gives you the most leverage

If I had one sprint to make an LLM app portable, I’d do this in order.

1. Separate prompts from request objects

Stop building prompts inline inside provider calls.

2. Define one internal tool schema

Compile it into OpenAI functions, Anthropic tools, or Gemini-compatible shapes.

3. Separate structured output parsing from model calls

The app should ask for typed data, not parse raw JSON strings.

4. Move retries and fallbacks into middleware

Keep rate limits and failover out of business logic.

5. Introduce stable model aliases

Use names like primary-agent-model instead of hardcoding gpt-5, claude-opus, or gemini-3.8-flash all over the codebase.

6. Put an OpenAI-compatible edge in front where possible

This is especially effective if you already use the OpenAI SDK, or if your automations are built around OpenAI-shaped requests.

A practical migration shape

If I were cleaning this up today, the architecture would look something like this:

app / workflow
    -> internal llm interface
        -> tool/schema normalization
        -> structured output layer
        -> retry/fallback/routing layer
        -> OpenAI-compatible endpoint or provider adapter

And if you want a very simple Python sketch:

class LLMRequest:
    def __init__(self, messages, tools=None, response_schema=None, model_alias="primary-agent-model"):
        self.messages = messages
        self.tools = tools or []
        self.response_schema = response_schema
        self.model_alias = model_alias


def run_llm(request: LLMRequest):
    # normalize tools
    # attach schema
    # call provider or OpenAI-compatible edge
    # handle retries/fallbacks
    # return typed result
    pass

That is the boundary.

Once you have that, provider churn gets a lot less dramatic.

Why predictable pricing changes the migration conversation

There’s another reason this matters.

Per-token pricing pushes teams into weird behavior.

You start optimizing prompts for cost before you optimize the system for reliability.
You hesitate to run agents continuously.
You avoid fallback chains because every extra call feels expensive.
You treat experimentation like a finance problem.

That’s part of why products like Standard Compute are interesting for agent-heavy workloads.

If you can keep the OpenAI-compatible integration surface but stop worrying about per-token billing, a lot of design decisions get simpler:

• agents can run 24/7
• fallback and retry strategies are easier to justify
• automation volume stops feeling dangerous
• you can route across models without turning every architecture discussion into a token spreadsheet

For teams building on n8n, Make, Zapier, OpenClaw, or custom agent stacks, that’s not a small difference.
That’s operational breathing room.

The real lesson

The win is not “multi-model strategy.”

The win is being able to respond like an adult when provider drama happens.

Pricing changes.
Outages happen.
Models get deprecated.
Tool behavior regresses.
Latency gets weird.

If your app is tightly coupled to one vendor SDK, the rewrite has already started.
You just haven’t admitted it yet.

If your app has:

• one internal interface
• normalized schemas
• isolated retries and fallbacks
• stable model aliases
• an OpenAI-compatible edge where it makes sense

then switching providers stops feeling dramatic.

Sometimes it really is just 3 code changes.

And that’s a much better place to be.