DEV Community

llong
llong

Posted on Fully Autonomous

We Migrated from Single-Model to a Multi-Model Gateway — Here's What Actually Changed

#api #architecture #llm #openai

We started with one OpenAI key. Twelve months later we were calling four models from three providers. Here's the migration story — what broke, what surprised us, and what I'd do differently.

Where we started

A simple FastAPI service. One OPENAI_API_KEY in env. Every LLM feature in the product called GPT-4o. It worked great. It worked for a long time.

The cracks showed up in three places:

  1. Cost. Long-context summarisation on GPT-4o was eating 60% of our LLM bill for 15% of requests.
  2. Resilience. A 20-minute OpenAI degraded-status page took our summarisation down with it.
  3. Capability gaps. Our review-classification task needed Claude's structured output, which GPT handled inconsistently.

What migration looked like

Step 1 was not "rewrite everything". It was "put a gateway in front".

# Before:
# client = OpenAI(api_key=os.environ["OPENAI_KEY"])

# After:
client = OpenAI(
    api_key=os.environ["GATEWAY_KEY"],
    base_url="https://api.agentoken.co/v1"  # <- only this changed
)
Enter fullscreen mode Exit fullscreen mode

The SDK didn't change. The request shape didn't change. We just pointed base_url somewhere else.

Step 2 was re-tagging which model each feature actually needed:

Feature Before After Why
Chat completion GPT-4o GPT-4o default
Long summarisation GPT-4o Claude 3.5 Haiku cheaper, longer context
Classification GPT-4o GPT-4o-mini trivial task
Structured extraction GPT-4o Claude 3.5 Sonnet better JSON reliability

Step 3 was routing by task. The gateway's model field became per-feature config, not a global constant.

What actually changed

Cost dropped ~35% in the first month. Not because we negotiated anything, but because cheap tasks stopped burning frontier tokens.

One outage stopped being a product outage. When OpenAI had a degraded window, summarisation automatically fell over to Claude. Users didn't notice.

Debugging got easier. One dashboard now shows every request: which feature, which model, how many tokens, latency. Before this was three separate billings and a spreadsheet.

What surprised me: nothing broke. Because the SDK contract was preserved, the migration took one afternoon. I'd mentally budgeted a week.

What I'd do differently

I'd put the gateway in on day one. The temptation is to wait until multi-model is "real" — but the moment you add a second feature, you're already multi-model, you're just doing it by hand.

If you're starting a new AI product today, one OpenAI-compatible gateway key is the default shape. Direct provider accounts are an optimisation, not a starting point.

Agent Token is the gateway we ended up on. Same OpenAI SDK, one bill across GPT / Claude / Gemini, automatic failover.

Curious who else has done this migration — what did your cost per request look like after?

Top comments (0)