Building a Unified LLM API Gateway: What I Learned Managing 5 Model Providers

I'm building taotok.io, a unified LLM API gateway. Over the past month, I've been wiring up GPT-4o, Claude 3.5, Gemini, and DeepSeek into a single endpoint — and I learned a few things the hard way.

If you're managing multiple LLM APIs in production, this post walks through the architecture we landed on and the practical tradeoffs. No product pitch — just the engineering.

The Problem Nobody Talks About

Every LLM provider does APIs differently. Not dramatically differently — just enough to be annoying.

Thing that's different	OpenAI	Anthropic	Gemini
Auth header format	`Bearer sk-xxx`	`x-api-key` header	query param or header
Streaming format	SSE with `data:` prefix	SSE with custom events	gRPC or SSE
Error codes	HTTP 429 for rate limits	HTTP 429 + custom body	HTTP 429 + retry-info
Request format	Compatible-ish	Messages API	`generateContent`

None of these are dealbreakers individually. But when you add a fourth model, then a fifth, the adapters multiply. Each new provider means rewriting auth, response parsing, error handling, and streaming logic.

The real cost isn't the initial integration — it's maintaining the adapters when each provider ships a breaking change.

The Architecture We Settled On

After a few false starts, we landed on a four-layer design:

Layer 1: Unified Ingress

A single POST /v1/chat/completions that accepts the OpenAI format. All client code talks to this. If a provider changes their API format, we fix it in one place.

Layer 2: Protocol Adaptation

Per-provider adapters that translate the OpenAI format into provider-native requests, then normalize responses back. Each adapter is ~150 lines of Python.

Layer 3: Intelligent Routing

A router that decides which provider gets the request. We use a simple priority queue:

If the model parameter matches exactly (e.g., gpt-4o), route directly
If the model is auto, pick based on cost + availability
Fall back to the next provider if the primary returns 429 or 5xx

Layer 4: Observability

Every request logs: provider, model, latency, tokens consumed, error (if any). We use this to catch failing providers and optimize cost allocation.

Code: What a Single-Endpoint Call Looks Like

Here's the client side. Notice you never touch provider-specific auth or endpoints:


python
from openai import OpenAI

# One client, one endpoint, one key
client = OpenAI(
    base_url="https://api.taotok.io/v1",
    api_key="your-gateway-key"
)

# Switch models by changing one parameter
for model in ["gpt-4o", "claude-3-5-sonnet", "gemini-1.5-pro"]:
    response = client.chat.completions.create(
        model=model,
        messages=[{"role": "user", "content": "Hello"}],
        stream=True
    )
    for chunk in response:
        if chunk.choices[0].delta.content:
            print(chunk.choices[0].delta.content, end="")
The gateway handles:

Translating the OpenAI format into provider-native requests
Managing provider-specific API keys (stored server-side)
Retrying on rate limits with exponential backoff
Normalizing streaming responses across providers
What We Got Right (and Wrong)
Right: Starting with the OpenAI-compatible format saved us weeks. Every major LLM client library supports it natively.

Right: Centralizing API keys server-side. During onboarding, users paste their provider keys once. The gateway stores them encrypted.

Right: Aggressive caching of model lists. Providers change available models rarely but listing them adds 200-500ms latency.

Wrong: Underestimated the complexity of retry logic. Anthropic and OpenAI have different rate limit headers with different semantics. A generic retry strategy doesn't cut it — you need per-provider retry policies.

Wrong: Built the streaming adapter incrementally. This is a bad idea. Stream processing is stateful and provider-specific — refactoring it later was painful.

Getting Started (Your Own or Ours)
If you want to build your own gateway, the minimum viable product is:

Nginx/LiteSpeed reverse proxy — handles TLS termination and basic rate limiting
Python FastAPI service — the protocol adaptation layer
Provider adapters — start with OpenAI and Claude, add more as needed
Simple health check — call each provider's models endpoint every 30s, mark unavailable if it fails
If you don't want to maintain this yourself: we built taotok.io so you don't have to. We support GPT-4o, Claude 3.5, Gemini 1.5 Pro, DeepSeek V3, and more through one API key. Try it free — $5 trial with 200 credits, enough to test every model.
What's your multi-LLM setup look like? Drop a comment — I'm always looking for better patterns.