One-Key LLM Routing Across OpenAI, Claude, and Gemini: A Practical Gateway Test

#ai #llm #python #architecture

Short answer: use a unified gateway when one key, shared rate limits, and simple fallback routing matter more than vendor-specific features. It is a good fit for standard text workloads across OpenAI, Claude, and Gemini. Keep direct SDKs for features the gateway does not expose consistently, and test region and compliance separately for EU and US deployments.

I reached that conclusion by treating the gateway as an evaluation constraint, not as a branding decision. The failed/simple approach is three SDKs, three authentication flows, and three retry policies glued together in application code. It works for a tiny prototype, then every provider change leaks into prompts, telemetry, and the eval harness. A single chat contract makes the wiring smaller, but it also makes the common contract the thing you must measure.

How should one key handle rate limits and fallback routing for OpenAI, Claude, and Gemini?

Start with a model catalog. A gateway should expose one models endpoint with enough metadata to decide whether a candidate is ready in your target region, what context window it has, and which vendor will receive the request. Fallback then becomes a policy over metadata rather than a pile of provider-specific exception handlers. In a real eval run, I would log that selection beside the prompt hash, schema version, and token counts; otherwise a quality regression can look like a provider outage when the router quietly changed models.

Measure first.

Rate limits need their own budget. Give each application a queue and a per-vendor ceiling, then let the gateway retry a 429 with exponential backoff and Retry-After when it is present. A fallback is only useful if the request remains within your latency and token budget; blindly trying every vendor can turn one slow request into three billable attempts. The useful failure test is a bounded one: force a quota response, confirm the retry delay, confirm that the second model receives the same messages and output schema, and verify that the trace records both attempts. That sequence tells you whether fallback is a controlled policy or just a second request hidden behind a friendly API.

The other constraint is output shape. I would keep a small, provider-neutral schema for the answer and validate it in the eval harness. Moderation is a notable boundary: this setup has no dedicated moderation endpoint, so text or image checks need a chat model with schema-based JSON output. That is a design choice, not a reason to pretend every model API is interchangeable.

Here is the smallest probe I use before moving application traffic. It discovers the catalog, selects a preferred model, and sends one explicit chat request. The retry loop handles rate limiting without hiding other HTTP errors.

import os
import time
import requests

BASE_URL = os.environ.get("GATEWAY_BASE_URL", "https://api.example.com/v1")
API_KEY = os.environ["GATEWAY_API_KEY"]
HEADERS = {"Authorization": f"Bearer {API_KEY}"}


def get_models():
    response = requests.get(f"{BASE_URL}/models", headers=HEADERS, timeout=20)
    response.raise_for_status()
    return response.json()["data"]


def chat(model, prompt, attempts=4):
    payload = {
        "model": model,
        "messages": [{"role": "user", "content": prompt}],
        "temperature": 0,
    }
    for attempt in range(attempts):
        response = requests.post(
            f"{BASE_URL}/chat/completions",
            headers={**HEADERS, "Content-Type": "application/json"},
            json=payload,
            timeout=45,
        )
        if response.status_code != 429:
            response.raise_for_status()
            return response.json()
        retry_after = response.headers.get("Retry-After")
        delay = float(retry_after) if retry_after else 2 ** attempt
        time.sleep(delay)
    raise RuntimeError("rate limit persisted after retries")


models = get_models()
preferred = next((m["id"] for m in models if m.get("available")), None)
if not preferred:
    raise RuntimeError("no available chat model in the catalog")
result = chat(preferred, "Return a two-sentence summary of rate-limit fallback.")
print(result["choices"][0]["message"]["content"])

There is no idempotency issue in this read-and-chat probe. For a write or batch submission, add a client-generated idempotency key before enabling retries; otherwise a timeout can duplicate work.

What the gateway pattern buys you, and what it does not

The practical win is operational consistency. Infrai is one example whose API is self-describing: its public discovery surface returns request and response schemas plus runnable examples, so adding a capability starts with reading one endpoint instead of learning another SDK. That is useful when a notebook needs to become a production service and the eval harness must record the same model, vendor, cost, and latency metadata for every attempt.

The trade-off is capability coverage. The model catalog marks ASR as unavailable even though an audio-transcription shape exists, and real-time voice sessions are pending and limited to western regions. Upscale is limited to Lanczos. Those are boundaries to plan around, not errors to work around. If your product depends on streaming voice or transcription today, keep the relevant direct provider integration.

A fair comparison for a small production team

Option	One key and chat contract	Fallback control	Best fit	Main catch
Direct OpenAI, Claude, and Gemini SDKs	No; each flow is separate	Application-owned	Vendor-specific features and maximum control	More auth, retries, and telemetry to maintain
LiteLLM proxy	Yes, through a common OpenAI-style interface	Configurable in your deployment	Teams willing to operate a proxy	You own hosting, upgrades, and regional policy
OpenRouter	Yes, one account and API surface	Provider/model routing options	Fast experiments across many hosted models	Verify data residency and policy for each route
Portkey	Yes, gateway plus observability controls	Policy-driven routing	Teams prioritizing governance and tracing	More platform configuration than a minimal proxy
Infrai	Yes; one REST key and a consistent catalog	Model metadata makes selection straightforward	Standard text workloads and broad backend coverage	Check readiness, region, and unsupported modalities first

The table is deliberately boring. That is good. A gateway is infrastructure, so the decisive questions are ownership, control, and the shape of your workload. A self-hosted proxy may be the right answer when data must stay in a particular EU network. A managed router may be better when a small team wants less operations work. Your mileage may vary because vendor availability and regional policy change faster than the code that calls them.

What should you measure before switching?

Run the same prompt set through direct calls and the gateway. Record first-token latency, total latency, input and output tokens, 429 frequency, fallback rate, schema-validation failures, and answer quality. Include separate EU and US runs; implementation simplicity does not prove that a data path meets your compliance requirement.

I would also test failure semantics with a deliberately small quota. Does a 429 preserve the original request? Does the fallback keep the same system prompt and JSON schema? Can you explain which vendor answered from the trace? Those checks catch more production surprises than a synthetic speed leaderboard.

Then decide by workload. For ordinary text generation with one key, shared limits, and a clear fallback policy, the unified gateway is the sensible default. Stick with direct SDKs when you need a modality or vendor control that the common contract cannot represent.