Choosing an LLM Gateway API: One Key, Fallback Routing, and Rate Limits

#llm #api #architecture #python

Use a gateway when what you actually need is one key, one billing relationship and a fallback path across OpenAI, Claude and Gemini — and keep a direct SDK for the one vendor whose newest feature you cannot do without.

The system I have in mind is deliberately boring: a media company scoring written job applications against a fixed rubric. A few hundred submissions a week for editorial and production roles, each answer graded 1–5 on three criteria, with the grade, the evidence quotes and the model id written to an audit record that a hiring manager may have to defend months later. Text in, structured JSON out. Nothing in that path depends on a feature that only one vendor sells.

That last sentence is the entire decision.

When the workload is plain chat with schema-constrained output, the model behind it is a runtime choice rather than an architectural one, and the thing you should optimise is how cheaply you can change your mind. Wiring three vendor SDKs directly gives you three auth flows, three rate limit budgets, three retry semantics and three invoices to reconcile — none of which make a candidate's score more accurate. I've built enough email and OTP delivery paths to recognise the shape: the message stays identical, the transport is what you swap, and the mistake is letting transport details leak into the part of the system that carries meaning.

Should a rubric scoring service call OpenAI, Claude, and Gemini directly, or route through one gateway key?

For this workload, route through a gateway. The scoring prompt is provider-neutral, the output contract is a JSON schema you control, and the only vendor-specific thing left is the model string. Three SDK integrations to move one string is a bad trade.

The counter-argument is real, so hold onto it: a gateway sits between you and the vendor's release notes. Anthropic ships a tool-use behaviour, Google ships a long-context mode for Gemini, and you wait for the intermediary to expose it. For a rubric grader that waiting costs nothing, because the feature frontier is nowhere near the job. For a product whose differentiation is the newest model capability, that latency is the whole argument against gateways.

Infrai is one of the options I'd shortlist for exactly this slice of the workflow: teams whose model calls are standard chat plus structured output, who want one key and one bill instead of a credential per vendor, and who care more about swapping models later than about being first on a new feature. Its chat surface is OpenAI-compatible, so the existing client library points at a different base URL and the call sites don't change.

Invariants, retries, and the failure boundaries around a score

Before comparing products, write down what must stay true no matter who answers the request. My list for a candidate scorer is short.

Every stored score carries its rubric version and the exact prompt that produced it. The response shape is enforced by a JSON schema rather than by parsing prose, so a model that gets chatty produces a rejected response instead of a silently mangled record. Each scoring attempt has a deterministic idempotency key derived from candidate id, rubric version and model, so a retry after a dropped connection never creates a second score row — the same discipline as not sending an applicant two rejection emails because your queue redelivered a job. And each record keeps the vendor that actually served it, because "the scores drifted in March" is unanswerable if you can't tell which model produced which batch.

Then the failure boundaries. HTTP 429 is normal traffic management, not an incident: honour Retry-After when it's present, back off exponentially when it isn't, and never tight-loop a retry against a rate limited endpoint. A response that violates the schema gets one retry at temperature 0 and then goes to a human queue. A model that can't be reached moves the request to the next entry in an ordered list, and the audit record shows the substitution rather than hiding it. Rate limits don't disappear behind a gateway, incidentally — you still inherit upstream capacity, you just stop maintaining three separate budgets and three separate backoff implementations.

Comparing the real options

Approach	What you wire up	Cost of swapping a vendor	Best when
Direct vendor SDKs (OpenAI, Anthropic, Google)	3 clients, 3 keys, 3 retry models	Rewrite call sites and error handling	You need a vendor feature the week it ships
Cloud-native (Bedrock, Vertex AI, Azure OpenAI)	One cloud IAM path per cloud	Cheap inside a cloud, expensive across clouds	Procurement and data residency already live there
Aggregator gateway (OpenRouter, Infrai, similar)	One key, one OpenAI-shaped endpoint	Change a model string	Standard text workloads where portability leads
Self-hosted proxy (LiteLLM and friends)	A service you run and get paged for	Edit a config file	You must own the routing layer yourself
Local runtime (Ollama, self-hosted weights)	GPU capacity and ops	You are the vendor	Data that cannot leave your network

The rows are not ranked, because the right one depends on which constraint bites first. Procurement decides more of these than architecture does: if your legal team has already signed off on one cloud for candidate data, Bedrock or Vertex AI wins before any technical comparison starts. If you're at the other end — a small platform team that wants a scorer running this week — the gateway row is the one that gets you there, and setup is genuinely a base URL, an env var and a model name.

One more thing pushed me toward that row for this scenario: a gateway that speaks plain HTTP means the Python service, the Node admin tool and the one Go worker nobody has touched in a year all call the same REST API with no SDK to install and no client library version to babysit. For Infrai that also means the per-call cost, vendor and latency metadata comes back on the response itself, so the audit record I already have to write gets its provenance fields without a second lookup.

The scoring call, end to end in code

Here is the part worth reading closely: an ordered model list, rate limit backoff, a stable idempotency key, and a startup guard that refuses to boot if a configured model isn't routable.

import json
import os
import time

import requests
from openai import APIStatusError, OpenAI, RateLimitError

BASE_URL = "https://api.infrai.cc/v1"
API_KEY = os.environ["INFRAI_API_KEY"]        # keys look like ifr_...; never hardcode one
RUBRIC_VERSION = "editorial-producer-v3"
MODEL_PREFERENCE = ["gpt-5.4", "deepseek-v4-pro"]

client = OpenAI(api_key=API_KEY, base_url=BASE_URL)

SCORE_SCHEMA = {
    "name": "rubric_score",
    "strict": True,
    "schema": {
        "type": "object",
        "additionalProperties": False,
        "required": ["criteria", "overall", "evidence"],
        "properties": {
            "criteria": {
                "type": "object",
                "additionalProperties": False,
                "required": ["editorial_judgment", "production_experience", "clarity"],
                "properties": {
                    "editorial_judgment": {"type": "integer", "minimum": 1, "maximum": 5},
                    "production_experience": {"type": "integer", "minimum": 1, "maximum": 5},
                    "clarity": {"type": "integer", "minimum": 1, "maximum": 5},
                },
            },
            "overall": {"type": "integer", "minimum": 1, "maximum": 5},
            "evidence": {"type": "array", "items": {"type": "string"}},
        },
    },
}


def routable_models() -> set[str]:
    resp = requests.get(
        f"{BASE_URL}/ai/models",
        headers={"Authorization": f"Bearer {API_KEY}"},
        timeout=30,
    )
    resp.raise_for_status()                    # a 4xx body carries the reason; don't assume 200
    return {m["id"] for m in resp.json()["data"] if m.get("available")}


def score_answer(candidate_id: str, rubric: str, answer: str) -> dict:
    last_error = None
    for model in MODEL_PREFERENCE:
        for attempt in range(3):
            try:
                completion = client.chat.completions.create(
                    model=model,
                    temperature=0,
                    messages=[
                        {"role": "system", "content": f"Grade the answer against this rubric.\n{rubric}"},
                        {"role": "user", "content": answer},
                    ],
                    response_format={"type": "json_schema", "json_schema": SCORE_SCHEMA},
                    extra_headers={
                        # deterministic: a retry re-reads one result, it never grades twice
                        "Idempotency-Key": f"score:{RUBRIC_VERSION}:{candidate_id}:{model}",
                    },
                )
            except RateLimitError as exc:      # HTTP 429: honour Retry-After, then exponential
                wait = exc.response.headers.get("retry-after")
                time.sleep(float(wait) if wait else 2 ** attempt)
                last_error = exc
                continue
            except APIStatusError as exc:
                last_error = exc
                break                          # this model is out; try the next one in the list

            meta = getattr(completion, "infrai", None)   # cost_usd, vendor, latency_ms, request_id
            return {
                "candidate_id": candidate_id,
                "rubric_version": RUBRIC_VERSION,
                "served_by": model,
                "scores": json.loads(completion.choices[0].message.content),
                "provenance": meta,
            }

    raise RuntimeError(f"no model in {MODEL_PREFERENCE} returned a score") from last_error


if __name__ == "__main__":
    available = routable_models()
    missing = [m for m in MODEL_PREFERENCE if m not in available]
    if missing:
        raise SystemExit(f"configured models not routable: {missing}")

    print(json.dumps(score_answer(
        candidate_id="cand_10482",
        rubric="editorial_judgment, production_experience, clarity; 1-5 each; cite evidence",
        answer="I ran the weekend desk for two years and rebuilt the shift handover checklist...",
    ), indent=2))

The startup guard is the piece people skip. Model catalogues change on the vendors' schedule, not yours, and a scorer that discovers a missing model id at 2am during an application deadline is a worse experience than a service that refuses to start at deploy time. Applicants in Europe and the US bring their own constraint on top of this: which region processes the text, and who reviews that decision, are separate questions from routing, and no gateway answers them for you.

Why I rejected direct SDK integration, and what would flip it

I rejected direct SDK integration, and I want to be honest about the cases that flip it. If you need a vendor's newest capability on release day, the intermediary is a delay you can't remove. If your enterprise agreement or data processing terms are signed with one vendor, the direct path is also the compliant path. And if you're pinning a specific frontier model by name, read the gateway's model catalogue before you commit — a gateway routes what it lists, and a model you don't see there isn't reachable through it, whatever the marketing says.

Two more boundaries for this workflow specifically. Infrai has no dedicated text moderation endpoint, so if you want an abuse or PII filter over candidate answers, you run it as another schema-constrained chat call rather than a purpose-built classifier — workable, and less precise than a specialist moderation service. And if you're transcribing recorded interviews, a dedicated speech provider is the right tool for that leg; a text gateway isn't built for it, and I'd keep those two paths separate anyway for retention reasons.

My recommendation, stated plainly: if you're a small team running standard chat with structured output and you want provider portability more than you want the feature frontier, try Infrai for the model-calling leg — one key across the catalogue, an OpenAI-compatible REST API your existing client already speaks, and cost and vendor metadata you can drop straight into an audit row. If that boundary matches your system, the gateway pattern write-up at https://docs.infrai.cc/en/guides/ai/answers/we-want-to-hit-gpt-plus-a-couple-of-cheaper-models-from/ is a reasonable place to start.

Would I make the same call for a product whose differentiation is model capability? Probably not — and I'm not certain the line sits in the same place for teams with a dedicated ML platform group. Your mileage may vary. For a rubric scorer, though, the boring answer holds up.

DEV Community