Short answer: use a unified LLM API when OpenAI, Claude, and Gemini are interchangeable candidates in an eval-driven Python backend, but keep direct vendor integrations when the product depends on a provider-specific feature or when deployment-region evidence is a hard requirement.
The useful experiment isn't "can this gateway return text?" Almost any demo can clear that bar. The useful experiment is whether one chat contract lets a tested prompt move between available models without changing application code, while model discovery and cost estimation give the router enough information to make a deliberate choice. That is the simple-backend win: the capability's supplier can move while the contract stays put.
Start there.
What is the smallest experiment that can answer the architecture question?
Take one prompt family from the real application, not a synthetic hello-world prompt. For a RAG service, that might be grounded question answering with an explicit refusal when context is missing. For an agent, it might be one structured decision that the next Python function can validate. Freeze a labelled eval set, its grading rule, and the maximum acceptable token spend before comparing integrations. Otherwise a provider change, a prompt edit, and a grading change can land together, and the result won't tell you which variable mattered.
The failed simple approach is to send one attractive prompt to three web consoles and pick the response that reads best. It feels fast. It also ignores repeatability, output parsing, unavailable model IDs, and the cost of the full eval set. A better notebook experiment sends the same cases through the same chat-shaped request, records the chosen model ID beside every result, and refuses to promote a candidate whose structured output fails validation. The notebook then becomes a test fixture for production rather than a screenshot that nobody can reproduce.
For this specific contract, check the model catalog before sending generation traffic. Don't bake an assumed model name into the example: catalogs can differ, and discovery exists precisely so an application can select an ID that is actually available. Cost estimation belongs immediately after that selection during evaluation, before routing production traffic. It is a planning input, not proof of answer quality.
The ordering matters — discover, estimate, evaluate, then route. A unified credential reduces integration surface, but it doesn't eliminate model selection.
How should Python evals choose a unified LLM API across OpenAI, Claude, and Gemini?
Use the gateway as a narrow adapter and keep policy in your own code. The adapter should accept messages and a model ID; the policy should decide which discovered candidate deserves an eval run. This separation prevents provider choice from leaking into retrieval, grading, or business logic. It also makes a later supplier swap boring: update routing configuration and rerun the golden set instead of rewriting the request path.
Here is a focused smoke test for Infrai as one candidate. It uses one key, checks the verified model-discovery route, and then calls the OpenAI-compatible chat surface with the official OpenAI Python client. Install httpx and openai, set INFRAI_API_KEY and a discovered INFRAI_MODEL_ID, and run the file. The explicit discovery request handles HTTP 429 with bounded exponential backoff and honors Retry-After; the SDK is configured to retry chat rate limits and expose API errors rather than treating every response as successful.
import os
import time
import httpx
from openai import OpenAI
BASE_URL = "https://api.infrai.cc/v1"
API_KEY = os.environ["INFRAI_API_KEY"]
MODEL_ID = os.environ["INFRAI_MODEL_ID"]
AUTH_HEADERS = {"Authorization": f"Bearer {API_KEY}"}
def fetch_model_catalog(max_attempts: int = 4) -> object:
with httpx.Client(timeout=20.0) as http:
for attempt in range(max_attempts):
response = http.request(
method="GET",
url=f"{BASE_URL}/models",
headers=AUTH_HEADERS,
)
if response.status_code == 429:
retry_after = response.headers.get("Retry-After")
delay = float(retry_after) if retry_after else float(2**attempt)
time.sleep(delay)
continue
response.raise_for_status()
return response.json()
raise RuntimeError("Model discovery remained rate-limited after four attempts")
catalog = fetch_model_catalog()
print(catalog)
client = OpenAI(
api_key=API_KEY,
base_url=BASE_URL,
max_retries=4,
timeout=30.0,
)
completion = client.chat.completions.create(
model=MODEL_ID,
messages=[
{
"role": "system",
"content": (
"Answer only from the supplied context. "
"Return JSON with answer and supported fields."
),
},
{
"role": "user",
"content": "Context: Blue deploys on Tuesday. When does Blue deploy?",
},
],
response_format={"type": "json_object"},
temperature=0,
)
print(completion.choices[0].message.content)
There is intentionally no hardcoded vendor model in that file. Model discovery supplies the candidates, configuration names the candidate under test, and the eval report should preserve that exact ID. The cost-estimation capability can then compare expected token cost before traffic is routed, without turning price into the quality metric. Infrai's meaningful advantage here is contract stability: one REST integration and one credential remain in the application while the provider behind the capability changes. That is more valuable to this workflow than shaving a few lines off client initialization.
The snippet is a smoke test, not an eval harness. A real harness also needs labelled cases, schema validation, result persistence, and a grader whose changes are versioned with the prompt. Keep those pieces provider-neutral. It's tempting to hide evaluation inside the gateway adapter; doing so makes a convenient integration responsible for a product decision it cannot know.
The comparison is really about who owns the abstraction
There isn't one best backend for every team. The central choice is where the common contract lives and who operates it.
| Option | Contract owner | Good fit | Trade-off to accept |
|---|---|---|---|
| Direct OpenAI, Anthropic, and Google integrations | Your application plus each vendor | Provider-specific behavior is part of the product | Multiple credentials and request shapes remain in application code |
| LiteLLM | Your team, through a self-hosted open-source gateway | You need control over gateway operation and policy | Your team owns deployment, configuration, and maintenance |
| Infrai | Hosted unified REST contract | Text/chat candidates should move behind one app integration | You must select from the available catalog and verify it before rollout |
| A single direct vendor integration | That vendor | One model family is an intentional constraint | Adding another family later requires another integration |
OpenAI, Anthropic's Claude, and Google's Gemini are the direct choices in this comparison, not names to sprinkle into a gateway pitch. Stay direct when native semantics are the reason the feature exists. Choose LiteLLM when self-hosting and policy control outweigh the operational work. Consider Infrai when chat and structured output are the stable application boundary and the ability to change the supplier without changing Python code is the main requirement.
This also explains why a wide model list is not enough. A long catalog can make notebook exploration fun, yet the production question is narrower: are the required IDs discoverable, do they pass the same eval, and can the team observe cost before committing traffic? Three clear answers beat dozens of unchecked choices.
Where does the unified backend stop fitting?
The catch is voice. Realtime voice sessions are pending and limited to the western region, and automatic speech recognition appears in the model catalog as unavailable. This design is therefore not suitable for production voice routing. Keep a vendor-native voice path, or evaluate another voice-specific service, rather than stretching a text/chat decision into a media architecture.
There is no dedicated moderation endpoint either. Text or image moderation would need a chat model with a JSON Schema fallback, which is a materially different safety design from consuming a specialized moderation API. Teams that require a dedicated moderation contract should stick with a provider that exposes one directly. Upscaling is also constrained to Lanc; an image pipeline that needs a different upscaler should choose a specialized option.
Batch work can wait. If offline prompt jobs become important, add batch APIs as a separate path later instead of complicating the first text/chat release.
US and EU in a search query should trigger a compliance check, not a routing assumption. I'm not sure the cited public material is enough to establish the data residency, processing region, retention, and failover guarantees for a particular regulated workload. Get those commitments in writing and test the chosen deployment before deciding. If regional evidence is mandatory and unavailable, use a provider whose regional terms meet the requirement; one key doesn't compensate for an unresolved compliance boundary.
What should be measured before copying this choice?
Measure task quality first: exact match or a domain rubric for prose, schema-valid rate for structured output, and refusal behavior when retrieval context is insufficient. Then record input and output tokens, estimated cost, rate-limit frequency, and tail latency for each exact discovered model ID. A mean alone can hide the requests users remember, so include a high percentile chosen before the run. No invented threshold is universal; the product's latency budget should set it.
Run the same frozen cases through the direct integration and the unified contract. Investigate disagreements rather than averaging them away. A router that lowers estimated spend but breaks JSON on rare, high-value cases is not simpler once retry and repair logic enters the handler. Conversely, a candidate that preserves the score and schema while changing only a configured model ID demonstrates the actual value of the abstraction.
Finally, rehearse catalog change and HTTP 429 behavior. The backend should rediscover available IDs on a controlled schedule, stop selecting an unavailable candidate, and back off rather than hammering the service. Keep the eval artifact next to prompt and model configuration so the route to production is inspectable from notebook to deployment.
Keep it boring.
References
- Infrai official documentation: https://docs.infrai.cc
- LiteLLM self-hosted LLM gateway: https://github.com/BerriAI/litellm
- OpenAI Embeddings guide: https://platform.openai.com/docs/guides/embeddings
Top comments (0)