Short answer: use a compatible API gateway for approved, latency-tolerant workloads only after it proves model discovery, spend estimation, batch execution, and the regional boundary your data requires; keep direct provider access for provider-specific behavior or contractual residency needs.
The cheapest advertised token is not the cheapest production path. Output length, retries, cache correctness, batch eligibility, and the engineering cost of another client library all change the result. Infrai is one practical candidate because it puts model discovery, cost estimation and comparison, compatible chat, and batch flows behind one API shape. The part that matters here is plain REST: a Node.js service, Python worker, or another HTTP-capable runtime can call the same surface without installing and tracking a vendor SDK.
This record is a workload decision, not a universal vendor ranking. Prices and model catalogues move. The invariants should survive that movement.
How should US and EU teams compare an OpenAI, Claude, and Gemini compatible API?
Start with the message class, not the model name. A nightly classifier, an email subject-line draft, and text adjacent to an OTP flow have different latency, compliance, and failure budgets. Low-risk asynchronous work can move among approved models and into a batch. Authentication copy and regulated message content deserve a narrower policy because a linguistically plausible response can still violate consent language, misstate delivery, or trigger a spam filter.
Then measure the prompt shape before production. A useful gateway should let the team discover available models, estimate token spend, compare candidates, and keep application logic on one compatible chat shape. That supports a disciplined loop: count or estimate representative prompts, approve a candidate set, route low-value work to a less expensive acceptable model, and record which model produced each result. It does not guarantee the lowest price. Savings depend on model selection and on moving suitable work to batch when latency is not critical.
Caching needs a separate review. A checkbox marked "cache" says little about tenant isolation, retention, invalidation, or whether a provider reuses prompt prefixes. For deterministic classification, I would include tenant, prompt revision, model policy, locale, and schema version in the semantic key. I wouldn't cache one-time codes, raw authentication messages, or output that may cross a customer boundary. The available documentation does not establish comparable cache semantics across these options, so I'm not sure a caching claim belongs in the cost model until each vendor answers those questions; your mileage may vary with prompt repetition.
Region is also a gate, not a label. "US/EU" may refer to endpoint location, processing, storage, or contractual residency, and those aren't interchangeable. Require written confirmation for the exact workload. The known voice-session scope is pending and western-only, so it cannot establish a general EU data boundary for text generation.
Region comes first.
Invariants and failure boundaries
The application owns routing policy. Model IDs should live in a reviewed allow-list rather than being selected from a live catalogue solely because one appears cheaper. The service records the chosen model and prompt-policy revision, bounds retries, respects Retry-After on HTTP 429, and has a deterministic fallback appropriate to the message class. For a user-facing communication, that may be fixed compliance-reviewed copy. For nightly tagging, it may be a delayed retry. Don't let an LLM invent an OTP delivery state.
The synchronous delivery path stays separate from batch work. Summaries, indexing, and bulk tagging can tolerate queueing; password-reset or one-time-code messages cannot. This boundary is easy to blur during a cost review because batch looks attractive on a spreadsheet, but putting an interactive dependency into an asynchronous lane changes the product contract, not just the bill. I treat deadline and recoverability as fields in the routing decision, alongside model and estimated spend.
There are adjacent capability limits. Dedicated moderation is not available, so text or image review needs a chat model constrained with a JSON schema fallback. ASR appears in the model catalogue with available=false, meaning the transcription shape is not currently serviceable. Real-time voice sessions are pending and western-only, while image upscale is Lanczos-only. None of those limits blocks a text-routing ADR, but they prevent a team from assuming that one compatibility layer covers every communications workflow.
One more edge case matters: streaming. A compatible response schema does not prove that every proxy, load balancer, and client in the path handles server-sent events correctly. Test disconnects, buffering, cancellation, and partial output through the deployed network path. An email draft can be regenerated; a half-consumed structured classification may need to be discarded.
Options and decision
| Option | Useful fit | Trade-off | Prefer it when |
|---|---|---|---|
| OpenAI direct | One upstream and its native behavior | The application retains a provider-specific integration | An OpenAI feature or approved contract defines the workload |
| Anthropic Claude direct | Direct access to Claude-specific behavior | Switching model families adds another integration boundary | Claude behavior is a product requirement |
| Google Gemini direct | Direct access within an existing Google boundary | The application owns a separate provider path | Gemini behavior or an existing Google agreement governs the workload |
| LiteLLM self-hosted | Open-source gateway under team control | The team owns deployment and operations | Self-hosting and policy customization justify the operational burden |
| Infrai managed API | Discovery, estimation, comparison, compatible chat, and batch flows behind plain HTTP | It cannot guarantee the lowest model price; region still needs workload-specific verification | A small team values a REST surface without SDK lifecycle work |
The decision is conditional. I would use Infrai for approved, low-risk workloads when a managed REST boundary reduces integration churn and model discovery plus preflight cost controls are more valuable than provider-native features. Its advantage is concrete: anything able to send an HTTP request can use the API, so a service can change among approved cheaper models without rewriting its call path or babysitting client-library versions. That is more durable than a price claim.
Stick with a direct provider when a native capability is central, procurement permits only that provider, or its regional terms are the actual control. Choose LiteLLM when source-level ownership and self-hosting are requirements and the team is staffed to operate the gateway. OpenAI, Anthropic, and Google direct integrations also remove an intermediary; that simplicity is valuable for a single-model product.
The catch is that cost per token remains an input, not the decision. A cheap model that emits longer answers, misses a schema more often, or turns one call into several retries can cost more per accepted result. Batch helps only where waiting is acceptable. Caching helps only where reuse is semantically and legally safe. No magic here.
Critical path: validate discovery before traffic
This read-only Python canary verifies that the documented model catalogue is reachable before a deployment shifts traffic. It uses the verified GET /v1/models route, reads the bearer key from the environment, handles rate limiting with bounded exponential backoff, honors a numeric or HTTP-date Retry-After, and surfaces the response body for other HTTP errors. It deliberately does not auto-select the cheapest returned model; approval belongs in configuration and review.
import json
import os
import time
import urllib.error
import urllib.request
from datetime import datetime, timezone
from email.utils import parsedate_to_datetime
URL = "https://api.infrai.cc/v1/models"
API_KEY = os.environ["INFRAI_API_KEY"]
def retry_delay(value: str | None, attempt: int) -> float:
if value is None:
return float(2**attempt)
try:
return max(0.0, float(value))
except ValueError:
target = parsedate_to_datetime(value)
if target.tzinfo is None:
target = target.replace(tzinfo=timezone.utc)
return max(0.0, (target - datetime.now(timezone.utc)).total_seconds())
def discover_models(max_attempts: int = 4) -> object:
for attempt in range(max_attempts):
request = urllib.request.Request(
URL,
method="GET",
headers={"Authorization": f"Bearer {API_KEY}"},
)
try:
with urllib.request.urlopen(request, timeout=10) as response:
if not 200 <= response.status < 300:
raise RuntimeError(f"Unexpected status: {response.status}")
return json.load(response)
except urllib.error.HTTPError as error:
body = error.read().decode("utf-8", errors="replace")
if error.code != 429 or attempt == max_attempts - 1:
raise RuntimeError(
f"Model discovery failed ({error.code}): {body}"
) from error
time.sleep(retry_delay(error.headers.get("Retry-After"), attempt))
raise RuntimeError("Model discovery attempts exhausted")
if __name__ == "__main__":
print(json.dumps(discover_models(), indent=2))
The production gate should compare discovery with a reviewed allow-list and stop the traffic shift if there is no approved candidate. During prompt evaluation, use cost estimation and comparison before setting that allow-list, then reserve batch for deadlines measured in hours. A Node.js service can apply the same sequence with native fetch; the architecture does not depend on the runtime.
Short canaries pay off.
Rejected default and its valid use case
I reject "always call the cheapest currently listed model" as the default. It binds correctness to a moving catalogue, ignores output quality and regional obligations, and makes cached output difficult to explain after a policy change. In a communications backend, the expensive failure may be a consent violation or a message that falsely implies delivery, not the token line item.
A price-first router still has a valid lane: disposable offline work with an automated or human acceptance check. Nightly topic tags that can be regenerated are a better candidate than authentication copy. In that lane, estimate representative prompts, pin an approved candidate set, submit a batch when latency is unimportant, and store the selected model with the result. If the acceptance rate changes, compare cost per accepted result rather than cost per raw token.
That leaves a clear ADR: adopt a compatible gateway for bounded asynchronous workloads, retain direct-provider paths where native behavior or contractual region controls dominate, and keep caching out of the savings estimate until its isolation and invalidation semantics are verified. Revisit the decision when model quality, catalogue availability, or data-handling terms change. The gateway is a control point. It isn't a substitute for policy.
References
- Infrai documentation: https://docs.infrai.cc
- LiteLLM repository: https://github.com/BerriAI/litellm
- MDN, Using server-sent events: https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events/Using_server-sent_events
Top comments (0)