Choosing an LLM Gateway in 2026: Put Retry Policy Before Model Count

I would start a gateway evaluation with a failure, not a successful completion.

Switching the model field is useful, but it tells me little about production behavior. I want to know what happens when the primary provider times out, whether the next route preserves the task’s contract, and whether I can account for every billable attempt.

The real decision is where the control plane lives: application code, a managed service, a self-hosted proxy, or an edge route.

Start With the Ownership Boundary

These five options solve overlapping problems, but leave different amounts of work in your codebase.

Option	Where policy lives	Main advantage	What I would scrutinize
Hosted unified API	Application code for the documented fallback pattern	Minimal client integration	Retry state, circuit breakers, and route budgets
Portkey	Managed gateway configuration	Routing policy and attempt-level observability together	Provider-specific behavior
OpenRouter	Hosted model and provider routing configuration	Provider-marketplace controls	End-to-end application tracing
LiteLLM	Self-hosted proxy and router configuration	Customization and operational control	Proxy, storage, upgrades, secrets, and policy operations
Cloudflare AI Gateway	Edge routes	Routing, logs, and limits alongside Cloudflare infrastructure	Supported API and authentication paths

My default shortlist would be Portkey for managed policy, LiteLLM when owning the gateway is a requirement, OpenRouter for provider selection, and Cloudflare for an existing edge deployment. A hosted unified endpoint makes sense when I deliberately want fallback logic to remain in the application.

A unified endpoint with application-owned fallback

CometAPI provides a large hosted model catalog through the OpenAI-compatible base URL https://api.cometapi.com/v1. The same client and API key can select another catalog model by changing model.

Its public model directory exposes model IDs, capabilities, prices, and endpoints for deployment-time validation. Usage comes through responses and quota and daily usage queries; request logs, a dashboard, per-key quotas, and request-level output limits cover basic visibility and controls.

The important boundary is the documented fallback pattern: call the primary model, switch models on a retryable failure, and optionally call an official provider last. That sequence runs in application code.

That is a valid production design. It also means each service owns retry logic, circuit-breaker state, and per-route budgets unless you centralize that code yourself.

Portkey: managed routing and tracing together

Portkey’s gateway combines a universal API with config-driven routing. Its controls include conditional routing, prioritized fallbacks, retries, circuit breakers, and load balancing.

The reason I would evaluate it first for managed policy is the combination of native fallback configuration, attempt-chain visibility with Config ID and Trace ID, and cost management. Token and cost attribution, budgets, rate limits, and policy guardrails belong in the same operational workflow.

That reduces custom control-plane code. It does not remove the need to test provider-specific behavior.

OpenRouter: provider selection is the point

OpenRouter’s provider routing is useful when the choice is not just which model, but which provider for that model.

It supports provider ordering, price or latency preferences, parameter compatibility, automatic provider fallback, and configurable model routing. Price sorting, maximum-price rules, and key limits give that selection process spending constraints.

Its Analytics and Activity history cover usage. I would still plan for another observability layer if I needed a request traced through the application and every downstream attempt; that is a different requirement from usage history.

LiteLLM: control comes with an operations bill

LiteLLM exposes an OpenAI-compatible proxy and router across many providers, with model aliases in self-hosted configuration.

The control surface is broad: retries, fallbacks, budgets, rate limits, spend and token tracking by user, key, or project, plus logging hooks and external callbacks. It also supports per-key and per-model limits.

I would choose it when customization or infrastructure ownership matters enough to justify operating the proxy, storage, upgrades, secrets, and policy configuration. Self-hosting moves the responsibility; it does not eliminate it.

Cloudflare AI Gateway: policy on the edge

Cloudflare AI Gateway fits naturally when the surrounding infrastructure already runs on Cloudflare.

Its Dynamic Routing can select routes by conditions, enforce rate or budget limits, and send failed or over-limit requests to fallback models. Unified routes, dashboard analytics, persistent request logs, and spend limits round out the operational controls.

Before standardizing on it, I would verify the API and authentication path supported by the intended deployment.

Define the Contract Before Testing the Gateway

I care about five capabilities, but I would not score them as simple checkboxes.

Model switching needs a stable client contract—typically an OpenAI-compatible /chat/completions endpoint—with selection through a parameter, alias, configuration, or route. All five options support switching. None of that proves every model accepts the same parameters.

Fallback needs an explicit error policy. “Try another model whenever something fails” is how configuration mistakes become expensive and difficult to diagnose.

Usage tracking needs model attribution, request counts, prompt tokens, and output tokens for every attempt, not just the final successful response. Failed attempts that consume partial tokens can still be billed upstream.

Tracing needs to connect latency, status, model, provider, and route decisions under one request ID. A final HTTP 200 can conceal a broken primary route for weeks.

Cost control needs enforceable limits. A dashboard is reporting, not a guardrail. Unbounded retries can turn one request into hundreds of attempts; an unnoticed fallback to a 10x pricier model can make the spending impact severe.

Run the Same Failure Test Against Every Candidate

I would use one repeatable script covering:

• A normal request.
• A deliberately rate-limited request.
• A timeout.
• An invalid API key.
• An invalid model ID.

The safe default policy is:

Failure	Default action
Connection error or timeout	Bounded retry or fallback
HTTP 408 or 429	Bounded retry or fallback
Temporary HTTP 5xx	Bounded retry or fallback
HTTP 400	Hard failure
HTTP 401 or 403	Hard failure
Unknown-model HTTP 404	Hard failure

Invalid requests, unsupported parameters, authentication problems, and unknown models should not silently turn into alternate-provider traffic.

Cap retries, apply backoff, and add a circuit breaker. Otherwise, a provider outage can become a spend multiplier.

The minimum expected log shape is:

{"request_id": "...", "model": "...", "status": 200, "latency_ms": , "usage": {...}}

That is a schematic record, not literal JSON to submit. The test passes only when failed attempts also appear under the same request ID. I would additionally require provider, route decision, tokens, and cost so the fallback chain is reconstructable.

A fallback must preserve the task, not merely return text

For every alternate model, verify:

• Tool-call behavior.
• Structured-output and JSON-schema compatibility.
• Streaming format.
• Safety and content-policy behavior.

A route that returns text can still silently fail the task. Gateway-level compatibility is not model-level interchangeability.

Model the Cost of Attempts, Not Responses

For prices quoted per million tokens:

attempt cost = (input tokens × input price + output tokens × output price) / 1,000,000

The source’s dated pricing example, attributed to the public model directory API on September 02, 2026, uses:

Model	Input per million tokens	Output per million tokens
Gemini 3.7 Flash	$0.75	$3.75
Claude Opus 5	$5	$25

Treat those as the example’s dated inputs, not a substitute for checking live prices.

At 1,000 successful Gemini requests, averaging 2,000 input tokens and 500 output tokens, the modeled primary cost is $3.375.

If 5% also run on Claude Opus 5 as a quality-first fallback with the same token volume, those additional attempts cost $1.125. The total becomes $4.50, before any billable partial primary attempts.

This is why I want separate counts for primary and fallback attempts, alongside tokens, latency, and cost. A successful-response count cannot explain the bill.

For reconciliation, compare:

1. API response usage.
2. Attempt-level request logs.
3. Daily usage or quota reports.
4. The final invoice.

Model, attempt count, and token volume should agree across those records.

My Deployment Gate

Before sending production traffic, I would require:

• Explicit retry, fallback, and hard-failure rules.
• Bounded retries, backoff, and a circuit breaker.
• One request ID across every attempt.
• Model, provider, status, latency, tokens, and cost in the logs.
• Per-tenant quotas or budgets, with alerts before the hard limit.
• Live-catalog validation of model IDs.
• Compatibility tests for tools, structured output, streaming, and safety behavior.
• A review of data retention, provider routing, and regional requirements before enabling logs.

A gateway does not automatically lower costs. It gives you mechanisms to select cheaper routes, constrain spending, and see retries. The result still depends on model mix, failure rate, route policy, and billable failed attempts.

I would make the final choice from four artifacts: a dated feature matrix, a repeatable failure test, an attempt trace, and a reconciled cost calculation. Those tell me much more than a catalog size or a “fallback supported” badge.