Every "best LLM gateway" list we've read ranks products by feature checklist: routing ✓, caching ✓, guardrails ✓. Checklists are where evaluation goes to die — every gateway claims every feature, and the differences that decide whether your production traffic survives a bad Tuesday live one level down, in behaviour you can only see by testing.
We build nRouter, so apply whatever discount you like to our inclusion in the set below — but the criteria are the point, and they're testable against any gateway, including ours. Here's the test bench we'd run before trusting one with real traffic.
1. Failover you can prove, not a diagram
The test: kill the primary provider mid-request in staging — a hard failure, not a graceful one — and watch three things: does the client see an error, how much latency did the fallback add, and did the fallback go where your policy said it would?
What separates gateways: whether fallback chains are explicit policy (this model, then that provider, then degrade) or a provider's internal behaviour you can't inspect. Silent substitution is the failure mode to hunt: the request "succeeds," but on a different model than the one your evals approved.
2. Retry discipline
The test: force 429s and 500s and count what the gateway actually does. Which statuses does it retry? With what backoff? Does it respect idempotency — or will it happily re-run a non-idempotent call and bill you for the duplicate?
A gateway that retries everything is a cost incident waiting for a trigger. A gateway that retries nothing is a reliability feature that exists only in the brochure. The behaviour table should be documented, and your staging run should match it.
3. Budget enforcement in the request path
The test: set a small cap on one key, then deliberately exceed it. What happens — and when?
Alerts are not enforcement. Dashboards are not enforcement. Enforcement is the request being refused before the spend happens, per key and per team, with a refusal you can distinguish from a provider error (a budget 429 and a provider 429 must never look identical in your logs). If the cap only exists as a chart, a runaway loop will find the difference before you do.
4. A request log that works as an audit trail
The test: pick one request from last week and reconstruct it fully: which model actually served it, what it cost, how long each attempt took, whether a fallback fired, which key made it.
If you can't answer all of that from the log, the gateway is a black box you're renting blind. Two sub-tests: prompts and outputs need redaction controls (logs are where PII goes to be forgotten about), and the log must be exportable — an audit trail you can't take with you isn't yours.
5. Keys and access control that match how teams actually work
The test: one app gets compromised keys at 2 a.m. How fast can you revoke exactly those keys — without touching any other team's traffic?
One shared key for a whole organisation fails this by construction. Look for virtual keys scoped per app, environment and team, with revocation measured in seconds. Then check the boring part: can a key be limited to specific models, or is every key an all-access pass?
6. Guardrails placed in the path, at a known cost
The test: run a prompt-injection and PII suite through the gateway and measure two numbers — what it catches, and the latency it adds per call.
Every serious gateway offers checks of some kind now (OpenRouter documents its own, as do we); the real differences are placement and price in milliseconds. A check bolted on as a separate service adds a hop and a failure mode. A check in the request path adds neither — but should tell you exactly what it costs.
7. An exit you can afford
The test: if you left this gateway next quarter, what breaks?
The honest answer should be "one base-URL change." An OpenAI-compatible surface, standard auth, your logs exportable, your prompts portable. The best gateway is confident enough to be easy to leave — lock-in is a business model, not an architecture.
The set we'd run these against
Run the bench on the gateways you're actually considering. Ours would include: OpenRouter (a marketplace model — broad catalogue, provider choice as the product), Portkey (observability-led, deep analytics heritage), LiteLLM (self-hosted and open source, total control at the cost of running it yourself), and nRouter (a managed gateway built around routing policy, in-path budget enforcement and audit-grade logging). Different architectures optimise for different criteria above — which is exactly why the checklist version of this decision keeps failing people. Run the tests. The behaviour table doesn't care whose logo is on it.
What did we miss? If you've production-tested a gateway, which criterion decided it — and which one on this list would you strike?
Deeper version of the gateway's job description: What is an LLM gateway?
Top comments (0)