Pick an LLM Gateway by Who Runs It, Not by Feature Count

An LLM gateway is a reverse proxy between your applications and model providers. It exposes one API, usually OpenAI-compatible, and applies policy to every request: which model answers, what happens on failure, whether a cached answer gets reused, who may spend how much, and what gets logged.

Every comparison of them ends in a ranked list, and the ranking is useless, because a self-hosted MIT proxy and a credit-funded model marketplace solve different procurement problems. The question that actually narrows the field is who operates the thing.

Four roles:

Self-hosted open-source proxies — LiteLLM, Portkey Gateway, Bifrost, llmgateway.io, Helicone AI Gateway. You own the keys, the logs, and the patching.
Managed gateways — Cloudflare, Vercel, OpenRouter, Portkey Cloud, TrueFoundry. Live this week; you inherit a billing model instead of an ops burden.
API-management platforms with LLM policies — Kong, Azure API Management, Apigee. Cheapest to adopt if your platform team already runs one, hardest to leave.
Cloud patterns — AWS publishes a reference architecture that runs LiteLLM in your own account. No gateway license, just infrastructure.

What makes it an LLM gateway

A classic API gateway terminates TLS, authenticates callers, rate-limits by request count, and routes by path. The LLM layer adds token quotas, semantic caching, provider-format translation, fallback on 429 and 5xx, prompt guardrails, and per-token cost attribution.

One test separates the two: can it reject a request because a team spent its token budget, and can it tell you which key, model, and prompt consumed it? A classic gateway answers neither without custom plugins.

The request path, in order: client → virtual-key auth → policy checks → cache lookup → router → provider adapter → provider API → response guardrail → logging and cost attribution → client. Each stage can short-circuit. An over-budget key stops at policy. A cache hit stops before routing. A provider failure loops back for a fallback.

Read the license before the feature list

This is where the open-source category gets interesting, and where a lot of shortlists go wrong. As documented on 22 October 2026:

• LiteLLM is MIT outside its enterprise/ directory. Virtual keys, budgets, spend tracking, the admin UI, load balancing, guardrails, and an MCP gateway are in open source. The license key gates SSO and SCIM beyond 5 users, audit logs, org and team admins, and secret-manager integrations.

• Portkey Gateway is MIT, with a 2.0 pre-release that merges the enterprise gateway into open source. Last repository push was May 2026, so check which branch you're deploying.

• Bifrost is Apache-2.0 and ships as a single Go binary. Clustering, guardrails, and the MCP gateway are commercial with no published price.

• llmgateway.io is AGPLv3 for the core with a commercial ee/ directory. If you modify the gateway and expose it as a service, have legal read the license before the pilot, not after.

• Helicone AI Gateway is Apache-2.0 and still carried a public beta badge, with its last push in November 2025. Verify maintenance before shortlisting.

The pattern worth internalizing: a LICENSE file that points to a second file for an entire directory means the feature you're evaluating may not be under the license you think it is.

The billing models split four ways

Managed gateways remove the patching duty and replace it with a bill whose shape matters more than its size.

• Free core, paid logs. Cloudflare keeps analytics, caching, and rate limiting free; log retention follows your Workers plan. Observability depth and spend move together.
• Pass-through. Vercel charges no markup and no platform fee on tokens, with payment processing fees unless you invoice as enterprise.
• A cut of credits. OpenRouter takes a percentage on credit purchases, with provider pricing passed through. Painless at $2,000 a month, very visible at $200,000.
• Per recorded log. Portkey Cloud prices by logs recorded, with a free developer tier the vendor itself labels not production-suitable.

And the fifth, for self-hosting: infrastructure only, which looks free until you cost the engineer who runs it and answers the 3 a.m. page.

Eight questions for a one-hour vendor call

Four ways gateways fail

Each is manageable. Each costs engineering time the pricing page doesn't show.

The extra hop. Every gateway adds a round trip and a translation step. Vendors publish their own overhead figures; the only number worth anything is the one measured on your path. Compare p50 and p99 with and without the gateway on identical prompts.

A new tier-one dependency. A gateway everything depends on is a single point of failure. Self-hosted needs HA and a runbook. Managed moves the risk to a vendor's status page, which is a different risk rather than a smaller one. Keep a documented bypass for apps that can't tolerate gateway downtime.

Cache staleness. A cached answer about a price, a policy, or an account balance is wrong the moment the source changes. Scope caching to prompts whose correct answer holds for the TTL, exclude anything personalized, set the similarity threshold per endpoint, and log cache hits separately so support can trace a stale response.

Lock-in through config. Routing rules, guardrail configs and virtual-key hierarchies in one vendor's format are the migration cost you pay later. Prefer plain YAML or JSON in your repo, and treat the OpenAI-compatible interface as the contract your apps depend on.

A two-week pilot

1. Shortlist two gateways from different operating roles — one self-hosted, one managed — so you're comparing cost models rather than feature lists.
2. Mirror production traffic. Send a copy of real requests through each and discard the responses. Shadow traffic surfaces translation bugs that synthetic prompts never hit.
3. Write SLOs before measuring. A p99 ceiling, an availability target, a fallback success rate. A pilot without SLOs produces opinions.
4. Measure cost per accepted request — total spend including engineer-hours, divided by requests that produced a response your application actually used. Cache hits and fallbacks move this more than token prices do.
5. Exercise failure. Revoke a provider key, force a 429, fill a budget. Record what the gateway did and what your application saw.
6. Test the exit. Export the routing config, the logs, the virtual-key hierarchy. If the export needs a vendor ticket, the lock-in surface is bigger than the docs suggest.
7. Decide and document rollback. Write the bypass procedure for direct provider calls, and date the decision so pricing gets rechecked next quarter. One caveat on fallbacks, since it bites people in step 5: every fallback needs its own cost ceiling, or an outage at your cheap provider turns into a bill from your expensive one.

Most teams should adopt rather than build, and that includes teams who could pay someone to build it. LiteLLM, Cloudflare, or Vercel can run on shadow traffic within days from documentation alone. A bespoke gateway is justified when policy requirements exceed every license tier you can afford — data residency that rules out SaaS, air-gapped deployment, MCP governance across dozens of teams. Otherwise the interesting work isn't the gateway, it's the evaluation set behind the routing rules and the observability pipeline into your SIEM.

Full breakdown — per-vendor tables with sources and retrieval dates, pricing comparison, and the full criteria list: see the full guide

Written by Zhanna Yuskevych, CPO at Teamvoy. More engineering writing at teamvoy.com/blog.