OpenRouter's own numbers put it at 200+ trillion tokens routed per month, across 400+ models from 70+ providers, for more than 10 million users. A proxy in front of model APIs stopped being a convenience and became production infrastructure. At the same time, the companies building these proxies had a brutal 2026: Palo Alto Networks bought Portkey in May, Mintlify absorbed Helicone in March and froze it, Martian pivoted to interpretability research, and Not Diamond shrank its pitch to routing for coding agents only.
This is the short version of a longer guide I published on DevToolLab, Best LLM Gateways and API Routers in 2026, which has the full pricing breakdowns and more code.
What a gateway actually does
An LLM gateway (same thing as an AI gateway or API router) is a proxy between your app and the providers. Your code calls the gateway instead of api.openai.com, and the gateway handles everything around the model call:
- One OpenAI-compatible format for every provider, so switching models is config, not a rewrite.
- Automatic fallback and retries when a provider has an outage or rate-limits you.
- Centralized credentials: services get scoped virtual keys, the gateway holds the real ones.
- Caching (exact or semantic) so duplicate prompts do not hit the bill twice.
- Budgets, rate limits, and per-team cost attribution.
It is the load balancer argument, applied to model APIs. The pain that pushes teams there is always concrete: twelve services calling gpt-4o directly all time out together during one OpenAI incident, or a production Anthropic key ends up pasted into three repos because there was no virtual key system.
The four main options
| Gateway | Type | Self-hosted | Standout cost fact |
|---|---|---|---|
| OpenRouter | Hosted proxy | No | No token markup; 5.5% fee (min $0.80) on card top-ups |
| LiteLLM | OSS SDK + proxy | Yes | Free; enterprise tier is quote-only |
| Portkey | OSS gateway + hosted | Gateway only | Hosted Production tier $49/mo for 100k logs |
| Cloudflare AI Gateway | Hosted proxy | No | Free on all plans; Logpush $0.05/M records past 10M |
OpenRouter
OpenRouter is the default answer when you want the widest catalog and zero infrastructure. There is no markup on per-token pricing; the business runs on a 5.5% credit top-up fee (minimum $0.80), 5% on crypto, and 5% on bring-your-own-key traffic past the first million free requests a month. It raised a $113M Series B in May 2026 led by CapitalG with Nvidia's NVentures participating, at roughly a $1.3B valuation, so it is not going anywhere soon.
Integration is a base URL swap on the official OpenAI SDK, with provider/model slugs:
from openai import OpenAI
client = OpenAI(
base_url="https://openrouter.ai/api/v1",
api_key="<OPENROUTER_API_KEY>",
)
completion = client.chat.completions.create(
model="anthropic/claude-sonnet-4.5",
messages=[{"role": "user", "content": "Summarize the CAP theorem in two sentences."}],
)
print(completion.choices[0].message.content)
The tradeoff: it is a closed-source hosted proxy sitting between you and every model call.
LiteLLM
LiteLLM is the opposite bet: fully open source (about 55,000 GitHub stars, built by YC W23 company BerriAI), unifying 100+ providers behind an OpenAI-compatible interface you run yourself. Use it as a Python SDK, or more commonly as a standalone proxy your whole team points at, which also gives Go and Node services the same unified HTTP interface. A config.yaml maps friendly model names to real deployments, and litellm --config config.yaml puts the proxy on port 4000.
One thing you should not skip if you self-host it: CVE-2026-42271, a CVSS 8.7 command-injection flaw in LiteLLM's MCP test endpoints, affected versions 1.74.2 through 1.83.6. Chained with a Starlette host-header bypass (CVE-2026-48710) it became unauthenticated remote code execution, and it landed in CISA's Known Exploited Vulnerabilities catalog with active exploitation observed. Fixed in 1.83.7. Patch promptly and keep the admin and test endpoints off the public internet.
Enterprise pricing (SSO, RBAC, audit logs) is negotiated per deal, not published.
Portkey
Portkey pairs a fast open-source gateway (~12,500 stars, 1,600+ models, 50+ built-in guardrails) with a hosted observability layer: semantic caching, load balancing, virtual keys, content guardrails. Self-hosting the OSS gateway is free with no log limits; the hosted ladder runs from a free 10k-logs Developer tier (3-day retention, explicitly not for production) to Production at $49/month for 100k logs and Enterprise for compliance-heavy deployments.
The 2026 caveat is ownership. Palo Alto Networks closed its acquisition of Portkey on May 29, 2026, positioning it as part of securing AI agents alongside its Protect AI and CyberArk buys. The OSS gateway still works today, but there is no public roadmap commitment yet, so treat a multi-year bet on it as something to monitor.
Cloudflare AI Gateway
If you already run anything on Cloudflare, this is the five-minute option. It is free on every plan and covers analytics, logging, caching, rate limiting, and fallback across Workers AI, OpenAI, Anthropic, Gemini and Replicate. The paid extras are Logpush ($0.05 per million records past 10M/month, requires Workers Paid) and the new Unified Billing, which puts third-party model spend on your Cloudflare invoice for a 5% fee. It is deliberately not the most feature-dense gateway; it is the one you turn on because you are already there.
Enterprise honorable mentions
Kong AI Gateway extends an existing Kong API-management deployment with semantic caching, prompt compression, PII sanitization and MCP/agent traffic governance; it makes sense mainly if Kong is already your gateway of record. TrueFoundry targets the same buyer with 250+ models, RBAC, budget controls and on-prem deployment, priced from a free tier through $499/month Pro and $2,999/month Pro Plus, with about $21M raised across Peak XV and Intel Capital rounds.
What the 2026 consolidation means for your choice
Four routing startups changed identity within roughly five months: Portkey (acquired), Helicone (acquired and moved to maintenance mode after processing 14+ trillion tokens for ~16,000 orgs), Martian (now an interpretability research company), and Not Diamond (narrowed to coding-agent routing). The category is not dying; it matured fast enough that platforms wanted to own it, and pure routing turned out not to be a defensible standalone business.
The practical filter: pick something you can fork and self-host (LiteLLM, Portkey's OSS gateway), or something hosted by a platform with an obvious reason to keep funding it (OpenRouter, Cloudflare). Be wary of small independents whose roadmap could change hands next quarter.
A minimal first setup
The fastest no-account path is LiteLLM: pip install litellm[proxy], export your provider keys as environment variables, write a config.yaml listing your models, start the proxy, and change only the base_url in your existing OpenAI client to http://localhost:4000. Then add a fallbacks list per model so a provider outage degrades gracefully instead of erroring at your users. The original guide walks through all seven steps with the exact config snippets.
Before you route anything, DevToolLab's AI Token Counter estimates token usage and cost across GPT-5, Claude, Gemini and Llama, and the cURL to Code Converter turns gateway curl examples into fetch, Axios or Python code.
Recommendations
- Widest catalog, nothing to host: OpenRouter.
- Full control, self-hosted: LiteLLM, staying current on patches given the CVE history.
- Guardrails and per-team observability first: Portkey, watching the Palo Alto roadmap.
- Already on Workers: Cloudflare AI Gateway is close to a free lunch.
Calling model APIs directly from a dozen services was never going to survive the second provider outage. In 2026 a gateway is standard infrastructure, not a nice-to-have.
Top comments (0)