An LLM feature usually starts with one OPENAI_API_KEY in a .env file. It stops being that once a second provider shows up, then a second team, and then someone asks why inference spend doubled last month and nobody can break the number down.
What an AI gateway is
An AI gateway is a proxy between your applications and the model providers you call. Instead of every service holding its own provider keys and its own retry logic, they all talk to one endpoint.
Without a gateway, every service is on its own:
┌──────────────┐
│ chat-api │──── OPENAI_API_KEY ──────────▶ OpenAI
├──────────────┤
│ summarizer │──── ANTHROPIC_API_KEY ───────▶ Anthropic
├──────────────┤
│ batch-job │──── OPENAI_API_KEY (copy) ───▶ OpenAI
└──────────────┘
Keys copied into every service. No shared view of cost.
A provider outage is handled — or not — three separate times.
With one in place:
┌──────────────┐
│ chat-api │──┐
├──────────────┤ │ ┌─────────────────────────┐ ┌───▶ OpenAI
│ summarizer │──┼──▶│ AI GATEWAY │──────┼───▶ Anthropic
├──────────────┤ │ │ │ └───▶ Bedrock
│ batch-job │──┘ │ • routing & failover │
└──────────────┘ │ • virtual keys │
one base URL │ • budgets, rate limits │
one credential │ • cost per request │
│ • traces & metrics │
└───────────┬─────────────┘
│
▼
Prometheus / OpenTelemetry
→ your existing dashboards
The gateway can decide which provider serves each call. That gives you failover when one goes down, cost attributed per team, budgets that stop runaway spend, and traces in the dashboards you already use.
Proxying is the easy part; every product does it. The differences are in that second list — and in how much of it you get before hitting a paywall.
In this article, we’ll explore four AI gateways: Bifrost, LiteLLM, Cloudflare AI Gateway, and Vercel AI Gateway.
The two families
These four gateways split cleanly into two distinct categories, and this split matters more than any individual feature:
Self-hosted (Bifrost, LiteLLM) — you run the gateway and keep its configuration, provider credentials, governance state, and logs on infrastructure you control. Prompts remain inside your network only when the upstream model endpoint does too. You also own the database, upgrades, backups, and on-call.
Managed (Cloudflare, Vercel) — the vendor runs it. Setup takes minutes, there is no infrastructure to manage, and both integrate tightly with their native platforms. You cannot self-host, and your prompts transit their network.
That choice constrains everything that follows. Within each pair, the differences come down to emphasis:
| Strongest at | Main limitation | |
|---|---|---|
| Bifrost | Governance on a bare container; MCP gateway; low overhead | Fewer providers than LiteLLM; HA, SSO, audit logs, and guardrails are Enterprise-only |
| LiteLLM | Provider coverage; routing flexibility; external observability integrations | Production topology is genuinely complex |
| Cloudflare | DLP and guardrails; visual dynamic routing; no infrastructure | Account-scoped tokens; Response-side DLP buffers streaming responses |
| Vercel | Developer experience; modality breadth; provider-level routing | Advanced governance is metered; no self-hosting |
Self-hosted: Bifrost
Apache 2.0, Go, in front of 1,000+ models across 23+ providers, with 7.7k stars on GitHub.
What stands out: governance is running before you configure anything.
Telemetry, logging, and governance are auto-loaded built-ins — in the maximhq/bifrost:latest image I pulled on 30 August 2026, seven internal plugins registered at startup, those three among them. What that means in practice: /metrics serves Prometheus without adding an exporter, and the virtual key, team, budget, and customer endpoints answer immediately against the default SQLite store. Creating a key with a daily spend cap and a token rate limit is one call, no external database:
curl -X POST localhost:8080/api/governance/virtual-keys \
-H 'Content-Type: application/json' -d '{
"name": "team-search",
"budget": {"max_limit": 10.0, "reset_duration": "1d"},
"rate_limit": {"token_max_limit": 1000, "token_reset_duration": "1m"}
}'
The virtual key is the organising primitive: model access, budgets nested across key/team/customer, rate limits, routing, and which MCP tools the key may reach all hang off it. That single point of control is the design's real argument.
Observability is built in rather than bolted on. Inputs, outputs, model, provider, token counts, cost, latency, and errors are recorded automatically, stored in SQLite or PostgreSQL, and exportable to OTel, Prometheus, Datadog, Kafka, Splunk, BigQuery, and Pub/Sub. Content logging can be disabled for the internal log store, per request, and separately for each observability connector, so metadata such as cost and latency can be retained without exporting prompt bodies.
Fallbacks are per-request. I pointed a provider at a dead address and the request completed through the fallback, with the substitute model named in the response.
It is also an MCP gateway: MCP servers and tools are centrally managed, and each virtual key can be restricted to a subset of tools. For agent workloads that is the same governance model extended one layer up.
Limitations. Provider coverage is roughly a quarter of LiteLLM's, and the community is the smallest here — fewer existing answers when something breaks. Clustering, RBAC, OIDC provisioning, audit logs, and guardrails are Enterprise, as are vendor-supported private-network, on-premise, and air-gapped deployments — the OSS container itself runs anywhere you can run a container. The zero-config SQLite is also a single-instance story: the migration docs call it "single-writer and single-host by design, which is incompatible with HA clustering, cross-region replicas, and the connection-pool patterns Enterprise relies on," so scaling out means moving the config and log stores to PostgreSQL. Semantic caching needs a vector store attached before it does anything.
Self-hosted: LiteLLM
Python SDK plus proxy, 100+ providers — the widest coverage here — and the larger open-source project by a wide margin at 57.6k GitHub stars against Bifrost's 7.7k.
What stands out: the widest provider catalogue by a large margin.
OpenAI, Anthropic, Azure OpenAI, Bedrock, Vertex AI, Cohere, Mistral, Groq, Ollama, vLLM — cloud and self-hosted, across chat, embedding, image, audio, and rerank endpoints. If you evaluate new models as they land, or run your own inference alongside hosted APIs, this is the catalogue most likely to already cover it.
Routing and multi-tenancy are correspondingly flexible: virtual key auth, user/team/organization hierarchy, budgets at key, user and team level, RPM and TPM limits, model groups, load balancing across deployments, retries, in-model failover, cross-model fallback, A/B testing, and traffic mirroring.
The free tier is not a teaser. Per the OSS page and enterprise docs: virtual keys, spend tracking, budgets, rate limits, global guardrails, request/response logging, fallbacks, load balancing, and Prometheus metrics. That last one is worth pinning down, since third-party comparisons list it as Enterprise. It isn't — enabling the callback on the OSS container (main-latest, pulled 30 Aug 2026) exposed around 150 litellm_* series covering request counts, latency histograms, and spend, labelled by model, team, and key, with no license gate:
litellm_settings:
callbacks: ["prometheus"] # OSS; served at /metrics/
If you already run an observability stack, this is the gateway that fits it: Langfuse, OpenTelemetry, LangSmith, Datadog, Arize, MLflow, S3, GCS, Azure Blob, Azure Sentinel, and custom callbacks are all supported destinations.
Limitations. Production topology is the real cost. Virtual keys need PostgreSQL (the docs are specific it is not SQLite), Redis is required "as soon as you run more than one proxy instance," and past "roughly 1000+ requests per second" the production guide recommends a Redis transaction buffer. That's sound engineering — contended budget counters across replicas need a shared transactional store — but it is more to stand up than Bifrost asks for, and the guide is required reading rather than optional.
SSO is free for up to five users. Larger SSO deployments, SCIM, audit logs, secret-manager integrations, key rotation, and advanced RBAC require Enterprise.
Managed: Cloudflare AI Gateway
23+ providers, with analytics, logging, caching, rate limiting, retries, dynamic routing, guardrails, and DLP. Nothing to self-host, so everything here is from Cloudflare's docs rather than something I ran.
What stands out: security controls that live in the gateway.
DLP applies to prompts and responses passing through, with built-in profiles for financial and identifier data and custom profiles available under Cloudflare One. Guardrails detect harmful content and can flag or block it. If your reason for wanting a gateway is that prompts might contain data that should not reach an upstream model provider uninspected, this is the one that treats that as a first-class problem.
Dynamic Routing is configured in a visual editor or as JSON: conditional branches on request body, headers, or metadata; per-user, per-project, per-team rate limits; budget limits; model fallback; retries and timeouts; A/B tests; percentage rollouts; route versioning; and instant rollback. Changing which model serves traffic doesn't require an application deploy — and someone who doesn't write code can read the routing graph.
Spend limits put dollar budgets in the request path, and Dynamic Routing can step down to a cheaper model when the primary model’s budget is exhausted. LiteLLM and Bifrost also support budget-triggered fallback paths; Cloudflare’s distinction is that the policy is expressed directly in the same visual routing graph.
Telemetry isn't trapped in the dashboard. The OpenTelemetry integration exports spans in OTLP JSON or protobuf to any compatible backend, carrying model, provider, token counts, prompts, completions, cost estimates, and custom metadata. A GraphQL analytics API and Workers Logpush cover raw logs.
Limitations. Two are worth knowing before you commit. Tokens are account-scoped: per the authentication docs, AI Gateway Run permission "cannot be restricted to a single gateway," so isolating tenants means separate accounts or Worker-side bindings. And DLP on streaming responses removes the streaming — it "buffers the full streamed response before scanning," so time-to-first-token grows to the full generation time. Request-only scanning avoids that.
Managed: Vercel AI Gateway
Hundreds of models across many providers behind one endpoint, with BYOK on the paid tier.
What stands out: the shortest distance from idea to working feature.
Integration with the Vercel AI SDK is seamless — provider/model strings, one endpoint, many models — and it also speaks OpenAI Chat Completions, OpenAI Responses, and Anthropic Messages, so it isn't Next.js-only. Modality coverage is the broadest here: text, image, video, speech, speech-to-text, and realtime voice through one gateway, which makes it closer to a multimodal API hub than an LLM proxy.
Provider routing is unusually granular. By default the gateway "dynamically chooses the default providers based on a combination of recent uptime and latency," and you can override with provider options: order to set preference, only to restrict, and sort by cost, ttft, or tps. Per-provider timeouts for BYOK credentials trigger fast failover; model fallbacks handle the case where the entire model is unavailable. Controlling which provider serves a given model is something the others expose far less directly.
Telemetry has a property none of the others match by default: trace drains emit an OpenTelemetry trace per request, and "traces contain request metadata but not prompt or completion content." The others can be configured to exclude prompt bodies; here it is not configurable, which is a shorter conversation with a compliance reviewer.
Budgets can be set at team, project, API-key, and team-member level. Member budgets are enforced through spend attribution: API keys assigned to a user count against both that member’s budget and the team budget. Zero Data Retention routing restricts traffic to providers with ZDR agreements. Audit Log Drains forward team activity to S3, Splunk, Datadog, or Panther.
Limitations. Tokens carry no markup, but several governance and observability features are metered separately, per the pricing page. Team-wide provider allowlisting and team-wide ZDR each cost $0.10 per 1,000 requests. Custom reporting costs $0.075 per 1,000 tag, user-ID, or quota-entity writes, plus $5 per 1,000 reporting queries. Trace drains cost $0.05 per 1,000 traces plus egress. Per-request provider filtering and per-request ZDR do not carry these team-wide surcharges.
DLP and content guardrails aren't the focus they are at Cloudflare; PII masking and prompt injection detection stay in your application.
Cost structure
| How you pay | |
|---|---|
| Bifrost | OSS free; you pay for infrastructure. Enterprise quoted individually |
| LiteLLM | OSS free; you pay for infrastructure. Enterprise quoted on annual request capacity |
| Cloudflare | Analytics, caching, rate limiting free. Unified Billing adds 5% on credit purchases; BYOK avoids it |
| Vercel | Provider list price, no token markup. Governance add-ons metered per request; BYOK needs paid tier |
The OSS licenses are free, but production deployments can still add infrastructure and operational costs: PostgreSQL, Redis, vector storage, backups, monitoring, and the people to run them, depending on the topology and features you enable.
How to choose
Bifrost — you want to self-host, want observability and governance included rather than assembled, need virtual keys tying together budgets, routing and model access, and want an MCP gateway in the same product. Fewer moving parts to stand up than LiteLLM: no Postgres or Redis until you run replicas. Start with the quickstart; the product page covers where the Enterprise line falls.
LiteLLM — provider coverage is the deciding factor, you use niche or self-hosted models, you already run Langfuse/Datadog/OpenTelemetry and want the gateway to feed them, or you need complex routing and multi-tenancy. Assumes you have a platform team.
Cloudflare — you already use Cloudflare Workers or Cloudflare One, you don't want to manage gateway infrastructure, DLP and content guardrails matter, or you want visual routing with A/B tests and staged rollouts.
Vercel — you're on Next.js or the Vercel AI SDK, you want the fastest path to a working feature, you need image/video/voice alongside text, or you want automatic provider selection with a clear dashboard.
Bottom lines
Pick the family first — self-hosted or managed — because that decision constrains everything downstream. Then pick on the constraint that actually binds you: provider coverage, data residency, DLP, or developer velocity.
The alternative to picking one is the arrangement at the top of this post: provider keys copied across services, and spend that only becomes visible when the invoice lands.
Disclosure: this post was made in collaboration with the Bifrost team; the testing, the comparisons, and the conclusions are mine.
Top comments (0)