TL;DR: An LLM gateway sits between your application and LLM providers, handling routing, failover, cost controls, and observability. I tested five gateways against ten evaluation criteria. Bifrost won on latency and governance. LiteLLM wins on provider coverage. Kong and Cloudflare suit different enterprise needs. Here is the full breakdown.
What Is an LLM Gateway?
An LLM gateway is a reverse proxy purpose-built for LLM API traffic. It normalises requests across providers like OpenAI and Anthropic, adds routing logic, failover, cost controls, caching, and observability without changing your application code. Think of it as an API gateway, but designed specifically for the economics and reliability challenges of LLM calls.
If you are calling more than one LLM provider, or spending more than $500/month on API calls, you need one.
The 10 Evaluation Criteria
I benchmarked and tested five gateways over three weeks. Here is what matters, and what I found.
a. Latency Overhead
The gateway itself should add near-zero latency. You are already waiting 500ms-2s for LLM responses. If your gateway adds another 8-15ms, that compounds across multi-step agent chains.
I measured gateway overhead (not LLM response time) using a standardised Go benchmarking harness:
- Bifrost: 11 microseconds. Written in Go, handles 5,000 RPS sustained. Benchmark details.
- LiteLLM: ~8ms. Python-based, solid for moderate traffic. GitHub repo.
- Kong AI Gateway: ~3-5ms. Built on Kong's proven proxy layer. Product page.
- Cloudflare AI Gateway: Sub-1ms at edge (but limited to Cloudflare's network). Docs.
- Databricks Unity AI Gateway: Not independently benchmarkable. Tied to Databricks runtime.
If latency matters (agents, real-time apps), Bifrost is in a different league.
b. Provider Coverage
- LiteLLM: 100+ providers. Broadest coverage available.
- Bifrost: 19+ providers (OpenAI, Anthropic, Azure, Bedrock, Gemini, Mistral, Cohere, Groq, and more).
- Kong AI Gateway: Major providers via plugins.
- Cloudflare: Major providers only.
- Databricks Unity: Focused on Databricks ecosystem plus external endpoints.
If you need obscure providers, LiteLLM wins. For the top 15-20 providers, any gateway here works.
c. Routing Flexibility
Bifrost supports weighted, priority-based, and conditional routing. Split traffic 70/30 between GPT-4o and Claude Sonnet, or route coding tasks to one model and summarisation to another. LiteLLM has basic load balancing. Kong does routing via plugins. Cloudflare and Databricks offer simpler options.
d. Failover and Reliability
When a provider goes down (and they do), what happens? Bifrost's failover supports automatic retries with configurable backoff and fallback chains. If OpenAI 429s, it rolls to Anthropic automatically. LiteLLM has similar fallback support. Kong uses health checks. Cloudflare and Databricks offer basic retry/fallback options.
e. Cost Governance
This is where gateways diverge sharply. Bifrost has a four-tier budget system: per-key, per-team, per-project, and global with hard limits, soft warnings, and rate limits. Full governance docs. LiteLLM has budget controls via its proxy. Kong and Cloudflare offer rate limiting. Databricks ties into Unity Catalog.
f. Caching
Caching identical or similar LLM calls reduces cost and latency dramatically.
Bifrost supports dual-layer semantic caching with exact match and semantic similarity. Backend options include Redis for exact caching, Weaviate for vector-based semantic matching, and Qdrant as an alternative vector store.
LiteLLM has basic caching support. Cloudflare caches at the edge (great for repeated queries). Kong and Databricks have limited native caching options.
g. Observability
Bifrost's observability captures request/response pairs, token counts, latency, cost, and model metadata with under 0.1ms overhead. Audit logging and virtual key tracking built in. LiteLLM has a dashboard plus integrations. Kong plugs into existing stacks. Cloudflare and Databricks have built-in analytics.
h. MCP Support
MCP (Model Context Protocol) is becoming the standard for tool integration. Gateway-level MCP support matters for managing tool sprawl.
Bifrost's MCP support includes a Code Mode that generates TypeScript declarations instead of raw tool definitions. At 500 tools, this saves 92% on tokens. Tool-level scoping and access control are built in.
Databricks Unity just added MCP governance. Kong v3.14 added A2A (Agent-to-Agent) support in April 2026. LiteLLM and Cloudflare have basic or no MCP-specific features.
If you are building multi-agent systems with many tools, MCP governance is not optional.
i. Deployment Model
-
Bifrost: Self-hosted. Zero-config setup via
npx -y @maximhq/bifrostor Docker. - LiteLLM: Self-hosted (open-source) or managed (enterprise).
- Kong: Self-hosted or managed (Konnect).
- Cloudflare: Managed only. You are on Cloudflare's infrastructure.
- Databricks Unity: Managed. Tied to Databricks workspace.
Self-hosted means your data never leaves your VPC. If you are in a regulated industry, this matters.
j. Open Source vs Proprietary
- Bifrost: Fully open-source. GitHub.
- LiteLLM: Open-source core, enterprise features behind a paid tier. Note: LiteLLM had a supply chain security incident in March 2026 that affected its PyPI package. Worth reviewing before deploying.
- Kong AI Gateway: Kong's core is open-source, but AI Gateway features require an enterprise licence.
- Cloudflare: Proprietary managed service.
- Databricks Unity: Proprietary, part of Databricks platform.
Comparison Table
| Criteria | Bifrost | LiteLLM | Kong AI | Cloudflare | Databricks Unity |
|---|---|---|---|---|---|
| Latency overhead | 11us | ~8ms | ~3-5ms | Sub-1ms (edge) | N/A |
| Providers | 19+ | 100+ | Major | Major | Ecosystem |
| Routing | Weighted, priority, conditional | Basic LB | Plugin-based | Simple | Model serving |
| Failover | Full fallback chains | Fallback support | Health checks | Basic retry | Endpoint fallback |
| Cost governance | Four-tier budgets | Budget + rate limits | Rate limiting | Basic | Unity Catalog |
| Caching | Semantic (Redis/Weaviate/Qdrant) | Basic | Limited | Edge caching | Limited |
| Observability | Sub-0.1ms, full audit | Dashboard + integrations | Stack integration | Built-in analytics | MLflow |
| MCP support | Code Mode, 92% savings | Basic | A2A (v3.14) | None | MCP governance |
| Deployment | Self-hosted | Self-hosted/managed | Self-hosted/managed | Managed only | Managed only |
| Open source | Yes | Core only | AI features paid | No | No |
Decision Framework
Pick Bifrost if you need lowest latency, granular cost governance, semantic caching, and MCP tool management. Self-hosted, open-source. Get started here.
Pick LiteLLM if you need the widest provider coverage and can tolerate 8ms+ overhead. Factor in the March 2026 security incident.
Pick Kong AI Gateway if you already run Kong and want LLM routing added to existing infrastructure. A2A support in v3.14 is promising.
Pick Cloudflare AI Gateway if you want zero-ops and are already on Cloudflare. Limited governance for multi-team setups.
Pick Databricks Unity AI Gateway if you are all-in on Databricks. Strong MCP governance but locks you into the ecosystem.
Trade-offs to Accept
No single best gateway exists. Bifrost's 19 providers cover 95% of production traffic but are fewer than LiteLLM's 100+. LiteLLM's Python runtime is slower but easier to extend. Kong is battle-tested as a proxy but its AI features are catching up. Cloudflare is easiest to set up but gives you the least control. Databricks is powerful within its ecosystem and limiting outside it.
Pick the one that solves your biggest bottleneck first.
Top comments (0)