If you are building agentic workflows where multiple LLM agents need to share tools and context, you have probably hit this wall already: tool definitions bloat your context window, routing logic gets hardcoded, and there is no clean way to control which agent gets access to what.
Model Context Protocol (MCP) solves the standardization problem. But MCP alone does not solve routing, governance, or performance at scale. For that, you need something sitting between your agents and your MCP servers.
Let me walk you through the five gateways that handle MCP tool routing in 2026, what each one does well, and where each one falls short.
TL;DR: Bifrost leads on MCP routing performance (sub-3ms latency, Code Mode for 50%+ token reduction) and is open-source. LiteLLM offers wide provider support with basic MCP proxying. Kong AI Gateway and Azure API Management are enterprise plays. Cloudflare AI Gateway is managed but limited on MCP flexibility. Comparison table and honest trade-offs below.
What Does "Routing Tools and Context" Actually Mean?
When you have multiple LLM agents, each one needs different tools. Your coding agent needs filesystem and git tools. Your research agent needs web search and database access. Your customer support agent needs CRM and ticket system tools.
Without a gateway, you either send every tool definition to every agent (wasting tokens and creating security risks) or you hardcode routing logic in your application layer.
An MCP gateway handles this for you. It sits between your agents and your MCP servers, and gives you request-level control over which tools each agent can see and execute. The good ones also handle failover, budget limits, and observability across all your MCP traffic.
Here is what you should evaluate.
1. Bifrost
GitHub: git.new/bifrost | Docs: getmax.im/bifrostdocs | Website: getmax.im/bifrost-home
Bifrost is an open-source LLM gateway written in Go by the team at Maxim AI. It was built for production workloads, and its MCP implementation is where it really differentiates.
Here is what you get for MCP routing:
- Code Mode: Instead of dumping 100+ tool definitions into the context window, Code Mode has the LLM write TypeScript to orchestrate tools in a sandboxed environment. This cuts token usage by 50%+ and reduces execution latency by 40-50%.
- Agent Mode: Autonomous tool execution with configurable auto-approval. You define which tools run automatically and which need human sign-off.
- Request-level tool filtering: Use HTTP headers to control exactly which MCP servers and tools are available per request. Tool filtering runs at 50-200 nanoseconds per tool.
- Four connection types: InProcess (~0.1ms), STDIO (~1-10ms), HTTP, and SSE. Each optimized for different deployment patterns.
- Governance layer: Budget controls, routing rules, and per-tool rate limiting built in.
- Automatic failover: If your primary provider goes down, requests route to a backup automatically.
- Dual-layer semantic caching: Exact match first, then vector similarity. Repeated tool calls can skip the provider entirely.
- Observability: Full visibility into MCP traffic, cache hit rates, latency, and costs.
Performance numbers: Sub-3ms MCP latency. 11 microseconds of gateway overhead. 5,000 RPS sustained on a single instance. You can verify these yourself using the benchmarking tools.
Quick setup example:
npx -y @maximhq/bifrost
Then filter tools per request:
curl -X POST http://localhost:8080/v1/chat/completions \
-H "Content-Type: application/json" \
-H "x-bf-mcp-include-clients: filesystem,websearch" \
-H "x-bf-mcp-exclude-tools: filesystem/delete_file" \
-d '{
"model": "openai/gpt-4o-mini",
"messages": [
{
"role": "user",
"content": "Search for recent papers on multi-agent routing"
}
]
}'
Each agent gets its own tool set through headers. No gateway reconfiguration needed. Follow the setup guide for the full walkthrough.
2. LiteLLM
LiteLLM is a Python-based LLM proxy that supports 100+ providers. It has added basic MCP proxy support, letting you route tool calls alongside regular LLM requests through a single interface.
What you get:
- MCP tool call proxying through its unified API
- Provider-agnostic tool routing
- Virtual key management for access control
- Open-source and self-hostable
- Wide model coverage
Where it falls short for MCP routing: LiteLLM is written in Python, which means higher latency at the gateway level (roughly 8ms overhead per request vs. Bifrost's 11 microseconds). There is no Code Mode for token optimization, no request-level tool filtering via headers, and no multi-connection-type support. MCP is not its core focus. It is a good LLM proxy that happens to support basic MCP.
If you are already using LiteLLM for multi-provider routing and your MCP needs are simple, it works. If your agents need fine-grained tool control, you will hit limits.
3. Kong AI Gateway
Kong is a battle-tested API gateway that extended into AI territory. Starting with version 3.6, it added LLM support, and its plugin architecture lets you handle MCP routing through request transformation and service mesh integration.
What you get:
- MCP routing through Kong's plugin system
- Rate limiting and authentication via existing Kong plugins
- Request/response transformation for MCP payloads
- Service mesh integration for internal MCP servers
- Enterprise support and documentation
Where it falls short: MCP is handled through plugins, not native architecture. There is no Code Mode, no dynamic tool discovery, and no MCP-specific latency optimizations. Configuration complexity is real. If you are setting up Kong just for MCP routing, you are deploying a full API gateway platform for a narrow use case. The AI features require Kong Gateway Enterprise or Kong Konnect, so the MCP-relevant parts are not open-source.
Best fit: enterprises already running Kong that want to add MCP routing to existing infrastructure.
4. Cloudflare AI Gateway
Cloudflare AI Gateway uses its edge network to proxy and manage AI API calls, including MCP requests. The global edge presence gives you low latency to end users.
What you get:
- MCP request proxying through the edge network
- Built-in caching for tool responses
- Rate limiting per tool or per user
- Analytics dashboard for MCP usage tracking
- Integration with Cloudflare Workers for custom logic
Where it falls short: MCP flexibility is limited. There is no Code Mode for token optimization, no request-level tool filtering granularity, and no self-hosting option. MCP support is secondary to its core AI gateway features. You are also locked into Cloudflare's ecosystem, which may or may not work for your stack.
Best fit: teams already on Cloudflare that want managed MCP proxying with global edge caching.
5. Azure API Management
Azure API Management has added AI gateway capabilities, including the ability to route MCP traffic through its enterprise API management layer. It is tightly integrated with the Azure ecosystem.
What you get:
- MCP routing through Azure's API management policies
- Integration with Azure OpenAI Service and other Azure AI services
- Enterprise-grade authentication and authorization
- Built-in monitoring through Azure Monitor
- Policy-based request transformation
Where it falls short: This is Azure-native. If your stack is not on Azure, the integration overhead is significant. There is no Code Mode, no MCP-specific latency optimizations, and no open-source option. Configuration is policy-based, which is powerful but verbose. MCP support is part of a broader API management platform, not a purpose-built feature.
Best fit: enterprises running on Azure that want MCP routing integrated into their existing API management.
Comparison Table
| Feature | Bifrost | LiteLLM | Kong AI GW | Cloudflare AI GW | Azure APIM |
|---|---|---|---|---|---|
| Open Source | Yes (Go) | Yes (Python) | Partial (Enterprise) | No | No |
| MCP Latency | Sub-3ms | ~8ms+ overhead | Not published | Edge-dependent | Not published |
| Code Mode (token reduction) | Yes (50%+) | No | No | No | No |
| Agent Mode | Yes | No | No | No | No |
| Request-Level Tool Filtering | Yes (headers) | No | Via plugins | Limited | Via policies |
| Connection Types | 4 (InProcess, STDIO, HTTP, SSE) | HTTP | HTTP (plugins) | HTTP | HTTP |
| Governance/Budget Controls | Built-in | Limited | Via plugins | Rate limiting | Via policies |
| Automatic Failover | Built-in | Basic | Via plugins | No | Via policies |
| Semantic Caching | Dual-layer | Redis/Qdrant | Enterprise plugin | Edge caching | No |
| Self-Hostable | Yes | Yes | Yes | No | No |
| Gateway Overhead | 11 microseconds | ~8ms | Varies | Edge-optimized | Varies |
Honest Trade-offs
No gateway is perfect. Here is what you should know before committing.
- Bifrost has a smaller community than LiteLLM or Kong. It is self-hosted only, so you own the infrastructure. If you need a fully managed MCP gateway, this is not it yet.
- LiteLLM gives you the widest provider support, but Python runtime latency adds up at scale. MCP is not its primary focus, so expect basic functionality.
- Kong AI Gateway is enterprise-grade and reliable, but deploying a full API gateway for MCP routing is a heavy lift. The AI plugins need an enterprise license.
- Cloudflare AI Gateway is managed and globally distributed, but you sacrifice MCP flexibility and self-hosting. Limited tool filtering granularity.
- Azure API Management works well if you are all-in on Azure. Outside that ecosystem, the integration cost is hard to justify for MCP routing alone.
Which One Should You Pick?
If MCP tool routing across agents is your primary concern, and you care about latency, token efficiency, and granular control, Bifrost gives you the most complete feature set. Code Mode alone can halve your MCP token costs, and the request-level tool filtering means each agent gets exactly the tools it needs, nothing more.
If you need a managed solution and are already on Cloudflare or Azure, their respective gateways handle basic MCP proxying without adding infrastructure.
If you are already running LiteLLM or Kong and need to add simple MCP routing, their existing support will get you started without swapping your stack.
For most teams building multi-agent systems in 2026, the decision comes down to this: do you need basic MCP proxying, or do you need full MCP orchestration with governance, caching, and token optimization? If it is the latter, start here.
Star the Bifrost repo on GitHub. Read the MCP docs. Check the full documentation to get running in under a minute.
Top comments (0)