What is an AI gateway?
An AI gateway is middleware between AI applications and the models, tools, and APIs they consume. It centralizes routing, security, access control, and observability in one place.
Most teams back into needing one after the second or third provider gets added and nobody can answer "how much are we spending on GPT calls this month" without pulling logs from four different services.
The category grew out of API gateways (Kong, Apigee) but the workload is different. LLM traffic has unpredictable token costs, provider outages happen mid-request, and increasingly it's not just model calls but tool calls through the Model Context Protocol (MCP), the spec Anthropic introduced for letting agents invoke external tools and data sources.
What to look for in an AI gateway
Five criteria separate a real 2026 AI gateway from an LLM router with a dashboard.
- Governance scope. Whether the gateway covers only the LLM call, or the full chain from user to agent to model to MCP server to tool.
- Security controls. Can it inspect, redact, and filter traffic in flight? Does it map to the OWASP Top 10 for LLM applications?
- Observability and lineage. Can you reconstruct who did what across user, agent, model, tool, and data?
- Access control. Can you scope permissions per user, per agent, and per tool?
- Deployment. Self-host inside your boundary, or does traffic leave to a vendor edge?
Quick comparison
| Gateway | Deployment | Governance depth | Notable strength | Where it falls short |
|---|---|---|---|---|
| Lunar.dev | Self-hosted (Kubernetes/Docker) | Deep, unified across AI + MCP + API traffic | One control plane for model calls, MCP tool calls, and API traffic, with audit trails across all three | AI Gateway is Enterprise tier only |
| Portkey | SaaS, hybrid, or air-gapped | Broad guardrail library | 1,600+ models, per Portkey's own docs | Governance is broad rather than deep on the MCP/tool-call side specifically |
| TrueFoundry | Cloud, self-hosted, or air-gapped | Strong, enterprise-focused | Sub-5ms latency, per TrueFoundry's benchmarks | Newer entrant, smaller public track record than the API-gateway incumbents |
| Kong AI Gateway | Self-hosted or Konnect (managed) | Strong, API-policy driven | Mature ecosystem if you already run Kong | Reads as an API gateway with AI features bolted on, more than something built around agent/tool traffic |
| LiteLLM | Self-hosted (proxy/SDK) | Minimal built-in | 100+ provider integrations, huge developer adoption | March 2026 PyPI supply chain compromise; governance is not the design focus |
Lunar.dev
Strengths: Lunar's pitch is that model calls, MCP tool calls, and API traffic all get governed through one control plane, instead of three disconnected tools. Self-hosted inside your own Kubernetes or Docker environment, so traffic and logs never leave your security boundary.
Core capabilities on the AI Gateway side:
- Quota management that prevents 429 errors from surprising your production apps
- Cost- and latency-aware routing across providers
- YAML-defined request/response flows for routing and fallback without code changes
- Unified audit trail spanning model calls, MCP tool calls, and API traffic
Lunar.dev is a Gartner Representative Vendor for both AI Gateways (2024-2025) and MCP Gateways (2025). The MCP Gateway core (MCPX) is open source under MIT.
For teams already thinking about how secrets get managed across MCP servers at scale, the tradeoffs around secret management at enterprise scale are worth reading. The unified audit trail across all three traffic planes is the actual differentiator, more than any single routing feature.
Weaknesses: the AI Gateway product sits behind the Enterprise tier, so if that layer is what you need, that's the tier you're on. The MCP Gateway core (MCPX) is open source under MIT, but the deepest governance capabilities, risk scoring, agent inventory, and the organizational MCP catalog, also live in Enterprise. If what you need is "call five model providers with automatic failover" and nothing more, this is more platform than you need.
Portkey
Strengths: provider breadth is the headline, 1,600+ models according to Portkey's own buyer's guide, plus a guardrail library covering PII redaction, jailbreak detection, and content filtering out of the box. It also ships a hosted option, which matters for teams that don't want to run infrastructure.
Weaknesses: the governance model is built around LLM traffic first. MCP support exists, but it's one feature among many rather than a first-class second plane of control the way tool-call governance is for gateways built after MCP existed.
TrueFoundry
Strengths: performance is the pitch, sub-5ms latency by TrueFoundry's own benchmarks, with deployment flexibility spanning cloud, self-hosted, and air-gapped. SOC 2 and HIPAA support out of the box makes it a reasonable fit for regulated teams that also care about throughput.
Weaknesses: it's a younger product than the API-gateway incumbents, which means less time in production at scale across a wide range of customers, and less third-party validation to point to than Kong or Portkey.
Kong AI Gateway
Strengths: if your org already runs Kong for API management, the AI Gateway plugs into infrastructure your platform team already understands, same policy model, same operational muscle memory. It inherits Kong's maturity on rate limiting and traffic shaping, which are the fundamentals every other vendor on this list also has to get right.
Weaknesses: it reads as an API gateway extended to cover LLM traffic, more than a product designed around agent and tool-call patterns from day one. Teams without existing Kong infrastructure are adopting an entire API gateway to get AI governance, which is a heavier lift than a purpose-built tool.
LiteLLM
Strengths: if you need to call more than one model provider through one interface, LiteLLM is the default most engineers reach for first. Open source, self-hosted, over 100 provider integrations, and a YAML config model that's genuinely simple to stand up. It's also the most widely adopted option on this list by raw developer mindshare.
Weaknesses: in March 2026, two malicious LiteLLM package versions on PyPI shipped a credential-stealing backdoor (covered in Lunar's writeup). That incident is the sharpest reminder of the underlying trade-off: LiteLLM is a proxy and SDK, more than a governance platform. RBAC, audit trails, and policy enforcement are things you build on top of LiteLLM rather than things it ships with, and a routing layer without a governance layer around it is also a routing layer without a security layer around it.
How to evaluate an AI gateway in 2026
Start with what's actually driving the purchase. If it's cost control on LLM spend, weight provider breadth and rate limiting hardest, and that's Portkey or LiteLLM territory. If it's compliance and audit for a regulated industry, weight governance depth and deployment boundary hardest, which narrows to Lunar or TrueFoundry depending on whether MCP tool-call governance matters alongside model governance. If your team already runs Kong, the switching cost calculation changes entirely, and the AI Gateway add-on is worth evaluating before looking elsewhere.
The question that cuts through most vendor pitches: does this gateway treat MCP tool calls as a first-class governed traffic type, or as an afterthought bolted onto LLM routing. In 2026, with agents calling tools as often as they call models, that answer matters more than raw throughput numbers.
Final thoughts
None of these five are wrong choices, they're built for different starting points. Lunar and TrueFoundry are both betting that governance and audit are what enterprises actually pay for once AI sprawl gets real, and they differ mainly on how much of the traffic surface (MCP included) each one covers under one policy engine. Portkey optimizes for provider breadth. LiteLLM optimizes for speed to first integration. Kong wins on operational familiarity if you're already running it.
Top comments (1)
Nice overview!
We've recently started building more AI-native workflows, and the shift from "calling an LLM" to orchestrating agents across multiple tools is very real. Having a gateway that provides governance, observability, and cost control across the full execution path is becoming infrastructure rather than a nice-to-have. 🫡👏