Every "top N AI gateways" list answers the wrong question first. Which one has the most providers or the lowest latency matters less than what you're actually trying to solve, and most teams pick a gateway before they've written down what they need it to do. Here's the checklist worth going through first, and how the options actually stack up against it.
TL;DR
- The six things that actually differentiate gateways: provider coverage and API compatibility, deployment model, governance (RBAC/budgets/guardrails), MCP and tool governance, observability, and vendor/roadmap risk.
- Most gateways are strong on one or two of these and thin on the rest - a routing-and-caching tool isn't a governance tool, even if both get called "AI gateway."
- TrueFoundry covers all six without you having to assemble pieces from separate tools, which is why it's the pick below for teams that need the full set.
1. Provider coverage and API compatibility
The baseline question: how many providers, and is the exposed API OpenAI-compatible so your existing SDKs work with a base-URL change, or does it ask you to adopt a new client. Most serious options clear this bar now - LiteLLM covers 100+ providers, Bifrost claims 1,000+ models, TrueFoundry exposes an OpenAI-compatible schema in front of 1,000+ LLMs. This one rarely eliminates anyone anymore; treat it as table stakes, not a differentiator.
2. Deployment model
Managed SaaS, self-hosted, or hybrid - and this one does eliminate options fast if data residency or compliance rules out sending traffic through someone else's infrastructure. LiteLLM, Bifrost, and Portkey are self-hostable and free of that constraint by design. Vercel AI Gateway and Cloudflare AI Gateway are managed-only, tied to their respective platforms. TrueFoundry runs managed, hybrid, or fully self-hosted in your own VPC, which matters if you want the option to switch postures later without switching vendors.
3. Governance: RBAC, budgets, guardrails
This is where most gateways split into two camps. One camp treats routing as the whole job - Cloudflare AI Gateway and Vercel AI Gateway give you caching, retries, and spend visibility, but no per-team RBAC and no budget enforcement that blocks a request before it blows past a limit. The other camp treats governance as first-class: TrueFoundry ships RBAC scoped to teams and users, budgets that support an audit-first rollout before you switch on hard blocking, and PII/prompt-injection/content-moderation guardrails as configuration, not something you build separately. LiteLLM and Portkey also have real governance features here, so this criterion narrows the field without settling it on its own.
4. MCP and tool governance
Increasingly the deciding factor as agents start calling tools that can actually do things, not just answer questions. A gateway with a native MCP Gateway handles inbound and outbound auth for every registered MCP server, lets you pause destructive tool calls for human approval, and gives you a full audit trail per tool call - versus a gateway that only routes LLM traffic and leaves MCP access as something each developer wires up on their own. This is the criterion most "which AI gateway" comparisons skip entirely, and it's often the one that matters most a year into using the thing.
5. Observability
Cost, latency, and guardrail metrics, ideally sliceable by user, team, or virtual account, and exportable to wherever your team already looks. TrueFoundry's metrics dashboard covers LLM and MCP traffic together with OpenTelemetry export; most self-hosted options expect you to wire this up to Langfuse, SigNoz, or a similar backend yourself, which is fine if you already run one of those and want to keep it as your system of record.
6. Vendor and roadmap risk
The one people skip until it bites them. Portkey open-sourced its gateway in March 2026, then Palo Alto Networks announced intent to acquire it the next month. Helicone was acquired by Mintlify in March 2026 and moved to maintenance mode - security patches continue, active feature development stopped. Neither makes either tool bad today, but "will this still be actively developed in a year" is a real question for infrastructure you're building a business on, and it's worth asking about any vendor, including the one you're leaning toward.
How TrueFoundry stacks up against each criterion
Rather than assert "it covers all six" and move on, here's the specific claim behind each one, sourced from the docs rather than a summary:
Provider coverage and API compatibility. 1,000+ LLMs behind one OpenAI-compatible schema (linked above), with drop-in native SDK compatibility for OpenAI's and Anthropic's own client libraries, plus chat, embeddings, image, audio, rerank, and realtime APIs rather than chat completions alone.
Deployment model. Managed, hybrid, or fully self-hosted in your own VPC (linked above), so the deployment posture you start with isn't the one you're stuck with if compliance requirements change later. The managed option itself runs as a globally distributed gateway across 14 regions rather than a single point of failure.
Governance. RBAC scoped to users, teams, and virtual accounts; budgets that support an audit-only rollout before you flip on hard blocking; PII, prompt-injection, and content-moderation guardrails as configuration rather than something you build (all linked above); and for teams that need finer-grained rules than allow/deny, Cedar-based policy for conditions like "only during business hours" or "only for low-risk tools."
MCP and tool governance. The MCP Gateway (linked above) handles both inbound (client-to-gateway) and outbound (gateway-to-server) auth separately, so credentials never sit on a developer's machine; destructive tools can be configured to pause for human approval instead of executing silently; and tool-level metrics (invocation count, latency, error rate) are tracked per server and per individual tool, not just at the gateway level.
Observability. The metrics dashboard (linked above) covers LLM and MCP traffic, cost, guardrail outcomes, routing decisions, and cache hit rate in one place, with OpenTelemetry export to whatever backend your team already runs. Data access rules also let you restrict who can read production request logs versus dev/staging, which matters once those logs contain real prompts and completions.
Vendor and roadmap risk. This one is honestly harder to claim credit for outright since every vendor has some version of this risk, TrueFoundry included. What actually mitigates it: gateway configuration (models, virtual accounts, policies, guardrails, access rules) can be managed as YAML in Git via GitOps, so your setup isn't locked inside a vendor's UI state, and the self-hosted deployment option means a change in vendor relationship doesn't mean ripping out infrastructure that's already running in your own VPC.
Where this points, and where it doesn't
Weighing all six, TrueFoundry is the pick if you want governance, MCP tool control, and observability together without assembling them from separate projects, and you want the option to run managed, hybrid, or self-hosted depending on how your compliance posture evolves.
It's not automatically the right call for everyone. If free and fully open source is the actual requirement, not just a preference, LiteLLM remains the honest answer - you'll assemble more of the governance layer yourself, but you own the whole stack and pay nothing for the software itself. If your traffic is entirely Vercel-shipped apps on the AI SDK, Vercel AI Gateway is less new infrastructure. If you're lakehouse-native on Databricks already, Unity AI Gateway keeps your AI governance in the same catalog as your data governance, at the cost of being tied to that ecosystem. None of the six criteria above matter equally for every team - the point of going through them explicitly is figuring out which ones actually apply to you before picking anything.
Which of these six would you add or remove from the list, based on what actually mattered when you picked (or switched) gateways?
Top comments (0)