DEV Community: Sahajmeet Kaur

Databricks AI Gateway alternatives: the best options for 2026

Sahajmeet Kaur — Fri, 31 Jul 2026 19:55:37 +0000

Databricks Mosaic AI Gateway is a governance and routing layer that lets you call external LLM providers and Databricks-hosted models through one policy-controlled endpoint, with rate limiting, payload logging, PII guardrails, and usage tracking. It is a solid fit if your models, data, and teams already live inside the lakehouse and Unity Catalog. The trouble starts when they do not. Teams running AI across clouds, outside Databricks, or in their own VPC keep looking for a gateway that is not tied to one platform. That is why teams search for a Databricks AI Gateway alternative.

This guide compares the strongest options and where each fits. We lead with TrueFoundry, which we build, and which we think is the best overall choice for production LLM and agent workloads that need to run anywhere. Details are from public documentation as of mid-2026, so confirm current specifics before committing.

What to look for in a Databricks AI Gateway alternative

Platform independence. A gateway that runs in your own VPC or on-prem, not one anchored to a single data platform.
Broad LLM coverage. One API in front of many providers, with routing, fallbacks, and load balancing.
MCP and agent governance. Tool-level control for agents, not just model-endpoint policies.
First-class observability. Token cost, latency percentiles, and per-tool metrics you can attribute by team or tenant.
Compliance. SOC 2, HIPAA, and GDPR support with SSO, RBAC, and audit logging.

The best Databricks AI Gateway alternatives

1. TrueFoundry: best overall

TrueFoundry is an AI-native gateway that runs in your own infrastructure and is not bound to any one data platform. Where Mosaic AI Gateway assumes the lakehouse, the TrueFoundry AI Gateway is designed to sit in front of whatever models, clouds, and agents you already use.

Where it pulls ahead of Databricks for teams running AI beyond the lakehouse:

1,000+ LLMs through one OpenAI-compatible API. Switch models by changing the model name, same URL and credentials, across providers and clouds.
Runs anywhere. Full VPC, on-prem, air-gapped, or hybrid deployment with no request data leaving your domain, and no dependency on a single platform.
Native MCP gateway. The MCP Gateway gives agents governed, tool-level access with OAuth, pre and post-tool guardrails, and a full audit trail.
Deep observability. A built-in metrics dashboard tracks token cost, latency percentiles (TTFT, ITL, P99), guardrail outcomes, and per-tool MCP metrics, and it is OpenTelemetry-compliant.
Model deployment included. Serve any model on vLLM, TGI, or Triton in the same control plane as the gateway.
Performance and compliance. Roughly 3 to 4 ms of overhead at 350+ RPS on a single vCPU, with SOC 2, HIPAA, and GDPR support, SSO, RBAC, and SCIM.

Pros: platform-independent, self-hosted in your VPC, MCP and agent governance, model deployment plus gateway in one place.
Cons: if you are all-in on Databricks and never leave it, the native gateway is closer at hand.
Best for: teams running LLMs and agents across clouds or outside Databricks that want AI-native routing and governance in their own infrastructure.
Pricing: enterprise pricing based on deployment and scale.

Explore the TrueFoundry AI Gateway or book a demo.

2. LiteLLM

LiteLLM is an open-source proxy and Python SDK offering one OpenAI-compatible endpoint for 100+ providers and 2,500+ models, with routing, load balancing, spend tracking, and virtual keys. Most of it is MIT-licensed, with SSO and RBAC in a commercial tier. It is not tied to any data platform, which is often the whole point for teams leaving a lakehouse-bound gateway.

Best for: developers who want the widest provider coverage and a self-hosted proxy with no per-request fee.
Pricing: open source; commercial tier for enterprise governance.

3. Portkey

Portkey is an observability-first AI gateway that routes to a large model catalog and ships 50+ guardrails plus an MCP gateway. The open-source gateway is Apache 2.0; hosted plans start at $49 per month and meter recorded logs.

Best for: teams that want guardrails and request-level observability shipped as a product.
Pricing: free open source, $49/mo hosted, custom enterprise.

4. Kong AI Gateway

Kong added open-source AI plugins to its Nginx-based API gateway, so LLM traffic inherits its large plugin ecosystem for auth, rate limiting, and traffic control. Good if Kong is already your API layer.

Best for: platform teams already standardized on Kong Gateway.
Pricing: AI plugins open source; Kong platform tiers for enterprise.

5. Cloudflare AI Gateway: best for edge caching

Cloudflare AI Gateway is a lightweight edge layer focused on caching, rate limiting, and analytics in front of model providers. Easy to adopt on Cloudflare, though lighter on MCP and agent governance.

Best for: teams on Cloudflare wanting a simple caching and analytics layer.
Pricing: usage-based, with a free tier.

Databricks AI Gateway vs the alternatives: comparison

Capability	TrueFoundry	Databricks (Mosaic)	LiteLLM	Portkey	Kong
Platform independence	Runs anywhere	Tied to lakehouse	Yes	Yes	Yes
LLM coverage	1,000+ via unified API	External + Databricks models	100+ providers	1,600+	Multiple via AI Proxy
MCP / agent governance	Native, tool-level	Limited	Limited	MCP gateway	Not a native focus
VPC / on-prem	Yes, no data egress	Within Databricks	Self-host	Hosted or self-host	Self-host
Model deployment	Included (vLLM, TGI, Triton)	Databricks serving	No	No	No
Observability	Built-in, per-tool	Inference tables, system tables	Via integrations	Observability-first	Via Kong plugins
Compliance	SOC 2, HIPAA, GDPR	Databricks enterprise	Commercial tier	Enterprise tier	Kong enterprise

Conclusion

Mosaic AI Gateway is a strong governance layer when everything lives in Databricks. Once AI has to run across clouds, outside the lakehouse, or in your own VPC, that coupling becomes the constraint. Among the Databricks AI Gateway alternatives, TrueFoundry is the best fit for production LLM and agent workloads because it is AI-native, platform-independent, and governs models and tools in one place.

If you are running AI beyond the lakehouse, book a demo or explore the TrueFoundry AI Gateway.

Apigee alternatives: the best AI gateway options for 2026

Sahajmeet Kaur — Fri, 31 Jul 2026 19:51:47 +0000

Apigee is a capable API management platform, and Google has added AI gateway features on top of it: model routing, token-based rate limiting, content moderation, and analytics. But Apigee was built to manage REST APIs, and teams shipping LLMs and agents keep running into the same gap. They want something AI-native, with first-class support for many model providers, MCP-based tool governance, and deployment inside their own VPC. That is why teams look for an Apigee alternative.

This guide compares the strongest options and explains where each one fits. We start with TrueFoundry, which we build, and which we think is the best overall choice for production LLM and agent workloads. Everything here is drawn from public documentation as of mid-2026, so verify current details before you commit.

What to look for in an Apigee alternative

AI-native routing. One API in front of many LLM providers, with fallbacks and load balancing designed for models, not just HTTP endpoints.
MCP and agent governance. Tool-level access control for agents, not only request-level API policies.
Deployment control. The ability to run in your own VPC or on-prem so prompts and data never leave your domain.
Observability built for LLMs. Token cost, latency percentiles, and tool-level metrics, not generic API dashboards.
Compliance. SOC 2, HIPAA, and GDPR support with SSO, RBAC, and audit logging.

The best Apigee alternatives

1. TrueFoundry: best overall

TrueFoundry is an AI-native gateway built for exactly the workloads Apigee retrofits for. Instead of adding LLM features to an API manager, the AI Gateway treats models, agents, and MCP tools as first-class citizens, and it runs in your own infrastructure.

Where it pulls ahead of Apigee for AI teams:

1,000+ LLMs through one OpenAI-compatible API. Switch models by changing the model name in the request, same URL and credentials.
Full VPC, on-prem, air-gapped, or hybrid deployment. No request data leaves your domain, which is the reason regulated teams pick it over managed API platforms.
Native MCP gateway. The MCP Gateway gives agents governed, tool-level access with OAuth, pre and post-tool guardrails, and a full audit trail. Apigee governs APIs; TrueFoundry governs agent tool calls.
Deep LLM observability. A built-in metrics dashboard tracks token cost, latency percentiles (TTFT, ITL, P99), and per-tool MCP metrics, and it is OpenTelemetry-compliant.
Performance that stays out of the hot path. Roughly 3 to 4 ms of overhead at 350+ RPS on a single vCPU.
Enterprise compliance. SOC 2, HIPAA, and GDPR support with SSO, RBAC, SCIM, and immutable audit logs.

Pros: AI-native, self-hosted in your VPC, MCP and agent governance, model deployment and gateway in one control plane.
Cons: More platform than a team needs for a simple side project.
Best for: enterprises running LLMs and agents in production that want AI-native routing and governance in their own infrastructure.
Pricing: enterprise pricing based on deployment and scale.

Explore the TrueFoundry AI Gateway or book a demo.

2. LiteLLM

LiteLLM is an open-source proxy and Python SDK that puts one OpenAI-compatible endpoint in front of 100+ providers and 2,500+ models. It is the fastest way to stop writing provider-specific code, and the proxy adds routing, load balancing, spend tracking, and virtual keys. Most of it is MIT-licensed, with SSO and RBAC in a commercial tier.

Best for: developers who want maximum provider coverage and a self-hosted proxy with no per-request fee.
Pricing: open source; commercial tier for enterprise governance.

3. Portkey

Portkey is an AI gateway that leads with observability and guardrails. It routes to a large model catalog, ships 50+ guardrails, and includes an MCP gateway. The open-source gateway is Apache 2.0; hosted plans start at $49 per month and meter recorded logs.

Best for: teams that want observability and guardrails as a product rather than a build-it-yourself project.
Pricing: free open source, $49/mo hosted, custom enterprise.

4. Kong AI Gateway

Kong added open-source AI plugins to its Nginx-based API gateway, so LLM calls become just another Kong service and inherit its 1,000+ plugins for auth, rate limiting, and traffic control. This is a natural fit if Kong is already your API layer, less so if you are starting fresh for AI.

Best for: platform teams already standardized on Kong Gateway.
Pricing: AI plugins are open source; Kong platform tiers for enterprise.

5. Cloudflare AI Gateway

Cloudflare AI Gateway sits at the edge and focuses on caching, rate limiting, and analytics in front of model providers. It is easy to adopt if you already run on Cloudflare and want a lightweight layer, though it is lighter on agent and MCP governance than a full platform.

Best for: teams on Cloudflare wanting a simple caching and analytics layer.
Pricing: usage-based, with a free tier.

Apigee vs the alternatives: comparison

Capability	TrueFoundry	Apigee	LiteLLM	Portkey	Kong
AI-native design	Yes	API manager + AI features	Yes	Yes	AI plugins on API gateway
LLM coverage	1,000+ via unified API	Multiple via routing	100+ providers	1,600+	Multiple via AI Proxy
MCP / agent governance	Native, tool-level	Limited	Limited	MCP gateway	Not a native focus
VPC / on-prem	Yes, no data egress	Managed (Google Cloud)	Self-host	Hosted or self-host	Self-host
LLM observability	Built-in, per-tool	API-style analytics	Via integrations	Observability-first	Via Kong plugins
Compliance	SOC 2, HIPAA, GDPR	Enterprise (GCP)	Commercial tier	Enterprise tier	Kong enterprise

Agent observability: How to trace prompts, tools, and models in production?

Sahajmeet Kaur — Fri, 31 Jul 2026 19:49:10 +0000

TL;DR: Agent observability is the practice of tracing everything a multi-step AI agent does: the prompts it sends, the tools it calls, the models it routes to, and what each step costs and how long it takes. A single agent request can fan out into dozens of model and tool calls, so per-request tracing is the only way to debug failures and control spend. TrueFoundry's AI Gateway captures this across LLM and MCP traffic in one dashboard.

When an agent misbehaves in production, the hard part is not knowing that it failed. It is knowing where. A single user request can trigger several model calls, a handful of tool invocations through MCP, a routing decision or two, and a guardrail check, all before an answer comes back. Without agent observability, all you see is a slow or wrong response and no trail explaining it.

This guide covers what agent observability actually means, what you need to instrument, and how TrueFoundry's AI Gateway captures it without you wiring up a tracing stack by hand.

What is agent observability?

Agent observability is the ability to see, measure, and debug the full path of an agent request as it moves through models, tools, and policies. Classic application monitoring tells you a service returned a 500. Agent observability tells you that the agent called the search_tickets tool, got a timeout, retried on a fallback model, hit a rate limit, and finally answered from a degraded context.

The reason it needs its own name is that agents are not single API calls. They are loops. The model decides what to do, calls a tool, reads the result, and decides again. Each hop is a place where latency, cost, and correctness can drift, and each hop needs to be recorded.

Agent observability matters most when:

Your agents call tools through MCP servers and you need to know which tool is slow or failing.
You route across multiple models or providers and want to see where traffic actually goes.
You are trying to attribute cost to a team, application, or tenant.
You run guardrails and need proof of what was blocked, flagged, or mutated.

What to instrument in an agent

Good agent observability comes down to capturing the right signals at every hop. The ones that matter in practice:

Requests and throughput. Requests per second across LLM calls, MCP tool calls, and agent responses, so you can see load and spot spikes.
Latency, broken down. Not just an average. End-to-end request latency plus streaming-specific numbers like time to first token (TTFT), inter-token latency (ITL), and time per output token (TPOT), at P50, P90, and P99.
Failures by type. Failure rates split by HTTP status and error type, per model and per tool, so a flaky MCP server does not hide inside an aggregate.
Cost and tokens. Input and output tokens and dollar cost over time, attributable by model, user, team, or custom metadata.
Tool-level detail. Which MCP tools get called most, which are slow, and which error out. Server-level averages are not enough when one tool is the problem.
Policy outcomes. Guardrail results (allowed, blocked, flagged, mutated) and routing or rate-limit decisions, so governance is auditable.

If you only track top-line latency and cost, you will see that something is wrong but not why. The tool and model breakdowns are where debugging actually happens.

How TrueFoundry handles agent observability

TrueFoundry's AI Gateway is OpenTelemetry-compliant and traces every request from prompt to tool and model execution, so it plugs into Grafana, Datadog, or Prometheus if you already run them. It also ships a built-in Metrics Dashboard that covers the signals above across both LLM and MCP traffic. A few things it gives you without custom instrumentation:

One view over LLM and MCP. The overview tab shows total cost, total LLM calls, and total MCP calls together, with incoming requests broken down by endpoint (/chat/completions, /mcp-server, /agent/responses, and more).
Model metrics with real latency percentiles. Requests per second, failure rate by error type, and latency including TTFT, ITL, and TPOT at P50 through P99, pivotable by model, user, team, virtual account, or custom metadata.
Tool-level MCP metrics. A dedicated MCP metrics view drills from server down to individual tool, showing per-tool request rate, latency, and failure breakdown, plus a method-calls breakdown (tools/list, tools/call, and so on).
Guardrail and routing visibility. Guardrail evaluations by outcome, and routing metrics showing which rules fire, how often rate and budget limits are hit, and where traffic gets load-balanced.
Export and API access. Download aggregated metrics as CSV or pull them programmatically, including a dedicated agent-metrics endpoint.

On performance, the gateway is built to stay out of the hot path: it adds roughly 3 to 4 ms of overhead and handles 350+ RPS on a single vCPU, while unifying access to 1,000+ LLMs through one OpenAI-compatible API. You get the tracing without the gateway becoming the thing you have to debug.

Best practices for agent observability

Trace per request, not per service. Tie every model and tool call back to the originating agent request so you can reconstruct the full path.
Attribute cost early. Send team, tenant, or feature as metadata from day one, so chargeback and budget questions have an answer later.
Watch tool-level failure rates. A single slow or erroring MCP tool often explains a whole class of agent failures.
Alert on percentiles, not averages. P99 latency is what your users feel when an agent stalls mid-loop.
Keep guardrail outcomes auditable. Log what was blocked or mutated, because "the agent said something it should not have" is a question you will eventually be asked.

What is a virtual MCP server? Curating tools across MCP servers

Sahajmeet Kaur — Fri, 31 Jul 2026 19:43:11 +0000

TL;DR: A virtual MCP server combines tools from several real MCP servers into one curated server your app connects to. You pick only the safe tools, skip the destructive ones, and expose them as a single remote MCP server with no extra deployment. It is how you give an agent GitHub and Slack access without handing it delete_project.

As agents start calling live tools, the access question gets sharp fast. The Model Context Protocol (MCP) is great at connecting an AI agent to a data source or a set of tools, but a raw MCP server is usually all or nothing. You connect an agent to your GitHub MCP server and it can now see every tool that server exposes, including the ones that delete branches and close pull requests. A virtual MCP server fixes that by letting you hand an agent a curated subset of tools drawn from one or more underlying servers, without standing up anything new.

This guide covers what a virtual MCP server is, how it works, how tool naming is handled, and where it fits inside a broader MCP gateway.

What is a virtual MCP server?

A virtual MCP server combines tools from multiple MCP servers into a single, curated MCP server that your application connects to. Instead of pointing an agent at each backend server directly, you assemble the exact set of tools you want it to have and publish that set as one server.

The example TrueFoundry uses in its docs makes it concrete. Say you have MCP servers for GitHub and Slack registered on the gateway. A team building an agent needs access to both, but you do not want to expose tools like delete_project or delete_pr. With a virtual MCP server, you create a new server that takes only the safe subset from GitHub and Slack. That new server is reachable like any other remote MCP server, and it does not require a deployment. The gateway manages it.

The practical win is least privilege for agents. You decide what an agent can touch at the tool level, not the server level.

A virtual MCP server works well when:

An agent needs tools from more than one backend, but not all of them.
You want to strip destructive or high-risk tools before an agent ever sees them.
Different teams or applications need different slices of the same underlying servers.
You want to scope tool access without deploying or maintaining a bespoke proxy server.

How a virtual MCP server works

The flow is straightforward once the underlying servers are registered on your MCP gateway.

Register your real MCP servers. Connect the backends you want to draw from, for example GitHub and Slack, to the gateway.
Select the safe tools. Pick the subset of tools from those servers that the agent should have. Leave out anything destructive or out of scope.
Publish the virtual server. The gateway exposes your selected tools as a single virtual MCP server. There is no separate deployment to run or scale, since the gateway hosts it.
Connect your app. Point your agent or MCP client at the virtual server the same way you would any remote MCP server, over streamable HTTP.

From the agent's side, it just sees one clean server with a sensible set of tools. All the curation happens at the gateway.

How tool names are handled

When you pull tools from several servers into one virtual server, name collisions are a real risk. Two backends can both expose a create_issue, for instance. TrueFoundry handles this by keeping each tool's original name and appending a short random suffix, so create_issue becomes something like create_issue_a1b2c3. The suffix keeps every tool unique within the virtual server.

Worth noting why it is done this way rather than prefixing with the server name (like github-create_issue). The MCP spec recommends keeping tool names within 64 characters, and a long server name can eat most of that budget. Since the tool name is the strongest signal an LLM uses when deciding which tool to call, preserving it in full while adding a short suffix keeps names both meaningful and collision-free.

Virtual MCP server vs a raw MCP server

Aspect	Raw MCP server	Virtual MCP server
Tool scope	All tools the server exposes	Curated subset you choose
Sources	One backend	One or more backends combined
Access control	Server-level, all or nothing	Tool-level, least privilege
Deployment	You run and manage the server	Managed by the gateway, no deployment
Best fit	Full trusted access	Scoped access per team, app, or workflow

The two are not competitors. A virtual MCP server sits on top of your real servers and gives you a governance layer over them.

Where virtual MCP servers fit in the MCP gateway

A virtual MCP server is one feature of a larger job: giving enterprise agents governed access to tools. On its own, MCP adoption tends to sprawl. Every developer wires up their own server connections in Cursor, VS Code, or Claude Code, credentials scatter across machines, and security teams get no visibility into which tools are being called or by whom.

An MCP gateway centralizes that. The TrueFoundry MCP Gateway gives AI agents a single point of access to many MCP servers, with a few capabilities that make virtual servers more useful in production:

Standard OAuth flows. Agents and developers authenticate through standard OAuth 2LO and 3LO flows for enterprise MCP servers, instead of ad-hoc keys per tool.
Governed, tool-level access. Access control is enforced centrally, which is what makes the curated tool subset in a virtual server meaningful rather than cosmetic.
Pre-tool and post-tool guardrails. You can run checks before and after a tool call to enforce policy, so a curated tool set can also be a guarded one.
Full audit trail and metrics. Every tool call is visible, with request rates, latency, failures, and usage patterns per server and per tool.
More ways to build servers. Beyond curating existing servers, the gateway can turn an OpenAPI spec into MCP tools and run CLI-style stdio servers as managed, hosted endpoints.

If you need to pass custom credentials through to the backends behind a virtual server, the gateway supports an x-tfy-mcp-headers header that forwards per-server headers to the underlying MCP servers. One current limitation to keep in mind: virtual MCP servers today support listing and calling tools, so plan around that if you rely on other MCP features.

When to use a virtual MCP server

Reach for a virtual MCP server when tool access needs to be scoped rather than wholesale. A few common cases:

Least-privilege agents. Give an agent exactly the tools its task needs and nothing that can cause damage.
Per-team or per-app tool sets. Publish different curated servers for different consumers off the same backends.
Multi-tenant tool catalogs. Offer curated, discoverable tool sets to different tenants without exposing raw servers.

LLM Gateways Explained: What They Do and Why You Need One

Sahajmeet Kaur — Tue, 28 Jul 2026 10:58:25 +0000

Without a gateway, your application code holds a provider SDK directly and calls it. Add a second provider and you're holding two SDKs with two different request shapes, with an if/else deciding which one to use. An LLM gateway sits in front of every provider instead, exposes one API, and translates your request into whatever the actual provider expects on the other side - this is what that looks like using TrueFoundry's gateway specifically.

TL;DR

An LLM gateway is one endpoint in front of every model provider, so your code changes a model name string instead of a client library when you switch or add one.
The three concrete reasons teams add one - provider outages, rate limits, and SDK lock-in - are all things TrueFoundry's gateway handles as routing features (automatic failover, per-user/team rate limits, one OpenAI-compatible schema across 1,000+ models), not things you build separately.
Once every call passes through one place, that place is also where RBAC, budgets, guardrails, and MCP governance naturally live - not because the gateway is "for" governance, but because it's the only component that sees every request.

What the gateway actually does

TrueFoundry's AI Gateway exposes an OpenAI-compatible schema in front of 1,000+ models across providers, with native SDK compatibility for OpenAI's and Anthropic's own client libraries - meaning your existing code often works with just a base-URL change, not a rewrite. Your application never holds a provider SDK directly; it points at the gateway and changes a model name string to switch what's behind it.

Reason one: providers go down, and you don't want to go down with them

OpenAI and Anthropic both had multiple incidents on their public status pages between February and May of 2025 - not exotic, just the reality of depending on someone else's infrastructure. Load balancing and fallback routes across multiple targets by weight, latency, or priority with automatic retries, so a provider outage means failing over to a healthy target instead of your application going down with it. The same mechanism supports canary rollouts - routing a small percentage of traffic to a new model and watching it before trusting it with everything.

Reason two: rate limits hit hardest exactly when you can't afford them

Azure OpenAI, like most providers, enforces tokens-per-minute and requests-per-minute quotas per model per region. Under a traffic spike, or a bug that puts an agent into a retry loop, you hit that ceiling and start getting 429s. Rate limiting applied per user, team, or application means a gateway that understands the limit can route overflow elsewhere instead of just failing the request - and it doubles as a cost control, capping what a single bug or a single team can run up.

Reason three: SDK lock-in makes switching models expensive later

If your codebase calls a specific provider's SDK in forty places, swapping in a cheaper or better model six months from now means touching forty places. With the gateway's unified schema, that's a config change - a model name string - not an application rewrite.

What you get once everything routes through one place

This is the part that surprises people who adopted a gateway purely for routing: RBAC scoped to teams and users, budgets with an audit-first rollout before you switch on hard blocking, PII/prompt-injection/content-moderation guardrails, and a native MCP Gateway for governing tool calls - all sitting on the same request path as the routing, because that path is the only place that sees every call regardless of which team or model made it. None of it is a separate system you have to keep synchronized with the gateway; it's configuration on the same one.

Deployment is the other piece worth knowing up front: managed, hybrid, or fully self-hosted in your own VPC, so "we can't send traffic through someone else's infrastructure" doesn't rule this out the way it would with a purely hosted option.

Where I'd push back on this

If you're calling one provider, for one use case, at low volume, none of this is solving a problem you have yet. Add a gateway once you're multi-provider, multi-team, or once an outage or a rate-limit wall has actually cost you something - not before, and not because RBAC and budgets sound like good practice in the abstract. This breakdown of gateway vs. proxy vs. router terminology is a good next read if the terms are still blurring together, and there's a solid walkthrough of wiring automatic provider fallback into an agent if you want to see the failover pattern in code first.

What pushed you to add a gateway, if you have one - an outage, a rate-limit wall, or something else entirely?

LLM Cost Controls That Actually Work: Budgets, Rate Limits, and Per-Team Attribution

Sahajmeet Kaur — Sat, 25 Jul 2026 17:28:31 +0000

The first sign something's wrong with LLM spend is usually a bill, three weeks after the damage is done. By then the agent that got stuck in a retry loop, or the prompt that kept growing every turn, has already run a thousand more times. Real cost control means seeing the spend as it happens and having a lever to stop it before the invoice, not after.

TL;DR

Cost visibility has to be real-time and per-model/per-team, not a monthly invoice - by the time a bill shows an anomaly, it's already happened repeatedly.
Budgets should roll out in audit mode first (watch what would have been blocked) before you flip on hard enforcement, and rate limiting is a cost lever in its own right, not just a reliability one.
None of this works without attribution - metadata tagging (team, cost center) has to be enforced on every request, or your cost dashboard just shows one big number with no way to trace it back to a cause.

What TrueFoundry AI Gateway actually is

TrueFoundry's AI Gateway, a single OpenAI-compatible endpoint in front of 1,000+ models across providers - your application code points at one URL and changes a model name string to switch providers, instead of holding a different client per vendor. It runs managed, hybrid, or fully self-hosted in your own VPC, and cost controls aren't a separate add-on bolted onto that routing layer - they sit alongside the same RBAC, guardrails, and MCP governance that also run through the gateway, using the same request metadata and the same policy engine. That's the reason budgets, rate limits, and attribution can share one metadata model instead of being three separate systems you have to keep in sync.

See the spend as it happens, not after the invoice

A metrics dashboard that tracks cost per model and per provider in real time is the actual starting point - not because dashboards are inherently useful, but because the alternative (a monthly invoice) tells you about a problem three to four weeks after it started, by which point whatever caused it has already run at that same rate the whole time. If a request is slow because it's generating 4,000 output tokens per call, that's also the request driving your bill, and the fix (lower max_tokens, a smaller model for that specific path) is the same fix either way.

Budgets that don't break things on day one

Budget limiting supports a hard block once a spend cap is hit, but the useful part is the rollout path: deploy a budget rule in audit mode first, watch real traffic against it for a full budget period, and only switch on enforcement once you trust the number. That matters because guessing a cap wrong in either direction is expensive - too low and you block legitimate traffic, too high and the rule does nothing. Audit mode answers "how often would real traffic actually have exceeded this" before you commit to blocking anything.

Rate limits are a cost control, not just a reliability one

Rate limiting usually gets framed as a reliability feature, but it's just as much a cost one: capping cost per developer or team means a bug - an agent in an infinite loop, a script that fires more requests than intended - has a ceiling instead of running up an unbounded bill. It's also how you'd tier cost across customers if you're serving multiple accounts with different usage allowances on the same underlying models, or burst from an on-prem GPU to a per-token cloud API without letting that burst traffic overwhelm the on-prem deployment when it's supposed to be an occasional overflow valve, not the default path.

Attribution: the part that makes the dashboard mean something

Cost tracking without attribution just tells you a total. Automatic metadata injection tags every request from a virtual account with metadata like team or cost_center without any client-side code changes, and a metadata validation guardrail can require those keys be present - and match a specific format - on every request before it reaches a model, rejecting anything that doesn't carry proper attribution instead of letting it through as an anonymous cost:

name: cost-attribution-enforcement
type: gateway-guardrails-config
rules:
  - id: require-cost-attribution
    when: {}
    llm_input_guardrails:
      - <guardrail-group>/cost-attribution
    llm_output_guardrails: []

Combined with tagged virtual accounts, this means cost reports can actually be broken down by team and cost center automatically, rather than reconstructed after the fact from whoever remembers which application made which calls.

Two more levers worth using: caching and routing

Semantic caching cuts cost and latency on requests that are semantically similar to ones already served, which matters more than it sounds like for anything with repeated or templated queries. And load balancing across models can route by weight or by cost as one of the criteria, not just latency or availability, so a canary rollout of a cheaper model can absorb a percentage of traffic and prove itself before taking over the rest.

Where this still requires a human decision

None of this decides what your budgets should be, or which teams should get which limits - it gives you the mechanism to enforce whatever number you land on, and the audit-mode step to test that number before it's load-bearing. The actual limits are still a judgment call, usually made after watching audit mode for a full cycle rather than guessed upfront.

How is your team attributing LLM cost right now - tagged at the gateway automatically, or reconstructed after the fact from application logs? Curious how common real-time, per-team attribution actually is versus a shared bill nobody's broken down.

MCP Access Control: RBAC, Per-Tool Restrictions, and Approval Gates for Your MCP Servers

Sahajmeet Kaur — Sat, 25 Jul 2026 11:28:57 +0000

An MCP server that anyone with the shared API key can call is fine right up until that server can also delete records, send messages, or push code. Access control for MCP isn't one setting - it's at least four separate questions: who can connect at all, which servers they can reach, which specific tools within those servers, and whether a given tool call needs a human to sign off before it runs.

TL;DR

MCP access control has to answer four questions separately: who can connect (inbound auth), which servers and tools they can reach (RBAC), whether specific actions need conditional rules beyond allow/deny (policy), and whether destructive tool calls pause for human approval.
TrueFoundry's MCP Gateway covers all four as configuration: RBAC scoped to teams and users, Cedar-based conditional policy, and approval gates on destructive tools, with every call traced.
If you're a single developer connecting to a single MCP server, none of this is necessary yet - a plain transport proxy is the right amount of tooling until more than one person or more than one server is involved.

TrueFoundry MCP Gateway

TrueFoundry’s approach is simple: if organizations are already managing AI infrastructure for LLMs, there is little value in fragmenting operations across separate systems for MCP tools. Instead, TrueFoundry unifies LLM infrastructure and MCP management into a single control plane with shared security, observability, governance, and performance characteristics. This centralized approach simplifies AI operations while giving engineering teams a consolidated platform for monitoring, deployment, and cost management.

One of the platform’s standout advantages is its performance-focused architecture. TrueFoundry achieves sub-3ms latency under load by handling authentication and rate limiting in-memory instead of relying on database queries. For AI agents making hundreds of MCP tool calls per interaction, this reduction in latency compounds into significantly faster and more responsive systems.

The platform also emphasizes enterprise-grade operational simplicity. Teams can deploy containerized MCP servers, integrate them directly with the AI Gateway, and manage authentication, access control, custom configurations, guardrails, fallback mechanisms, load balancing, and rate limits from a unified interface. Interactive playgrounds further accelerate development by generating production-ready code snippets across multiple languages, helping teams move quickly from experimentation to deployment.

Most importantly, TrueFoundry delivers unified observability and billing. Organizations already tracking LLM performance and costs gain visibility into MCP tool usage and infrastructure metrics from the same dashboard, preventing operational blind spots and unexpected budget overruns.

Key Features of TrueFoundry
Unified infrastructure for both LLMs and MCP tools through a single control plane
Sub-3ms latency under load with in-memory authentication and rate limiting
MCP Server Groups for logical isolation across teams and environments
Containerized MCP server deployment with centralized orchestration
Integrated AI Gateway with authentication and access control
Custom configurations, guardrails, fallback mechanisms, and load balancing
Built-in rate limiting and cloud-based model deployment support
Interactive playground with production-ready code generation in multiple languages
Unified observability, monitoring, and billing across AI workloads and MCP tool usage
Integrations with platforms such as n8n, Slack, and Claude Code

The four questions access control actually has to answer

1. Who can connect to the gateway at all. Inbound authentication supports a TrueFoundry API key, an identity provider token, or TrueFoundry OAuth, and the gateway validates whichever one shows up before anything else happens.

2. Which servers and tools a given user or team can reach. This is RBAC in the ordinary sense: grant a team access to a server, and separately decide who can manage that server's configuration versus who can only call it.

3. Whether a specific action needs a rule beyond simple allow or deny. Plain RBAC answers "can this user reach this server." It doesn't answer "can this user's support-copilot agent write to Jira, given that the same user's engineering agent is allowed to." For that, TrueFoundry uses Cedar policy, evaluated as a principal/action/resource/context tuple on every request, default-deny:

// The support copilot may only READ Jira, even when acting for any user.
permit (
  principal == AgentIdentity::"support-copilot",
  action == Action::"mcp:callTool",
  resource == Tool::"jira/issues.read"
);

// Block destructive tools for any agent more than two hops deep in a chain.
forbid (
  principal,
  action == Action::"mcp:callTool",
  resource in ToolGroup::"destructive"
)
when { context.actor_chain.length > 2 };

Whether a destructive action needs a human to actually confirm it. Tools marked destructive can be configured to pause and wait for user confirmation instead of executing the moment they're called - the difference between an agent that can propose deleting a record and one that can just delete it.

4. Watching what actually gets called

Access control without visibility into what's happening is half the job. The MCP Metrics dashboard has a server-level view (request rate, latency percentiles, failure rate by error type, which JSON-RPC methods make up the traffic) and a tool-level view for drilling into individual tools across every server:

Both views can be sliced by user, team, or virtual account, which is what turns "a tool failed" into "this specific team's agent is failing this specific tool at this rate."

Where guardrails fit in alongside access control

Access control decides whether a call is allowed to happen at all. Guardrails inspect the content of a call that's already been permitted - catching a tool argument or response that looks like a secret, a prompt injection payload, or a policy violation the RBAC layer was never meant to catch. The same guardrail metrics dashboard tracks evaluated requests and blocked/mutated rates for these checks:

What's the access-control gap your team has actually hit first - RBAC across teams, a destructive tool that ran without anyone confirming it, or just not knowing which agents can reach which servers at all?

MintMCP Alternative: Why TrueFoundry is a better choice for MCP Governance

Sahajmeet Kaur — Fri, 24 Jul 2026 12:17:00 +0000

TL;DR

MintMCP is a purpose-built MCP gateway: RBAC, Virtual MCP Bundles, Agent Bundles with M2M auth, and an Agent Monitor that extends beyond MCP to watch local non-MCP agent activity like bash commands and file operations.
TrueFoundry's case for being the better fit: the same MCP governance (RBAC, tool-level policy via Cedar, approval gates, per-tool metrics) sits under one control plane alongside LLM gateway routing and Agent Registry governance, rather than MCP being the whole product - and deployment (managed, hybrid, or fully self-hosted) is first-class rather than "available on request." TrueFoundry has an compliance attestation SOC 2 Type II / HIPAA-BAA

What MintMCP actually does well

MintMCP's own feature set is built specifically for MCP, not adapted from a broader platform. SSO and SCIM-driven RBAC, OAuth brokering for stdio and hosted servers, Virtual MCP Bundles that scope a curated tool set per role or team, Agent Bundles with machine-to-machine auth, tool-level allowlisting, and a tool-update policy that decides whether new upstream tools get auto-enabled or need admin approval. It also runs hosted connectors so teams don't manage connector runtimes themselves, and its Agent Monitor tracks local agent activity - bash commands, file reads and writes, prompt submissions - that happens outside of MCP tool calls entirely, which is a genuinely broader scope than pure MCP governance.

Where TrueFoundry's case is stronger

MCP governance isn't the whole product, it's one part of one. TrueFoundry's MCP Gateway sits alongside the same control plane's LLM gateway (RBAC, budgets, guardrails on model traffic) and Agent Registry (agent identity, versioning, audit). If your actual problem is governing AI usage across models, agents, and MCP tools together - not just the MCP layer in isolation - that's one system to reason about instead of MCP governance plus a separate LLM gateway plus a separate agent registry.

Deployment flexibility is a first-class option, not a special request. MintMCP is managed SaaS-first, with VPC or self-hosted "on request." TrueFoundry runs managed, hybrid, or fully self-hosted in your own VPC as an equally supported path, which matters if your compliance posture might shift and you don't want that to mean switching vendors.
Developer integration is designed for SDK-first teams: full fastmcp (Python) and @modelcontextprotocol/sdk (TypeScript) code snippets with async/await, proper headers, and live token injection ship directly in the platform UI. Most popular IDE clients are supported out of the box.

Fine-grained, conditional policy, not just allow-lists. MintMCP's tool-level allowlisting and rule-based policy covers a lot of ground. TrueFoundry's Cedar-based policy goes a step further into genuinely conditional rules - "this agent may only read Jira," "block destructive tools for any agent more than two hops deep in a chain" - evaluated as a principal/action/resource/context tuple on every request, default-deny.
Auth is another area of clear separation: TrueFoundry offers a two-layer, highly flexible model — authenticate the agent once at the gateway, then act with each end-user's own upstream token (On-Behalf-Of), alongside SSO, JWT from any IdP (Okta, Azure AD, Auth0, Cognito), SCIM, RBAC, Agent Identities, and M2M access. That's a meaningfully more flexible model than an act-as-agent-only approach.

Registering and observing servers, concretely:

Every server is registered once and consumed by many clients, with inbound and outbound auth handled separately - who can connect to the gateway versus how the gateway authenticates to each upstream server - and destructive tools can be configured to pause for human approval instead of executing on the spot.

The MCP catalogue has grown substantially: 120+ Managed MCPs and 200+ Official MCPs, with simple, guided steps to configure any MCP with no friction, plus support for deploying custom MCP servers and OpenAPI-based MCPs, which most gateways, including MintMCP, don't offer.

What's actually pulling you toward a dedicated MCP gateway versus a broader platform that includes MCP - the compliance attestation, the non-MCP agent monitoring, or something else on this list I haven't weighed heavily enough?

Best MCP Gateway for Claude in 2026: 5 Options Ranked

Sahajmeet Kaur — Thu, 23 Jul 2026 21:25:38 +0000

An MCP gateway and an AI gateway solve different problems, and "best MCP gateway" lists often blur them together. What actually matters for Claude specifically is narrower: does the gateway broker MCP servers to Claude Code and Claude Desktop with real auth handling, or is MCP support an afterthought bolted onto something built for routing LLM calls. Ranked below on that basis.

TL;DR

TrueFoundry ranks first because its MCP Gateway ties directly into Claude Code's and Claude Desktop's managed-settings mechanisms - MDM-enforced allowlisting, per-tool approval gates, and RBAC - not just generic MCP server hosting.
Docker MCP Gateway is the honest answer for a solo developer or small team wanting containerized MCP servers on their own machine with Claude Desktop or Code - genuinely simpler, but self-hosted and without fleet-wide governance.
Solo.io's agentgateway, Portkey, and Obot are all real options with real tradeoffs, covered below with where each one falls short specifically for Claude.

1. TrueFoundry

TrueFoundry's MCP Gateway is registration-once, consumption-by-many: a platform team registers an MCP server a single time, and every developer's Claude Code, Claude Desktop, or agent pointed at the gateway uses it without holding its own credentials. What puts it first specifically for Claude: MCP access from Claude Code is routed through the gateway and allowlisted directly inside managed-settings.json, deployed and locked via MDM across a developer fleet, rather than trusting whatever MCP servers individual developers wire up locally. On top of that: RBAC over which servers and tools a team can reach, destructive tools that pause for human approval instead of executing silently, Cedar-based policy for conditions finer than allow/deny, and per-tool metrics (invocation count, latency, error rate) sliced by server and by user.

Key Features:

Published benchmark - 7 to 12 ms overhead at 200 to 370 RPS
Gateway continues serving requests on its last-synced config if the control plane goes down
All rate limiting, load balancing, and auth checks run in memory, no external call in the request path
MCP gateway with tool-level RBAC, OAuth, and full audit trail, alongside model routing in the same platform

Where it's not the right call: if you're a solo developer running MCP servers locally with no fleet to govern, this is more infrastructure than you need.

2. Docker MCP Gateway (MCP Toolkit)

Docker's MCP Gateway runs each MCP server in its own container, with over 200 pre-built servers available through Docker Desktop and one-click setup for Claude Desktop and Claude Code. Cryptographically signed images and container isolation mean a compromised MCP server doesn't have direct access to your machine. Genuinely the simplest path if you already use Docker Desktop.

Where it's not the right call: self-hosted means you own the maintenance, scaling, and security burden yourself, and there's no fleet-wide RBAC or MDM-enforced policy - it's built for a single developer's machine, not centralizing access across a team.

3. Solo.io agentgateway

Solo.io's agentgateway is Kubernetes-native, contributed to the Linux Foundation in August 2025, with a Rust-based data plane and a CEL-based policy engine for MCP tool RBAC. A strong pick if your platform team already runs Envoy or a Kubernetes-centric gateway layer and wants vendor-neutral governance.

Where it's not the right call for Claude specifically: it's built for platform engineers who own the deployment layer generally, not for the specific problem of getting Claude Code's managed settings and Claude Desktop's preferences pointed at a governed MCP endpoint - there's no documented tie-in to either surface the way TrueFoundry's Claude-specific docs describe.

4. Portkey

Portkey extended its AI gateway into MCP with role-based endpoints that expose only the minimum tools required to each user or team, plus fine-grained auth controls at the org, team, and user level. If you're already running LLM traffic through Portkey, adding MCP governance on the same platform avoids a second tool.

Where it's not the right call: the same caveat as everywhere else Portkey shows up in this series - Palo Alto Networks announced intent to acquire it in April 2026, which is worth weighing before betting Claude's MCP access on its current trajectory.

5. Obot

Obot markets itself as purpose-built for MCP governance - open source, self-hostable on Kubernetes or Docker, with a built-in catalog and multi-role RBAC. Worth naming, with an honest caveat: the ranking claiming Obot is "the best MCP gateway" comes from Obot's own blog post, comparing itself against thirteen competitors and unsurprisingly placing itself first - the same self-serving-source problem worth watching for anywhere in this space, this post included.

Where this ranking doesn't hold

If you're one developer running a couple of MCP servers locally against Claude Desktop, Docker MCP Gateway is might be the right amount of tool, and TrueFoundry's fleet governance is overhead you don't need yet. The ranking above assumes you're solving for an organization with multiple developers, multiple MCP servers, and a real need to answer "who can reach what" - not a single machine.

Which of these are you actually running for Claude specifically, and did the Claude Code/Desktop tie-in factor into the choice, or was that not something you'd considered before picking?

Vercel AI Gateway vs OpenRouter: Managed Ecosystem vs Hosted Marketplace

Sahajmeet Kaur — Thu, 23 Jul 2026 21:25:20 +0000

Both Vercel AI Gateway and OpenRouter are one API key in front of many models, both are hosted rather than self-hosted, and it's easy to assume they're interchangeable. They're not, and neither is built to answer a third question worth asking alongside them: once you're past routing and into governing who can call what, is either of these actually enough.

TL;DR

Vercel AI Gateway is one endpoint (ai-gateway.vercel.sh/v1) with zero markup on tokens even with your own keys, automatic same-model failover across providers, and deep AI SDK integration - the tradeoff is it's built around the Vercel ecosystem.
OpenRouter is platform-agnostic with 315+ models, a genuinely useful free tier, and a fee structure (5.5% on card top-ups, 5% BYOK above 1M requests/month) instead of Vercel's flat $5/month credit plus zero-markup model.
TrueFoundry is the pick if you need RBAC, budget enforcement, guardrails, and MCP governance on top of routing - the thing neither Vercel AI Gateway nor OpenRouter does at all - though if you only need routing and billing, either of the other two is the simpler, sufficient choice.

What each one actually is

Vercel AI Gateway
Vercel AI Gateway routes through a single HTTP endpoint, with models referenced as creator/model-name strings - anthropic/claude-opus-4.8, for example - and in the AI SDK, passing that string as the model automatically routes the call through the gateway with no separate client. It includes embeddings alongside chat completions, spend monitoring, automatic retries, load balancing, and Bring Your Own Key. The standout feature is automatic failover to the same model on a different provider when one degrades, keeping output identical while switching which infrastructure serves it. Every team gets $5 a month in included credits from first use, and there's no markup on tokens, including with your own provider keys - you pay what the provider charges, full stop.

OpenRouter
It is a hosted aggregator: one API key and one credit balance in front of 315+ models, with an OpenAI-compatible API so switching models is a one-line change. The free tier gives 25+ free models at 20 requests/minute and 50 free-model requests a day, rising to 1,000/day once you've put at least $10 of credit on the account - a genuinely useful way to try things before committing. Pay-as-you-go is straightforward credit purchasing, and Enterprise adds SSO, SLAs, and negotiated support. OpenRouter doesn't mark up the model price itself, but there's a 5.5% platform fee on credit-card top-ups (with an $0.80 minimum) and a 5% bring-your-own-key fee on usage above 1M requests a month.

TrueFoundry
Truefoundry is the third option worth putting next to these two, and the one this post ranks first overall. Same starting point - one OpenAI-compatible endpoint in front of 1,000+ models - but with the governance layer neither Vercel AI Gateway nor OpenRouter has: RBAC scoped to users, teams, and virtual accounts, budgets that support an audit-only rollout before you switch on hard blocking, and PII/prompt-injection/content-moderation guardrails as configuration rather than something you assemble yourself. It also ships a native MCP Gateway with proper inbound/outbound auth handling and per-tool approval gates for destructive actions, which is a different category of feature than either of the other two offers at all. Deployment is managed, hybrid, or fully self-hosted in your own VPC, so data residency concerns that would rule out both hosted options entirely aren't a blocker here.

Head to head

	Vercel AI Gateway	OpenRouter	TrueFoundry
Tied to a platform	Vercel / AI SDK	None - fully platform-agnostic	None - managed, hybrid, or self-hosted
Provider markup	0%, including BYOK	0% on model price; 5.5% card-fee, 5% BYOK fee above 1M req/month	No markup; priced separately from provider cost
Free tier	$5/month included credits	25+ free models, 50-1000 free requests/day	No free hosted tier; self-hosted has no license cost
Same-model failover	Yes, across providers	Not a documented feature	Yes, with weighted/latency-based routing and canary rollouts
Zero data retention	Yes (OpenAI/Anthropic/Google), paid add-on team-wide	Not a core feature	Configurable per deployment posture
RBAC / per-team budgets	No	No	Yes, first-class
MCP server governance	No	No	Yes, native MCP Gateway

Where each one is the right call

Pick Vercel AI Gateway if your app already ships through Vercel and the AI SDK is already your interface - it's closer to a config change than new infrastructure, and the zero-markup-even-with-BYOK pricing is genuinely competitive. One independent benchmark found the native Anthropic SDK about 15-20% faster than the gateway on small prompts, with that gap nearly disappearing at large context, which is worth knowing if latency on short requests specifically matters to you.

Pick OpenRouter if you want a hosted option that isn't tied to any particular deploy target, or if the free tier matters for prototyping before you've committed to a paid plan anywhere.

Pick TrueFoundry if you've outgrown "just route the request" and need to answer who's allowed to call what, how much a team can spend before it's blocked, and whether your agents' tool calls are audited - all without assembling that governance layer yourself on top of Vercel AI Gateway or OpenRouter, neither of which is built to provide it.

Where this ranking doesn't hold: if your actual need is simple API routing for a small team or a single app, TrueFoundry's governance surface is more than you need, and the honest answer is Vercel AI Gateway (if you're on Vercel) or OpenRouter (if you're not) - both are simpler, and neither makes you configure RBAC and budgets you don't have a use for yet.

Which of these three are you actually running, and did the platform tie-in, the fee structure, or the governance gap end up being the deciding factor?

How to Choose an AI Gateway: 6 Criteria That Actually Matter

Sahajmeet Kaur — Sat, 18 Jul 2026 06:30:00 +0000

Every "top N AI gateways" list answers the wrong question first. Which one has the most providers or the lowest latency matters less than what you're actually trying to solve, and most teams pick a gateway before they've written down what they need it to do. Here's the checklist worth going through first, and how the options actually stack up against it.

TL;DR

The six things that actually differentiate gateways: provider coverage and API compatibility, deployment model, governance (RBAC/budgets/guardrails), MCP and tool governance, observability, and vendor/roadmap risk.
Most gateways are strong on one or two of these and thin on the rest - a routing-and-caching tool isn't a governance tool, even if both get called "AI gateway."
TrueFoundry covers all six without you having to assemble pieces from separate tools, which is why it's the pick below for teams that need the full set.

1. Provider coverage and API compatibility

The baseline question: how many providers, and is the exposed API OpenAI-compatible so your existing SDKs work with a base-URL change, or does it ask you to adopt a new client. Most serious options clear this bar now - LiteLLM covers 100+ providers, Bifrost claims 1,000+ models, TrueFoundry exposes an OpenAI-compatible schema in front of 1,000+ LLMs. This one rarely eliminates anyone anymore; treat it as table stakes, not a differentiator.

2. Deployment model

Managed SaaS, self-hosted, or hybrid - and this one does eliminate options fast if data residency or compliance rules out sending traffic through someone else's infrastructure. LiteLLM, Bifrost, and Portkey are self-hostable and free of that constraint by design. Vercel AI Gateway and Cloudflare AI Gateway are managed-only, tied to their respective platforms. TrueFoundry runs managed, hybrid, or fully self-hosted in your own VPC, which matters if you want the option to switch postures later without switching vendors.

3. Governance: RBAC, budgets, guardrails

This is where most gateways split into two camps. One camp treats routing as the whole job - Cloudflare AI Gateway and Vercel AI Gateway give you caching, retries, and spend visibility, but no per-team RBAC and no budget enforcement that blocks a request before it blows past a limit. The other camp treats governance as first-class: TrueFoundry ships RBAC scoped to teams and users, budgets that support an audit-first rollout before you switch on hard blocking, and PII/prompt-injection/content-moderation guardrails as configuration, not something you build separately. LiteLLM and Portkey also have real governance features here, so this criterion narrows the field without settling it on its own.

4. MCP and tool governance

Increasingly the deciding factor as agents start calling tools that can actually do things, not just answer questions. A gateway with a native MCP Gateway handles inbound and outbound auth for every registered MCP server, lets you pause destructive tool calls for human approval, and gives you a full audit trail per tool call - versus a gateway that only routes LLM traffic and leaves MCP access as something each developer wires up on their own. This is the criterion most "which AI gateway" comparisons skip entirely, and it's often the one that matters most a year into using the thing.

5. Observability

Cost, latency, and guardrail metrics, ideally sliceable by user, team, or virtual account, and exportable to wherever your team already looks. TrueFoundry's metrics dashboard covers LLM and MCP traffic together with OpenTelemetry export; most self-hosted options expect you to wire this up to Langfuse, SigNoz, or a similar backend yourself, which is fine if you already run one of those and want to keep it as your system of record.

6. Vendor and roadmap risk

The one people skip until it bites them. Portkey open-sourced its gateway in March 2026, then Palo Alto Networks announced intent to acquire it the next month. Helicone was acquired by Mintlify in March 2026 and moved to maintenance mode - security patches continue, active feature development stopped. Neither makes either tool bad today, but "will this still be actively developed in a year" is a real question for infrastructure you're building a business on, and it's worth asking about any vendor, including the one you're leaning toward.

How TrueFoundry stacks up against each criterion

Rather than assert "it covers all six" and move on, here's the specific claim behind each one, sourced from the docs rather than a summary:

Provider coverage and API compatibility. 1,000+ LLMs behind one OpenAI-compatible schema (linked above), with drop-in native SDK compatibility for OpenAI's and Anthropic's own client libraries, plus chat, embeddings, image, audio, rerank, and realtime APIs rather than chat completions alone.
Deployment model. Managed, hybrid, or fully self-hosted in your own VPC (linked above), so the deployment posture you start with isn't the one you're stuck with if compliance requirements change later. The managed option itself runs as a globally distributed gateway across 14 regions rather than a single point of failure.
Governance. RBAC scoped to users, teams, and virtual accounts; budgets that support an audit-only rollout before you flip on hard blocking; PII, prompt-injection, and content-moderation guardrails as configuration rather than something you build (all linked above); and for teams that need finer-grained rules than allow/deny, Cedar-based policy for conditions like "only during business hours" or "only for low-risk tools."
MCP and tool governance. The MCP Gateway (linked above) handles both inbound (client-to-gateway) and outbound (gateway-to-server) auth separately, so credentials never sit on a developer's machine; destructive tools can be configured to pause for human approval instead of executing silently; and tool-level metrics (invocation count, latency, error rate) are tracked per server and per individual tool, not just at the gateway level.
Observability. The metrics dashboard (linked above) covers LLM and MCP traffic, cost, guardrail outcomes, routing decisions, and cache hit rate in one place, with OpenTelemetry export to whatever backend your team already runs. Data access rules also let you restrict who can read production request logs versus dev/staging, which matters once those logs contain real prompts and completions.
Vendor and roadmap risk. This one is honestly harder to claim credit for outright since every vendor has some version of this risk, TrueFoundry included. What actually mitigates it: gateway configuration (models, virtual accounts, policies, guardrails, access rules) can be managed as YAML in Git via GitOps, so your setup isn't locked inside a vendor's UI state, and the self-hosted deployment option means a change in vendor relationship doesn't mean ripping out infrastructure that's already running in your own VPC.

Where this points, and where it doesn't

Weighing all six, TrueFoundry is the pick if you want governance, MCP tool control, and observability together without assembling them from separate projects, and you want the option to run managed, hybrid, or self-hosted depending on how your compliance posture evolves.

It's not automatically the right call for everyone. If free and fully open source is the actual requirement, not just a preference, LiteLLM remains the honest answer - you'll assemble more of the governance layer yourself, but you own the whole stack and pay nothing for the software itself. If your traffic is entirely Vercel-shipped apps on the AI SDK, Vercel AI Gateway is less new infrastructure. If you're lakehouse-native on Databricks already, Unity AI Gateway keeps your AI governance in the same catalog as your data governance, at the cost of being tied to that ecosystem. None of the six criteria above matter equally for every team - the point of going through them explicitly is figuring out which ones actually apply to you before picking anything.

Which of these six would you add or remove from the list, based on what actually mattered when you picked (or switched) gateways?

Best AI Gateways for Claude in 2026

Sahajmeet Kaur — Fri, 17 Jul 2026 16:21:02 +0000

"Does it support Claude" clears almost every gateway on this list - that part's been table stakes for a while. What actually separates them is whether Claude is governed everywhere it shows up in your org: the API, Claude Code on developer laptops, Claude Desktop, Claude Code Max subscriptions, and Claude Platform on AWS. Ranked below on that basis, not just "can it proxy Anthropic's API."

TL;DR

TrueFoundry works vest because it governs Claude across all four client surfaces plus the AWS deployment path, with MDM enforcement on Code and Desktop - not just server-side API routing.
LiteLLM, OpenRouter, Bifrost, and Vercel AI Gateway all proxy the Claude API well; none of their published docs describe governing Claude Code or Claude Desktop specifically across a device fleet.
If your Claude usage is API-only with no Code or Desktop in the picture, the surface-coverage argument for #1 doesn't apply to you, and any of the four below is a genuinely fine, simpler choice.

1. TrueFoundry

TrueFoundry's Claude-specific docs map governance to each surface separately, which is the thing that puts it first on this list:

Claude Code (CLI + VS Code): ANTHROPIC_BASE_URL set inside managed-settings.json, deployed and locked via an open-source binary pushed over MDM on macOS, Linux, and Windows, with the gateway token refreshed on a schedule.
Claude Desktop: a separate managed-preferences mechanism (its own bundle identifier), configured by the same deployment tooling so platform teams run one rollout process, not two.
Claude Code Max: Claude Code reserves the Authorization header for a developer's own Anthropic subscription login, so the gateway authenticates through a separate x-tfy-api-key header instead - developers keep their Max subscription exactly as before, while the gateway still gets usage visibility, quotas, RBAC, logs, and guardrails.
Claude Platform on AWS: authenticated through IAM rather than a bearer key, with both a quickstart managed policy and a least-privilege policy scoped to a workspace ARN documented for teams that want it locked down properly.
MCP access from Claude Code: routed through a central MCP Gateway and allowlisted in managed settings, rather than trusting whatever MCP servers individual developers wire up locally.

Underneath all of that sits the same RBAC, budgets, and guardrails that apply to every model call through the gateway, plus deployment as managed, hybrid, or fully self-hosted depending on your compliance posture. Where it's not the right call: if your Claude usage is entirely server-side API calls, this breadth of surface coverage is more than you need, and a simpler gateway lower on this list is the honest answer.

2. LiteLLM

LiteLLM routes to Claude as one of its 100+ supported providers through the same OpenAI-compatible proxy, with cost tracking and load balancing built in. Free, open source, and self-hosted, with the largest community of any option here.

Where it falls short for Claude specifically: it's API-level routing. Nothing in its published docs governs Claude Code or Claude Desktop as separate surfaces - if developers are running those locally, LiteLLM isn't the layer watching them.

3. OpenRouter

One API key, one credit balance, Claude among 315+ models, pass-through provider pricing with a 5.5% card fee and a 5% BYOK fee above 1M requests a month. Zero setup if you just want to call Claude (or switch between Claude and other models) without standing up infrastructure.

Where it falls short for Claude specifically: it's a hosted aggregator built around API access - no mechanism for governing Claude Code or Claude Desktop on developer machines, and no RBAC or per-team budget enforcement beyond the single account's credit balance.

4. Bifrost

Bifrost is a Go-based open-source gateway that markets itself on raw throughput - its own benchmarks claim well under 100 microseconds of overhead at 5,000 requests per second. Claude is one of the 1,000+ models it routes to.

Where it falls short for Claude specifically: same gap as the others - it's a high-performance API router, not a tool for governing Claude Code or Desktop across a device fleet. If raw throughput at extreme RPS is genuinely your bottleneck and Claude is just one of several models you're calling, it's worth a look; if surface governance is the concern, it isn't built for that.

5. Vercel AI Gateway

One endpoint, zero markup on tokens even with your own keys, and Claude works with a one-line config change if you're already on the Vercel AI SDK. Genuinely the least new infrastructure if your app already ships through Vercel.

Where it falls short for Claude specifically: it's routing and spend visibility for API calls from Vercel-shipped apps. It has no bearing on Claude Code or Claude Desktop running on a developer's own machine, which is a separate problem it was never built to solve.

Where the ranking doesn't apply

If your org's entire Claude footprint is server-side API calls - no Claude Code, no Claude Desktop, nothing running on developer laptops - the surface-governance case for TrueFoundry ranking first doesn't hold, and any of the four options above is a legitimately simpler, sufficient choice. The ranking here is specifically about governing Claude everywhere it can show up in a company, not just proxying the API well, which every one of these five already does.

Is your org's Claude usage mostly API calls, or is Claude Code/Desktop already running on developer machines without anything governing it yet? Curious how many teams have actually mapped out which surface they're exposed on before picking a gateway.

DEV Community: Sahajmeet Kaur

Databricks AI Gateway alternatives: the best options for 2026

What to look for in a Databricks AI Gateway alternative

The best Databricks AI Gateway alternatives

1. TrueFoundry: best overall

2. LiteLLM

3. Portkey

4. Kong AI Gateway

5. Cloudflare AI Gateway: best for edge caching

Databricks AI Gateway vs the alternatives: comparison

Related reading

Conclusion

Apigee alternatives: the best AI gateway options for 2026

What to look for in an Apigee alternative

The best Apigee alternatives

1. TrueFoundry: best overall

2. LiteLLM

3. Portkey

4. Kong AI Gateway

5. Cloudflare AI Gateway

Apigee vs the alternatives: comparison

Related reading

Agent observability: How to trace prompts, tools, and models in production?

What is agent observability?

Agent observability matters most when:

What to instrument in an agent

How TrueFoundry handles agent observability

Best practices for agent observability

Related reading

What is a virtual MCP server? Curating tools across MCP servers

What is a virtual MCP server?

A virtual MCP server works well when:

How a virtual MCP server works

How tool names are handled

Virtual MCP server vs a raw MCP server

Where virtual MCP servers fit in the MCP gateway

When to use a virtual MCP server

Related reading

LLM Gateways Explained: What They Do and Why You Need One

TL;DR

What the gateway actually does

Reason one: providers go down, and you don't want to go down with them

Reason two: rate limits hit hardest exactly when you can't afford them

Reason three: SDK lock-in makes switching models expensive later

What you get once everything routes through one place

Where I'd push back on this

LLM Cost Controls That Actually Work: Budgets, Rate Limits, and Per-Team Attribution

TL;DR

What TrueFoundry AI Gateway actually is

See the spend as it happens, not after the invoice

Budgets that don't break things on day one

Rate limits are a cost control, not just a reliability one

Attribution: the part that makes the dashboard mean something

Two more levers worth using: caching and routing

Where this still requires a human decision

MCP Access Control: RBAC, Per-Tool Restrictions, and Approval Gates for Your MCP Servers

TL;DR

TrueFoundry MCP Gateway

The four questions access control actually has to answer

4. Watching what actually gets called

Where guardrails fit in alongside access control

MintMCP Alternative: Why TrueFoundry is a better choice for MCP Governance

TL;DR

What MintMCP actually does well

Where TrueFoundry's case is stronger

Best MCP Gateway for Claude in 2026: 5 Options Ranked

TL;DR

1. TrueFoundry

2. Docker MCP Gateway (MCP Toolkit)

3. Solo.io agentgateway

4. Portkey

5. Obot

Where this ranking doesn't hold

Vercel AI Gateway vs OpenRouter: Managed Ecosystem vs Hosted Marketplace

TL;DR

What each one actually is

Head to head

Where each one is the right call

How to Choose an AI Gateway: 6 Criteria That Actually Matter

TL;DR