Without a gateway, your application code holds a provider SDK directly and calls it. Add a second provider and you're holding two SDKs with two different request shapes, with an if/else deciding which one to use. An LLM gateway sits in front of every provider instead, exposes one API, and translates your request into whatever the actual provider expects on the other side - this is what that looks like using TrueFoundry's gateway specifically.
TL;DR
- An LLM gateway is one endpoint in front of every model provider, so your code changes a model name string instead of a client library when you switch or add one.
- The three concrete reasons teams add one - provider outages, rate limits, and SDK lock-in - are all things TrueFoundry's gateway handles as routing features (automatic failover, per-user/team rate limits, one OpenAI-compatible schema across 1,000+ models), not things you build separately.
- Once every call passes through one place, that place is also where RBAC, budgets, guardrails, and MCP governance naturally live - not because the gateway is "for" governance, but because it's the only component that sees every request.
What the gateway actually does
TrueFoundry's AI Gateway exposes an OpenAI-compatible schema in front of 1,000+ models across providers, with native SDK compatibility for OpenAI's and Anthropic's own client libraries - meaning your existing code often works with just a base-URL change, not a rewrite. Your application never holds a provider SDK directly; it points at the gateway and changes a model name string to switch what's behind it.
Reason one: providers go down, and you don't want to go down with them
OpenAI and Anthropic both had multiple incidents on their public status pages between February and May of 2025 - not exotic, just the reality of depending on someone else's infrastructure. Load balancing and fallback routes across multiple targets by weight, latency, or priority with automatic retries, so a provider outage means failing over to a healthy target instead of your application going down with it. The same mechanism supports canary rollouts - routing a small percentage of traffic to a new model and watching it before trusting it with everything.
Reason two: rate limits hit hardest exactly when you can't afford them
Azure OpenAI, like most providers, enforces tokens-per-minute and requests-per-minute quotas per model per region. Under a traffic spike, or a bug that puts an agent into a retry loop, you hit that ceiling and start getting 429s. Rate limiting applied per user, team, or application means a gateway that understands the limit can route overflow elsewhere instead of just failing the request - and it doubles as a cost control, capping what a single bug or a single team can run up.
Reason three: SDK lock-in makes switching models expensive later
If your codebase calls a specific provider's SDK in forty places, swapping in a cheaper or better model six months from now means touching forty places. With the gateway's unified schema, that's a config change - a model name string - not an application rewrite.
What you get once everything routes through one place
This is the part that surprises people who adopted a gateway purely for routing: RBAC scoped to teams and users, budgets with an audit-first rollout before you switch on hard blocking, PII/prompt-injection/content-moderation guardrails, and a native MCP Gateway for governing tool calls - all sitting on the same request path as the routing, because that path is the only place that sees every call regardless of which team or model made it. None of it is a separate system you have to keep synchronized with the gateway; it's configuration on the same one.
Deployment is the other piece worth knowing up front: managed, hybrid, or fully self-hosted in your own VPC, so "we can't send traffic through someone else's infrastructure" doesn't rule this out the way it would with a purely hosted option.
Where I'd push back on this
If you're calling one provider, for one use case, at low volume, none of this is solving a problem you have yet. Add a gateway once you're multi-provider, multi-team, or once an outage or a rate-limit wall has actually cost you something - not before, and not because RBAC and budgets sound like good practice in the abstract. This breakdown of gateway vs. proxy vs. router terminology is a good next read if the terms are still blurring together, and there's a solid walkthrough of wiring automatic provider fallback into an agent if you want to see the failover pattern in code first.
What pushed you to add a gateway, if you have one - an outage, a rate-limit wall, or something else entirely?
Top comments (0)