A team in your org hits Anthropic during a production incident. Anthropic is down. The team is blocked for 20 minutes while someone scrambles to find an alternative API key, change a config, and redeploy. There's no fallback. There was never a plan for this.
Somewhere else, another team has quietly started billing their LLM usage to a personal credit card because procurement takes six weeks. A third team has hardcoded gpt-4o everywhere and doesn't know there's a cheaper model that handles their workload fine.
This isn't a story about AI risk. It's a story about missing infrastructure.
The pattern you've seen before
When teams first started running services in the cloud, the ones who moved fast didn't start with a load balancer and a CDN. They spun up an EC2 instance, pointed DNS at it, and shipped. That worked — until it didn't. Until traffic spiked, until the instance went down, until three teams had three different DNS setups and nobody knew which was canonical.
AI adoption is following the same curve, faster. Every team that can call an API is calling one. The platforms don't have a control plane yet. The control plane is somebody's laptop with a hardcoded key.
An AI gateway is what closes that gap. It's not a product pitch — it's the same thing you built for HTTP traffic: a single ingress point with routing, fallback, observability, and policy enforcement. The LLM is the backend. The gateway is the infrastructure around it.
What the gateway actually does
flowchart TD
A["Developer — MCP / SDK / REST client"] --> G["AI Gateway — single stable endpoint"]
B["Business user — internal tooling"] --> G
G -->|"primary"| P1["Anthropic Claude"]
G -->|"fallback on 5xx / timeout"| P2["AWS Bedrock — Claude / Titan"]
G -->|"cost-optimised route"| P3["OpenAI GPT-4o mini"]
G --> OBS["Observability — spend per team — request logs — latency"]
G --> POL["Policy — rate limits — budget caps — PII filter"]
Failover. When a provider goes down, the gateway retries against the next configured backend — same request, different model, no client code change. This is the feature that looks optional until someone spends 20 minutes blocked mid-incident.
Cost routing. Not every call needs a frontier model. Summarisation, classification, short completions — route those to something cheaper. The gateway makes this a config change, not a refactor.
Spend visibility. Every request is tagged by team, application, and model. You see where the money goes before finance calls you about it. You can set hard budget caps per team so nobody accidentally runs a batch job against claude-opus all night.
Provider abstraction. When a new model ships — or when your organisation wants to experiment with providers you don't use today — teams swap the gateway config, not their application code. The SDK they're using doesn't change.
Audit logging. In any regulated environment, knowing who sent what prompt to which model isn't a nice-to-have. It's table stakes for governance. The gateway is the right place to capture that, not each individual application.
What self-hosting looks like in practice
The two tools worth knowing are LiteLLM and Bifrost. Both expose an OpenAI-compatible API, which means existing applications need zero changes to the SDK call — only the base URL and API key change.
LiteLLM is the more battle-tested option. It runs as a Docker container, connects to a Postgres database for request logs and virtual key management, and supports every major provider out of the box. A minimal deployment on Kubernetes looks like this:
apiVersion: apps/v1
kind: Deployment
metadata:
name: litellm
spec:
replicas: 2
template:
spec:
containers:
- name: litellm
image: ghcr.io/berriai/litellm:main-latest
args:
- "--config"
- "/etc/litellm/config.yaml"
env:
- name: DATABASE_URL
valueFrom:
secretKeyRef:
name: litellm-secrets
key: database-url
volumeMounts:
- name: config
mountPath: /etc/litellm
volumes:
- name: config
configMap:
name: litellm-config
The config itself is where the routing logic lives:
model_list:
- model_name: claude-sonnet
litellm_params:
model: anthropic/claude-sonnet-4-5
api_key: os.environ/ANTHROPIC_API_KEY
- model_name: claude-sonnet
litellm_params:
model: bedrock/us.anthropic.claude-sonnet-4-5-20251001-v2:0
aws_region_name: eu-west-1
router_settings:
routing_strategy: least-busy
num_retries: 2
fallbacks:
- claude-sonnet:
- claude-sonnet
litellm_settings:
success_callback: ["langfuse"]
failure_callback: ["langfuse"]
Two entries for the same model_name is the whole failover mechanism. LiteLLM tries the first, fails, retries against the second. From the client's perspective, the call either succeeds or returns an error after both backends fail — never a 20-minute block.
The latency argument doesn't hold here
The pushback you'll hear is latency. "We can't add a proxy hop to every LLM call."
That argument makes sense for traditional APIs where you're measuring response times in milliseconds. It doesn't apply to LLMs. Time-to-first-token on a typical Claude call is 300–800ms. A gateway adds 5–15ms. That's noise, not overhead. The conversation about "proxy latency" is a conversation that predates LLMs and doesn't transfer.
The actual latency concern worth having is gateway availability — if the gateway goes down, everything goes down. The answer is the same as any other critical path: multiple replicas, a readiness probe, a PodDisruptionBudget, and a runbook. Nothing exotic.
The enforcement problem
One thing the gateway doesn't solve on its own: casual users who bypass it entirely. If teams can buy a Claude.ai subscription for $20/month and skip the gateway, many will — especially if the gateway requires a non-trivial onboarding step.
This is a governance problem, not a technical one. The gateway has to be the path of least resistance, not just the official path. That means:
- API keys that work without friction (virtual keys in LiteLLM's admin UI take 30 seconds to issue)
- A clear internal page that says "here is your endpoint and here is your key"
- A procurement policy that routes all model spend through the central key, not individual accounts
The technical infrastructure is the easier part. Getting adoption is the harder part, and that's a platform team problem from day one, not something you solve after the gateway is running.
Field notes
- Run the gateway in at least two replicas from day one. A single-replica proxy is a single point of failure you introduced yourself. Two replicas with a PDB that disallows simultaneous evictions cost almost nothing and remove the worst failure mode.
- LiteLLM's virtual key system is the right unit of cost tracking. One virtual key per team, budget cap attached, spend dashboard out of the box. Don't try to build this yourself — it's already there.
- Failover works at the gateway level, not the application level. Stop trying to handle retries in application code when you have a gateway. Put the retry and fallback logic in one place and let every team benefit from it automatically.
-
Tag every request with
metadata. LiteLLM forwards custom metadata to your observability backend. Use it to tag team, environment, and use case from the first day — you'll want that granularity when someone asks why the bill doubled. - SDK compatibility means the migration is low-friction. If you start with LiteLLM and later want to evaluate Bifrost or something else, the migration is a base URL change for application teams. That's the right amount of coupling.
The teams that will struggle with AI costs and reliability in 12 months are the ones treating model access like a SaaS subscription — something each team manages independently. The teams that won't are the ones who put a control plane in front of it now, before the sprawl hardens into something unauditable.
The gateway isn't the exciting part of the AI platform. It's the infrastructure that makes the exciting parts sustainable.
Top comments (0)