Three things are called "gateway" in your AI stack right now. They do completely different jobs.
If you're shipping AI features and trying to figure out whether you need an AI Gateway, an MCP Gateway, or an Agent Gateway — most blog posts will hand-wave and say "it depends." That's useless.
Here's the real difference, in the order you'll actually hit them.
1. AI Gateway — the proxy in front of model providers
An AI Gateway sits between your app and OpenAI / Anthropic / Google / Bedrock. It's the load balancer of LLM calls.
What it does:
- Routing — fall back from Claude to GPT if one provider 5xx's
- Rate limits & quotas — per-user, per-team, per-feature
- Caching — semantic or exact-match
- Cost tracking — per token, per route, per environment
- Auth — strip your provider key, give each team a virtual key
// Without a gateway
const res = await openai.chat.completions.create({ ... });
// With a gateway (Portkey / LiteLLM / OpenRouter / Cloudflare AI Gateway)
const res = await fetch("https://gateway.yourcompany.com/v1/chat", {
headers: { Authorization: `Bearer ${teamVirtualKey}` },
body: JSON.stringify({ model: "claude-opus-4.7", messages })
});
If you have three or more teams calling LLMs and you can't answer "how much did the support team spend on Claude last month?" in 10 seconds, you need this.
2. MCP Gateway — the proxy in front of tool servers
Model Context Protocol (MCP) lets an LLM call tools — read a file, run SQL, hit a Notion page, send a Slack message. Each capability lives in an MCP server.
When you have 12 MCP servers, you have a problem:
- Which user is allowed to call which tool?
- Which prompts can invoke
database.delete_row? - How do you audit every tool call?
- How do you stop one runaway agent from racing through 400 calls/min?
An MCP Gateway is the policy layer in front of tools. Same idea as an API gateway, but the consumer is an LLM, and the requests are non-deterministic.
If you're letting AI act on production systems, the gateway is where you put the "are you sure?".
3. Agent Gateway — the proxy between agents
This one is newer. An Agent Gateway is what you put when agent-to-agent communication starts happening:
- One agent dispatches to a specialist agent
- A user's agent talks to a vendor's agent (A2A protocol territory)
- You're running a multi-agent system and want a single audit log
It handles identity ("this agent represents user X, on tier Y"), permissions, and conversation routing. Less mature than the other two, but if you're building multi-agent workflows, this is the gap you'll hit next.
Which one do you actually need?
Start at the bottom of the stack and only add the next layer when it hurts:
| Pain you feel | Layer to add |
|---|---|
| "I have no idea how much we spend on OpenAI" | AI Gateway |
| "An agent just deleted production data" | MCP Gateway |
| "I can't tell which agent called which agent" | Agent Gateway |
Most teams need AI Gateway first, MCP Gateway when they connect tools to prod, and Agent Gateway only if they're going full multi-agent.
The non-obvious takeaway
These aren't competing — they're a stack. Calls go: App → AI Gateway → LLM → MCP Gateway → Tools → Agent Gateway → Other agents.
The mistake is buying "an agent platform" that bundles all three before you've felt the pain of any one of them. You end up with vendor lock-in for problems you don't have yet, and zero visibility into the problems you do.
Build the gateway you need today. Add the next one when the bill, the breach, or the audit forces you to.
Following LayerZero — we break down the infrastructure that ships AI products. Next up: why most teams set MCP permissions wrong, and the 3-line policy that fixes it.
Top comments (0)