A practical breakdown of what an AI gateway actually sees versus what an MCP gateway sees, and why production agents usually need both
Picture a fairly normal setup. You've got an agent behind an AI gateway. Prompts are rate limited, PII gets scrubbed, costs are tracked per team, jailbreak attempts get flagged. Solid setup. Then the agent picks up tool calling through MCP so it can hit your CRM, your internal search index, a couple of internal APIs. Nothing changes on the gateway side because, as far as the gateway is concerned, nothing changed. The model call still looks the same going in and coming out.
Except now there's a whole second leg of the request that nobody is watching.
Two different jobs, one request path
An AI gateway is a reverse proxy for LLM traffic. It sits between your app and whichever model provider you're calling (OpenAI, Anthropic, Bedrock, Gemini, self-hosted, doesn't matter) and normalizes all of that behind one interface. The stuff it's built for is specific to model traffic: token-based rate limits instead of raw request counts, streaming responses, provider fallback when one goes down, semantic caching, cost attribution, and content policy on the prompt going in and the completion coming out.
What it does not do, structurally, is see what happens after the model decides to call a tool. Once a completion includes a tool call, execution moves to whatever runs your agent's tool logic, and if that's happening over the Model Context Protocol, it's a different transport entirely.
That's the gap an MCP gateway is built to close. MCP is an open standard for letting an agent discover and call external tools and data sources through a consistent client-server interface, instead of every team hand-rolling a connector for every model-to-system combination. It's often described as turning an M×N integration problem (M agents, N systems, custom glue for every pair) into an M+N one, one client, one server per system, and anything MCP-compliant can talk to anything else MCP-compliant.
Convenient, but MCP itself has no opinion on governance. Nothing in the protocol stops an agent from calling every tool a server exposes, and nothing logs what it actually did with them. An MCP gateway sits in front of that traffic and adds:
- Auth for every agent-to-server connection, OAuth flows for remote servers included
- Per-tool, per-agent authorization, not just "can this agent reach the server" but "which specific tools on it"
- Inventory of MCP servers across the org, including the shadow ones nobody registered
- Audit logs of who called what, with what arguments, and what came back
- Inspection of tool calls and results for injection attempts or exfiltration hiding in a response
What that actually looks like
A rough shape for a per-tool policy an MCP gateway would enforce:
agent: support-bot
server: internal-crm
allow:
- tool: lookup_customer
max_calls_per_minute: 20
- tool: update_ticket_status
require_approval: false
deny:
- tool: export_customer_data
audit: all
None of that is something an AI gateway has any hooks for, because it's not model traffic. It's a different policy surface with a different unit of enforcement (per tool, per server, per agent identity) versus the AI gateway's unit (per model, per provider, per token).
So which one do you need
If your agent only sends prompts to a model and never touches a tool, an AI gateway on its own probably covers you. The moment it calls into MCP servers to query a database, hit an API, or take an action on an external system, you've got a blind spot without MCP-layer controls. Most production agents that do real work fall into the second bucket, which is why most serious deployments end up running both layers, ideally on shared infrastructure so policy isn't defined twice and drifting apart.
This is roughly the bet behind NeuralTrust's TrustGate, an open-source gateway handling LLM, MCP, and agent-to-agent traffic under one control plane. The reasoning holds regardless of which vendor or open-source project you land on: a threat that gets past the model-traffic layer can still get caught at the tool-traffic layer, and vice versa, but only if something is actually watching both. It's the same instinct behind the broader move toward dedicated agent security tooling rather than trying to stretch a general API gateway or a model-only gateway to cover agentic behavior it wasn't designed to see.
The standards bodies are catching up
This isn't just a vendor talking point anymore. In December 2025, Anthropic donated MCP to the newly formed Agentic AI Foundation, a directed fund under the Linux Foundation co-founded with Block and OpenAI, specifically to keep governance of the protocol vendor-neutral as adoption scales. And in February 2026, NIST's Center for AI Standards and Innovation launched an AI Agent Standards Initiative, with one of its three pillars aimed squarely at agent identity, authentication, and authorization. Two separate signals pointing at the same conclusion. The controls an MCP gateway provides today (identity, per-tool auth, audit trails) are on a trajectory to becoming baseline expectations, not optional extras.
If you want the deeper dive on AI gateways specifically, including where they overlap with plain API gateways, NeuralTrust has a longer breakdown here. And the original comparison this post is based on goes further into the architecture diagrams and a full feature-by-feature breakdown if you want the long version.
Bottom line: an AI gateway secures the thinking. An MCP gateway secures the acting. If your agent does both, so should your infrastructure.
Top comments (0)