Why Your AI Gateway Isn't Secure for Agents: The Tool-Call Security Gap

Paul Twist | AI Infrastructure Engineer, Berlin

---

You probably have an AI gateway. LiteLLM, Bifrost, Cloudflare AI Gateway, Kong—something routing traffic to your LLM providers. It's fast. It handles caching, failover, rate limiting. It's working fine.

But if you're running agents, your gateway is missing something critical: it cannot distinguish between a model invocation and a tool call.

This gap is emerging as the primary security boundary that separates LLM-only infrastructure from production agent infrastructure.

The Invisible Problem

Your gateway sees one stream of API calls:

1. Request to invoke Claude → route to Anthropic
2. Request to invoke GPT → route to OpenAI
3. Request to invoke an agent that will call your database → route to Anthropic

All three look identical to a traditional gateway. But they're operationally different:

• Call 1 and 2 are read-mostly. The LLM reasons, returns text. No persistent side effects.
• Call 3 is write-heavy. The agent will invoke tools, spend money, modify state, call your production systems.

Your gateway can enforce rate limits on call 3. It cannot tell whether the agent should be allowed to invoke your database tool.

If the agent is malicious, misconfigured, or compromised, it can invoke tools your gateway has no visibility into.

The Security Debt

Three converging pressures are making this gap dangerous:

1. Agents are now production workloads.

57% of organizations have agents in production, with another 30% actively developing agents for deployment. They're not chat interfaces—they're automating workflows, calling production APIs, executing transactions.

2. Tool authorization moved out of LLM context.

For years, security teams said: "just tell the model in the system prompt what it can't do." That never worked. Agents learn to work around system prompts. Modern production patterns separate deterministic execution (transactions, financial calculations, anything with a binary correct answer) from LLM reasoning.

That means authorization must live at the invocation layer, not the instruction layer. Your gateway needs to know: "This agent is allowed to call tool X. This agent is not."

3. Regulatory deadlines made this non-optional.

Quality is the production killer for agents, with 32% citing it as the top barrier. But compliance teams ask a different question: "Prove this agent was authorized to make that decision." Your gateway needs an immutable audit trail showing which agent invoked which tool, when, and why.

EU AI Act Article 14 (effective August 2) and Colorado AI Act (effective June 30) require audit trails for high-risk agent decisions. Your LLM gateway can log that OpenAI was called. It cannot log that an agent called a database deletion tool without permission.

What Gateway Security Looks Like for Agents

Agent gateway security has five layers:

Layer 1: Route model requests (your gateway does this)

• Which LLM provider should handle this?
• OpenAI? Anthropic? Fallback?

Layer 2: Authorize tool calls (most gateways don't do this)

• Which agent is making this call?
• Is this agent allowed to invoke this tool?
• What's the per-tool rate limit for this agent?

Layer 3: Audit tool invocation (your gateway can't do this without context)

• Which agent called which tool?
• What were the inputs?
• What was the result?
• Was the result unexpected (drift detection)?

Layer 4: Enforce deterministic boundaries (your gateway has no visibility)

• Model invocations go to LLMs.
• Tool invocations go to tool endpoints.
• These have different SLOs, different latencies, different security profiles.

Layer 5: Isolate agent identity (your gateway sees one API key)

• Agent A shouldn't inherit credentials from Agent B.
• Tool invocations from Agent A shouldn't have access to Agent B's data.
• Per-agent identity, not shared keys.

Traditional gateways handle 1 and partially 3. They skip 2, 4, and 5.

Why This Matters in Production

Here's what happens when your gateway is LLM-only but your agents are production-grade:

Scenario 1: Unintended tool access

Agent A (customer support) gets escalated to Agent B (billing).
If Agent B's API key has access to billing_modify tool,
Agent A might accidentally invoke it mid-conversation.
Your gateway sees it as "another API call to the backend."
It's allowed, so it routes it.
Agent A just modified a customer's billing without authorization.

Scenario 2: Credential sprawl

You hand every agent developer an API key.
They hardcode it in their harness.
An agent gets compromised.
Your gateway can't tell which agent-originated the malicious tool call.
Audit says: "Something called the database, we don't know what."
Compliance team asks: "Which agent was it?"
You have no answer.

Scenario 3: Cost explosion

Agent makes 50 tool calls per session (normal for multi-step workflows).
One tool is marked as free in your gateway (load testing endpoint).
Your gateway routes all 50 calls without cost attribution.
Agent behavior drifts. It discovers the free tool.
It starts calling it 500 times per session.
Your ops team discovers it when AWS bill arrives.

All three are gateway problems. All three require tool-call level authorization, not just LLM-level routing.

What Production Agent Gateways Are Building

The emerging pattern is explicit in the industry:

Agentic systems demand a new class of context-aware, AI-native gateways. If your gateway can't tell the difference between a tool call and a model invocation, it can't enforce meaningful security.

Production platforms are separating:

Data plane (fast, stateless):

• Route model requests to the right provider
• Cache responses
• Handle failover
• Minimize latency (sub-millisecond overhead)

Control plane (reliable, stateful):

• Manage agent identity
• Authorize tool invocations
• Track per-agent budgets
• Emit audit logs
• Enforce policies without redeployment

LiteLLM-Rust handles the data plane. LiteLLM Agent Platform handles the control plane. Your agents run on top, with both layers working together.

Governance increasingly sits at the AI gateway layer rather than inside AI agent development code itself. The gateway layer applies RBAC, token budgets, and immutable logs to every AI agent workflow.

Evaluating Your Gateway for Agents

Five questions to ask:

1. Can your gateway authorize tool calls per agent? (Or just per API key?)
2. Does it emit immutable audit logs of tool invocations? (Or just LLM calls?)
3. Can it enforce per-agent budgets at the tool level? (Or just token-level limits?)
4. Does it track which agent invoked which tool? (Or just aggregated logs?)
5. Can it change authorization policies without redeploying agents? (Or hardcoded in the application?)

If you answer "no" to any of these, your gateway is LLM-only, not agent-ready.

This doesn't mean your agents will fail. It means when they do, your audit trail is incomplete. When a tool gets called by accident, you won't know which agent did it. When compliance asks "prove authorization," you can't.

The Competitive Signal

Gateway platforms like Bifrost now unify LLM gateway, MCP gateway, and Agents gateway capabilities into a single platform. Designed for regulated industries and strict enterprise requirements, they support air-gapped deployments, VPC isolation, and on-prem infrastructure.

That language—"agents gateway"—signals a recognition that agents need infrastructure primitives that LLM-only gateways don't provide.

LiteLLM's approach treats agents as a distinct workload type that requires:

• Fast data plane for model routing (LiteLLM-Rust)
• Stateful control plane for tool authorization and audit trails (LiteLLM Agent Platform)
• MCP as the tool-call standard (not framework-specific tool definitions)

This isn't gateway optimization. It's gateway rearchitecture around the agent workload.

What Changes Next

Three things will accelerate adoption of agent-aware gateways:

1. The first incident. A team deploys an agent to production, it calls tools it shouldn't, and they have no audit trail. Compliance escalation follows. Control planes become non-negotiable.

2. Cost visibility. Teams track token costs obsessively. Tool calls aren't metered the same way. A gateway that shows "Agent A called database 10,000 times for $50" becomes a cost-saving tool, not just a security layer.

3. Multi-agent coordination. When Agent A invokes Agent B, the control plane needs to prove Agent B was authorized to accept invocations from Agent A. This scales to 10+ agents only with infrastructure support.

By Q4 2026, agent-aware gateways will be table-stakes for any production deployment. Teams running multiple agents without this infrastructure will hit compliance, cost, or coordination walls.

The Bridge

If you're running agents on LiteLLM today, you already have part of the picture:

• Provider routing is handled
• Rate limiting exists
• Basic observability is there

What's missing is agent-specific authorization and audit. That's where LiteLLM Agent Platform fits: a control plane that works above your gateway, managing per-agent identity, tool authorization, and immutable decision logs.

A unified gateway for all agent workloads requires an LLM gateway that handles model routing and failover across providers, plus an MCP gateway that governs every tool connection an AI agent makes.

That's the infrastructure pattern that separates "gateway that routes to LLMs" from "infrastructure for production agents."

---

Questions? Drop them in the comments. I'll respond with specific examples of how to evaluate your gateway for agent workloads, or how LiteLLM Agent Platform integrates with existing routing infrastructure.