Once you accept that AI agents in production need a governance layer, the natural next question is how that layer is actually built. What are its moving parts, how do they fit together, and how does a request travel through them? This article is a structured walk through the architecture of an agent gateway, using TrueFoundry's design as the running example. The goal is understanding, not a configuration manual: by the end you should be able to reason about why each component exists and how they reinforce one another.
We will look at four internal capabilities agent identity, the shared agent registry, human-in-the-loop approvals, and the skills registry — and then step back to see how the agent gateway sits over the model and tool layers. If you want the product-level overview of these pieces, the agent gateway documentation describes them in one place; here we are interested in the architecture underneath.
The Control Plane and the Request Path
Start with the shape of the system, because it explains many later decisions. An agent gateway is one component of an AI gateway control plane. That control plane also includes an LLM gateway, which governs calls to language models, and an MCP gateway, which governs calls to tools. These three are deliberately separate components with a shared design, because the three kinds of traffic — model calls, tool calls, and agent-to-agent calls — are different enough to warrant distinct handling while needing the same identity and policy backbone.
The deployment is split-plane. A central control plane holds configuration: registered agents, identities, access policies, approval rules, and skill definitions. A regional data plane sits in the actual request path and enforces those policies on live traffic. The separation matters for two practical reasons. First, latency: the data plane can run close to where agents and models live, so the governance hop is cheap. Second, data residency: sensitive request and response data can stay within a region or environment while configuration is managed centrally. The same split is what lets the gateway be deployed as SaaS, self-hosted, or fully air-gapped, and to operate under frameworks such as SOC 2, HIPAA, GDPR, and ITAR.
Agent Identity
Identity is the first layer because everything else depends on it. In this architecture, every agent has a verifiable identity. When an agent makes a call, the gateway resolves two facts: which agent is making the call, and who the agent is acting for. That second fact — the acting-on-behalf-of relationship — is what separates agent identity from a simple API key. A key tells you a credential was used; an identity tells you an accountable actor took an action for a specific principal.
Identity can be backed by the gateway's own identity system or federated to an existing identity provider, so that agent identities live alongside the human and service identities you already manage. Because identity is resolved at the gateway, it is attached consistently to every downstream call. When the agent then reaches a model through the LLM gateway or a tool through the MCP gateway, the same identity travels with the request. That consistency is what makes a coherent audit trail possible: a single action and its cascade of model and tool calls all carry the same accountable actor.
The Shared Agent Registry
The registry is where agents stop being isolated services and become a governed catalog. Every agent is registered, and registered agents can discover and call one another — but access is controlled per agent. Think of it as a directory with permissions baked in. Agent A can look up Agent B and invoke it only if policy allows that interaction.
This design solves a problem that appears quickly in real deployments: undocumented point-to-point integrations. Without a registry, when one agent needs another, someone wires them together directly, and that connection is invisible to everyone else. Multiply this across many agents and you get a tangle no one can audit. With a registry, every agent-to-agent relationship is an explicit, policy-governed, logged interaction. The registry also makes capability reuse discoverable: a team building a new agent can see what already exists rather than rebuilding it.
Human-in-the-loop Approvals
Autonomy is valuable, but some actions should never happen without a person in the loop. The architecture handles this by letting you designate certain tools as sensitive. When an agent attempts to use a sensitive tool, the gateway pauses the action and requires explicit human sign-off before it proceeds. Approvals carry a configurable validity window, so an approval can authorize an action for a bounded period rather than indefinitely.
The important architectural property is where this enforcement lives. The approval gate is enforced at the gateway, not inside the agent's code. That means it holds regardless of which framework the agent was built with, and it cannot be accidentally bypassed by a developer who forgot to add a check. The agent simply cannot complete the sensitive action until the approval exists and is still valid. Below is an illustrative policy that marks a tool as approval-gated with a validity window. This JSON is a generic, clearly-illustrative example to show the shape of such a policy — it is not a literal API schema.
{
"// note": "illustrative policy, not a literal API schema",
"tool": "payments.issue_refund",
"sensitivity": "high",
"approval": {
"required": true,
"approvers": ["role:finance-reviewer"],
"validity_window_minutes": 15
}
}
Read this as a statement of intent: the refund tool is high-sensitivity, it requires approval from someone in the finance-reviewer role, and once approved the authorization is good for fifteen minutes. The agent's reasoning does not change; what changes is that the gateway will not let the call through until the condition is met.
The Skills Registry
The fourth component addresses reuse. A skill is a capability you write once and attach across multiple agents. The skills registry stores these as versioned, access-controlled artifacts. Versioning lets you evolve a skill deliberately: you can improve it in one place and roll the new version out, rather than discovering that ten agents each carry a slightly different copy. Access control lets you decide which agents may use which skills, so a powerful skill is not implicitly available everywhere.
Architecturally, the skills registry pairs naturally with the agent registry. The agent registry governs who can call whom; the skills registry governs what reusable capabilities exist and who may attach them. Together they turn an estate of agents into something closer to a managed library of agents and capabilities.
How Agents Reach Models and Tools
Now connect the layers. An agent does not call a model provider or a tool directly; it calls through the gateway stack. Model calls go through the LLM gateway, tool calls go through the MCP gateway, and both inherit the identity and policy established at the agent layer. From the agent developer's perspective, this is often as simple as pointing an OpenAI-compatible client at the gateway's base URL, so existing code needs little change:
from openai import OpenAI
client = OpenAI(
api_key="your_truefoundry_api_key",
base_url="https://gateway.truefoundry.ai",
)
response = client.chat.completions.create(
model="your-model",
messages=[{"role": "user", "content": "Summarize this ticket."}],
)
The point of this snippet is not the specific call but what it implies: because the client talks to the gateway rather than directly to a provider, every request can be identified, policy-checked, and traced without the agent code having to implement any of that itself. The same principle extends to tool calls routed through the MCP gateway and to agent-to-agent calls routed through the agent registry. Identity and policy are consistent across all three because they are applied at the same layer.
How The Pieces Reinforce Each Other
It is worth seeing the architecture as a whole rather than four features in a list. Identity gives every action an accountable actor. The agent registry uses that identity to decide which agents may call which. Human-in-the-loop uses it to attach approvals to sensitive actions and to record who approved them. The skills registry uses it to control which agents may attach which capabilities. Trace data ties them together, so you can reconstruct a sequence of model, tool, and agent calls and see the identity and policy decisions at each step. Remove identity and the other three lose their anchor; that is why it is the foundation layer.
Key Takeaways
- An agent gateway is one component of an AI gateway control plane that also governs model calls (LLM gateway) and tool calls (MCP gateway), all sharing one identity and policy backbone.
- The split-plane design — central control plane, regional data plane in the request path — keeps the governance hop cheap and lets data stay where it must, enabling SaaS, self-hosted, and air-gapped deployments.
- Agent identity is the foundation: every call resolves which agent acted and on whose behalf, backed by the gateway or your own identity provider, and travels with downstream calls.
- The shared agent registry turns isolated agents into a permissioned catalog; human-in-the-loop gates sensitive tools behind approvals with a configurable validity window, enforced at the gateway so it holds across frameworks.
- Pointing an OpenAI-compatible client at the gateway base URL is enough for agents to inherit identity, policy, and tracing without rewriting their logic.
Top comments (0)