How frontier models turned privacy from an application concern into an infrastructure problem
Frontier models faithfully execute instructions. They also faithfully move data across system boundaries. That changes privacy from an application concern into an infrastructure concern — and most agent architectures have not caught up.
The problem isn't that models misbehave. The problem is that they behave exactly as instructed, which means every piece of data in the context window is a candidate for transmission to the next tool call. In banking, healthcare, legal, HR, and insurance systems, that is not a theoretical risk. It is a structural property of how agents work.
This post identifies where the risk actually occurs, explains why better models make it worse, and describes the middleware layer that has to exist between the agent and every tool it calls.
Travel booking is the domain we implemented first. The architecture applies everywhere.
Better Models, Bigger Problem
Everyone assumes more capable models reduce risk. The opposite is true for data handling.
GPT-3.5 accidentally protected some secrets because it was inconsistent. It would drop fields, misread nested structures, and fail to extract values cleanly. That unreliability was not a feature, but it occasionally functioned like one.
Frontier models have eliminated that accident. GPT-4o, Claude Sonnet, and Gemini 1.5 Pro will faithfully extract a passport number from a 3,000-token profile and pass it directly to a seat selection tool that has no use for it. They do this because the data is present and the instruction says "book the flight." The model is not doing anything wrong. That is the problem.
There are three failure modes, and all three require the model to be working correctly:
Over-sharing on tool input. An agent receives a full traveler profile to perform a search. The profile contains a passport number. The search tool does not need it. The model passes the whole profile because it was told to call the tool with the traveler's details, and the whole profile is the traveler's details. The passport number now appears in the tool's request log.
Credential leakage in context. A tool returns an API response that includes a session token or an authorization header in the body. The model incorporates it into its context. In subsequent tool calls, the model may reference or repeat that token. This is not a hallucination failure. It is a faithful reproduction failure.
PII in structured outputs. The booking confirmation tool returns a full object containing every passenger detail, payment method, and contact. The agent summarizes it for the user, but the full object is in the context window. If that context is logged, the log contains structured PII at booking confirmation density.
None of these require a malicious actor. They require normal model behavior operating on data that should have been scoped before the model saw it.
The Architecture Gap
Traditional systems have a well-understood control plane:
Application
↓
API Gateway
↓
Service
The gateway handles authentication, rate limiting, and policy enforcement. It is not optional. No one ships a production service by having the application call downstream services directly.
Agentic systems have reproduced the application layer and the service layer but skipped the control plane:
Agent
↓
[nothing]
↓
Tool Runtime
↓
Provider
The agent calls tools directly. Every tool call is a direct transmission of whatever the agent has in context, including data the tool was never intended to receive. The tool returns whatever the provider returns, including data the agent was never intended to see.
The missing layer is a privacy guard that sits at the tool boundary and enforces policy in both directions:
Agent
↓
Privacy Guard
↓
Tool Runtime
↓
Provider
This is the same architectural pattern as an Envoy proxy or an API gateway. The implementation is different because the data is structured JSON rather than HTTP traffic, and the policy is semantic rather than header-based. The structural role is identical: a mandatory chokepoint where policy is enforced before data crosses a boundary.
What the Guard Actually Does
The guard intercepts every tool call in both directions and applies four operations:
Scans the payload for known PII and secret patterns using deterministic detection: field name matching, regex patterns with validation (Luhn for card numbers, format validation for structured types), and domain-specific rules (booking references, passenger name records, loyalty identifiers in travel; account numbers and routing codes in banking; NPI numbers and diagnosis codes in healthcare).
Decides based on configurable policy: allow the data through, redact it with a placeholder, replace it with a deterministic cryptographic token, or block the call entirely. The policy is per-kind and per-severity. Secrets are blocked. High-severity PII is tokenized. Standard PII is redacted.
Audits every inspection, input and output, with a trace ID that correlates the two events. The audit event records what was found and what action was taken, without including the sensitive values or the paths to redacted fields.
Enforces by surfacing violations as structured errors the agent runtime can handle, rather than exceptions that crash the call.
The Implementation
We built this as a dependency-free Node.js package integrated at the invokeTool() boundary in the MCP runtime. One instance at startup, shared across all tool calls:
import { createPrivacyGuard } from "@ucp-travel/mcp-privacy-guard";
import { travelDetectors } from "@ucp-travel/mcp-privacy-guard/travel";
const guard = createPrivacyGuard({
detectors: travelDetectors,
policy: {
tokenizeKey: config.privacyTokenizeKey,
rules: [
{ match: { kind: "secret" }, action: "block" },
{ match: { kind: "pii", severity: "high" }, action: "tokenize" },
{ match: { kind: "pii" }, action: "redact" }
]
},
audit: async (event) => logger.privacy(event)
});
Every tool invocation passes through the guard twice:
export async function invokeTool(runtime, toolName, payload, options = {}) {
const requestId = options.requestId || randomUUID();
// Sanitize before the tool sees the payload
const safePayload = await runGuard(
runtime.guard, payload, "input", toolName, requestId, tenantId
);
// Execute the tool on the sanitized payload
const result = await dispatchToolCall(toolName, safePayload, context);
// Sanitize the response before the agent sees it
return runGuard(
runtime.guard, result, "output", toolName, requestId, tenantId
);
}
The input guard runs before the tool handler receives the payload. The output guard runs before the response reaches the agent's context. Neither the tool nor the agent is modified.
Per-Tool Output Allowlists
Not all PII in a tool response is a problem. A booking confirmation is supposed to contain the passenger name and contact email. Redacting them breaks the confirmation flow.
The solution is per-tool output allowlists that specify which paths are permitted through unredacted:
const OUTPUT_ALLOWLIST_BY_TOOL = {
[MCP_TOOLS.COMPLETE_CHECKOUT]: [
"$.booking.id",
"$.booking.passengers[*].given_name",
"$.booking.passengers[*].family_name",
"$.booking.contact.email"
],
[MCP_TOOLS.GET_BOOKING]: [
"$.booking.id",
"$.booking.passengers[*].given_name",
"$.booking.passengers[*].family_name",
"$.booking.contact.email",
"$.booking.contact.phone_number"
]
};
Everything not on the allowlist for that tool is redacted or tokenized according to the global policy. The search tool never returns PII even if the provider response includes it. The confirmation tool returns exactly the fields the agent needs.
The same pattern applies in other domains. A banking agent's account lookup tool might allowlist account nickname and last-four digits. A healthcare agent's appointment tool might allowlist appointment time and provider name. The policy is domain-specific; the mechanism is not.
Tokenization and Key Rotation
For high-severity PII, redaction is not enough. You lose the ability to correlate data across audit events. Tokenization replaces the value with a deterministic HMAC-based token tied to a key ID:
{ "email": "TOKEN:pii/2026-q3:a3f8c2..." }
The token is deterministic: the same input and key produce the same token, so you can correlate across audit events without storing the original value. The key ID is logged in every event that produces or reads a token. When you rotate keys quarterly, old tokens are not orphaned; you know exactly which key was active when they were created.
Rotation is a two-line environment change. The guard adds under 50ms to any call on payloads below the 200 KB size limit.
The Open Question
Tokenization solves the audit correlation problem. It does not solve the agent reasoning problem.
When the agent receives TOKEN:pii/2026-q3:a3f8c2... in place of an email address, it cannot send a confirmation to that address. If the agent needs to act on a value rather than acknowledge it, tokenization breaks the flow.
The current design handles this by allowlisting output paths the agent needs to surface to the user. It does not handle the case where the agent needs to use a value internally across multiple tool calls without seeing the raw value.
That is a harder problem. Reversible tokenization with a delegated decryption endpoint is one approach. Restricting which tools can receive which token kinds is another. Neither is fully satisfying. We have not solved it yet.
Most engineering articles end by claiming the problem is solved. This one does not, because it is not. The missing middleware layer exists and is deployable today. The question of how an agent reasons about data it has never seen in plaintext is still open, and the answer will shape how agentic systems handle regulated data at scale.
Almin Zolotic is the founder of Zologic and the author of the ucp.travel autonomous booking infrastructure. The architecture described in this article — bidirectional privacy enforcement at the MCP tool invocation boundary, with per-tool allowlists, deterministic tokenization, and domain-specific detection — was developed as part of the ucp.travel infrastructure and is documented here as a reusable pattern for agentic systems handling regulated data.
ucp.travel is autonomous travel booking infrastructure for operators who want to remove the human agent from the booking loop without losing the compliance properties the human agent currently provides.
Top comments (0)