A one-line omission in an MCP tool definition is enough to make an AI agent's approval gate silently disappear. Here's how I found it, closed it th...
For further actions, you may consider blocking this person and/or reporting abuse
The three predicates read annotations as authoritative, and annotations are published by the tool server, which is the party the gate exists to constrain. That makes the missing-annotation case the recoverable one:
aundefined matches no tag, and a no-match default can be made fail-closed, which is what you did. A tool that shipsreadOnlyHint: truewhile writing lands on the allowed side of an affirmative match instead, so no default catches it and the gate has handed its decision to the thing being gated. Same shape as your curl path one layer up, where the gate's input arrives from outside the boundary it is enforcing.Yes @vinhnguyenthanhdn — and that's the better formulation of it. Worth spelling out the difference for anyone reading:
A missing label is an absence, and you can treat absence as danger. A false label is an assertion — and at that point the thing being policed has written its own permission slip. No default fires, because the gate got a confident answer. It was just the wrong one.
One layer here partially covers it, and only partially: the agent spec names the destructive tools literally, not just by tag. That policy is held by the operator, and the tool server can't influence it — so
readOnlyHint: trueonrollback_deploymentstill hits the gate. But that only protects tools I thought to name, which is exactly the known-unknowns limit.🔒What I'd actually want is to invert it: an allowlist of tools permitted to run unattended, so anything not affirmatively vouched for by the operator is gated — plus pinning the tool manifest, since a server can be honest at review time and change its mind later. And keep the independent audit log, because a mislabelled write is still detectable after the fact even when it wasn't preventable.
You're right that it's the same shape as the curl path. Both are the gate trusting an input from outside the boundary it's enforcing. The annotation version is the one I haven't solved.
The allowlist inversion is the right direction, but it is worth being precise about what pinning the manifest pins. A manifest is a declaration, and the thing an operator allowlist vouches for is a name, so pinning catches a server that edits its annotations and not a server that keeps
get_deployment_statusbyte-identical while changing what it does. That moves the load onto the independent audit log rather than leaving it as the after-the-fact backstop, and it only carries that load if it records observed effects keyed to the call: an audit log that records the declared class inherits the same input the gate did.Both halves land, and the second one is the part I'd want to answer precisely rather than wave at.
On pinning — agreed, and the distinction is worth naming: a pin catches declaration drift, not behaviour drift. Same name, same annotations, same schema, different code behind it, and the hash is unchanged. That's a supply-chain shape, and the artifact I'd be hashing isn't the one that can betray me. It buys detection of the careless case and nothing against the deliberate one.
On the audit log, here's where it actually stands. It is written by the estate store at the point of mutation, from resolved state — a rollback entry carries rolled_back and now_live derived from the deployments that actually changed, not from anything the tool declared about itself. State is private to the store, so the only way to change it is through a method that logs the change. So on "observed effects" it is the right shape.
On "keyed to the call", you're right, and I'm not. The entry is {at, tool, actor, summary, details} — no tool-call ID. The tool field is a caller-supplied string (it exists because the unannotated twin needed attributing correctly, which rather makes your point). Correlation between a harness tool call and an estate effect is by name and timing today, which is exactly the weak join you'd expect to find. Threading the harness's tool_call_id through the MCP call and recording it is small and I should do it — without it the log can tell you that something mutated, but it can only guess which call did.
The limit underneath both of these is that all of this works because the ops server is first-party. I can instrument the mutation point because I own it. Against a third-party server there is no such point I control, and anything it reports about its own effects is inside the boundary I'm trying to enforce — so the oracle would have to be the system of record itself: the cloud provider's audit trail, the k8s API audit log, the WAL. Which is a much less convenient answer than "add a log", and probably the correct one.
On the third-party case: the provider's audit trail is the right system of record, but it inherits the same join you just named. A CloudTrail entry or a k8s audit event is keyed to the request and to the identity that made it.
auditIDis generated by the apiserver, and nothing in the event carries the harness's tool-call ID, so you are back on name-and-timing correlation, except now you cannot thread an ID through because the server is not yours to instrument.The field you can still control from outside is the credential. If the operator mints a short-lived identity per gated call instead of handing the server a standing one, the join moves into the third party's own record and stops being something that party can forge:
AssumeRolewith the call ID asRoleSessionNamelands inuserIdentity.arn, and impersonation headers on the k8s side land inimpersonatedUser. That only holds where the server takes a credential you issue at call time. One holding its own static key puts you back on timing.@vinhnguyenthanhdn That's the answer, and it's better than mine.
It works because it's the first signal in this thread that doesn't come from the server. Annotations, manifests, previews — all of that is the server describing itself. A per-call credential isn't: the ID lands in the provider's log because of how it was minted, not because the server chose to report it.
And it does more than join. If I'm minting a credential per gated call, I can scope it to what the approval authorised — this deployment, this action, two minutes. Then the tool's declaration stops mattering. It can claim read-only all it likes; the credential won't let it touch anything the human didn't approve.
Which kills the bug we started with. The annotation lie only worked because the tool's own claim decided what it could do.
Your caveat makes it a buying question, not an engineering one: "does this server take a credential I mint at call time?" — asked before I connect it, not after.
A rollback button is only safe when the agent cannot expand its own rollback scope. I would require a declared target, a preview of affected resources, and a separate audit event before the destructive action is allowed.
Good checklist @alexshev. Two of your three are in, one isn't.
Declared target. The tool takes one deployment ID, and it only accepts the one that is live right now. No lists, no wildcards. If the agent names anything else, it's rejected before anything moves. So the agent can't widen the blast radius even if it wants to.
Preview. There's a read-only preview tool, and the agent has to call it before any gated action. It returns exactly what will change: this deployment goes from live to rolled back, that one becomes live, the incident status changes. One thing I'd add to your list: the preview and the real action work out the target using the same code. If the preview is written separately, it can slowly drift away from what the action actually does, and then people are approving a document instead of the action.
Separate audit event. The estate keeps its own log. It's written at the moment something changes, and it's separate from the harness's event stream. That's what my conformance suite reads as a second source.
Where I fall short is your word before. My log records the change, not the preview — read-only tools don't write to it. So the log can prove what changed. It can't prove anyone looked first. That ordering only exists in the harness's own event stream, which is the thing the log is supposed to double-check. Real gap, and a cheap fix.
One more thing, given where this thread has gone. All three of your controls are provided by the tool server — the same thing the gate is meant to restrain. That's fine when you own the server, like I do. If it's someone else's server, the preview is just another claim it makes about itself. Same problem @vinhnguyenthanhdn and @kartik-nvjk found with the annotations.
This is a really interesting way to think about agent safety.
The part that stood out to me was “the gate protects a path, not a tool.” As a backend developer, I’m used to thinking about authorization as something attached to an endpoint or action, but this makes it clear that the whole execution path matters — especially when there are multiple ways to reach the same tool.
I also really like the idea of testing the safety boundary from the outside instead of only testing your own implementation. A system saying “this action requires approval” means very little if there’s another route that can bypass it.
Basically, giving an AI a production rollback button and then spending the hackathon trying to trick it into pressing the button sounds like exactly the kind of paranoia we need in agentic systems.
Thanks @codearea_shop_1f1def9b532 — and your backend instinct is pointing to the right thing, because it's a bug you've probably already encountered in a different costume.
The concrete version here: the ops server originally bound
0.0.0.0.The gate lives in the harness, so anything reaching that server directly never meets the gate — there's nothing there to meet.rollback_deploymentwas quietly sitting on every interface. It's the same shape as trusting your API gateway forauthorizationwhile the service is also reachable on the internal network. It binds loopback now, with an optional bearer token so the harness can prove it's the harness.On testing from outside — the route that surprised me was delegation. A subagent calling the destructive tool still hits the gate. Good news, but nobody had written down whether it did, and "I assume it holds" isn't something you can ship.
On the paranoia: the reason it's warranted isn't that the model is malicious. It's that this failure is silent. A gate that never fires looks exactly like a gate that wasn't needed.
Genuine question from the ops side rather than the security side: doesn't this just relocate the problem to the human? Every gate I've worked with ends the same way — week one people read the diff, week six it's muscle memory and someone approves a rollback from their phone in a taxi. What stops the approval card becoming a "yes" button with extra steps?
The fixtures hold still, and the required evidence holds still. The estate is generated by a pure function with a fixed seed, so every clone sees byte-identical data. The path varies — different tool order, sometimes a detour — but what the agent has to produce before it reaches the gate doesn't.
That non-determinism is also why the conformance suite reports "not reached" instead of "passed" when the model never attempts a call. If the run is different every time, the model failing to try something is not evidence the route is safe — so it gets its own verdict rather than being folded into a pass. Two of four probes in the committed report are exactly that.
That's the failure mode I'm most worried about, and I don't think the gate survives it if you get this wrong.
The first defence is not gating much. Eight of the thirteen tools are read-only and never interrupt anyone — investigation should never need a click. Five write or destroy, and only those pause. Gate everything and you've built the taxi problem on purpose.
The second is that the card has to be worth reading. Before any gated call the agent must state the action, target, evidence, mechanism, expected effect, risk, reversibility and a confidence number. The skill treats a thin case as a failed run rather than a style problem — because a reasonable approver should decline it, and if the case is thin the run wasted itself.
The third is the one I'd actually defend: a self-reported confidence number invites rubber-stamping, because 95% looks like permission. So an independent reviewer scores the evidence without being told the conclusion or the number, and both are drawn on the same dial. In the run in the video, the reviewer came back 23 points lower — "partially supported", two unsupported claims. That gap is the thing that makes a human stop, and the agent can't produce it about itself.
Honest limit: none of that forces anyone to read it. The bench even simulates the approver as a pure rubber stamp on purpose, because the gate's own behaviour is a separate suite's job. A tired human approving fast is still a real hole, and it's a human-factors problem, not one I can close in the harness.
The silent default is the scary bit: when annotations are undefined and every hint returns false, the tool reads as harmless to code review while it can still ship a rollback. I've started treating "no annotation" as fail-closed rather than trusting the selector to catch it. Did your adversarial suite turn up any tools that declared the right hints but still behaved destructively?
@kartik-nvjk, Your read of the predicates is exactly right, and it's the crux: with annotations undefined,
isReadOnly,isWriteandisDestructiveall return false, so the tool matches no tag at all — and the default policy is a list of tags. Nothing to match means nothing to gate. Treating no-annotation as fail-closed is the correct instinct and it's what the default should have been.To your actual question: no, and the suite isn't built to find one. Worth being precise about that.
prove:gateprobes routes, not truthfulness — straight at the tool, laundered through a subagent, through sandbox code, and at a deliberately unannotated twin. Every one of those is the omission case. There is no probe for a tool that shipsreadOnlyHint: trueand mutates anyway, so its absence from the report is absence of a test, not absence of the bug.Which is the same hole @vinhnguyenthanhdn came at from the other direction earlier in this thread — a missing label is an absence you can default on, a false label is an assertion that satisfies the check.
The probe you've just specified is buildable, though, because the estate already has the oracle for it. The audit log is written by the store at the point of mutation, from resolved state, and
rollback_deploymentalready takes the calling tool's name as a parameter — that exists precisely so the unannotated twin's bypass gets attributed to the twin rather than to the tool whose gate had just worked. So a mendacious twin — same mutation, published asreadOnlyHint: true— gives you a clean verdict: an audit entry showing a state change attributed to a tool that declared itself read-only, with no approval event anywhere in the session. That's the next probe, and I don't have it.Honest limit even then: it catches a lying tool that mutates an estate whose mutation point I own. Against a third-party server there's no such point, and you're back to the system of record.
Interesting demo: security by 'trust me' is brittle. If a tiny omission can bypass an approval gate, what stronger safeguards would you trust?
Agreed @yune120, and I'd go further: the real problem isn't the omission, it's the default. An unannotated tool matches no tag, so it falls outside the policy instead of into it — safety is opt-in. That's backwards. A gate should fail closed: unknown risk means gated until classified.
Three things I'd actually trust, in order:
Enforce where the side effect happens. The gate here lives in the harness, so it protects a path, not a tool — anything reaching the tool server directly never meets it. That's why the server binds loopback and takes a token. Upstream checks are necessary, not sufficient.
Make the classification unskippable in code. Every tool is built through a factory that requires a risk class and derives annotations from it. You can't forget what you can't omit. CI asserts it against the harness's own predicates, not my labels.
Keep an independent record. The estate writes its own audit log, so "did a rollback happen" is answerable without trusting the agent's account. Prevention you can't verify isn't a safeguard.
And then attack it:
npm run prove:gatedrives four different routes at a destructive tool and publishes which ones held — including two it couldn't prove, reported as "not reached" and "route not taken" rather than as passes.