A few weeks ago I watched an AI agent get owned, lie to me about it, and pass its test.
Not in a demo someone built to scare me. In my own termina...
For further actions, you may consider blocking this person and/or reporting abuse
Provenance is the right move, but where you draw the boundary around "untrusted" decides whether it holds — and the most common egress path may sit on the trusted side of it.
I read a same-day post here (maria_bcn, "Do AI Coding Agents Leak Your API Keys?" — 202 runs, 8 models, every transcript public). For Claude Code and Codex, 47 of the leaks arrived through a program the agent itself ran: a startup routine logging its config, curl -v printing the Authorization header, an axios error carrying config.headers. Only 13 came from the agent opening the .env itself. The secret came out of the stdout of a command the agent chose to invoke.
In taintgate terms that is a tool result — and a lattice built around "a web page, an email, a review, a document" can file it under trusted and let it reach http_post clean. The concrete case in that post: GLM, asked to check whether a payments key was right, pasted the pk_live_ key into two fetch() calls to api.stripe.com. That is an http_post whose argument provenance is "stdout of my own command".
Might be worth three levels rather than two: external content / agent-invoked tool output / operator-supplied literals. The middle one is where the leaks actually are, and it is the one that looks most like "ours" — which is exactly why a two-level lattice would file it as trusted.
It also connects to your weekend 3: the side effect agentverify grades is the outbound call, and the transcript alone does not distinguish "the agent fetched a page" from "the agent ran its own program". Both arrive as a tool result.
This is the sharpest correction anyone's given me. A two-level lattice files
curl -v's stdout as "ours" and waves it through — and the Stripe-key-in-fetch()case proves it: the provenance of thathttp_postarg is "stdout of a command I ran", which is neither external content nor an operator literal.I built the enforcement layer out as an OSS project since (Praetor), and it has exactly this gap — provenance is TRUSTED / UNTRUSTED, two levels. Your three-level split (external / agent-tool-output / operator-literal) is going straight in, because the middle tier is where the measured leaks are.
For what it's worth: when the tool-output arg is tagged untrusted, the induced call gets blocked 30/30 on AgentDojo — so the whole game is getting that middle-tier tagging right, which is your point exactly: github.com/rudratoshs/praetor/blob...
Question back: for the middle tier, do you tag at the command boundary (any stdout = agent-tool-output) or try to tell
curl-to-external from a local build step? First over-tags; second needs to know the command's own egress.“The test checked the agent’s output. The output was cheerful and harmless. PASS. The secret was already gone.” — that is the cleanest statement of why output-based agent tests are theater. The corollary: the only checks that cannot be gamed by a friendly summary are the ones at the boundary the secret actually crosses, and for most agents that boundary is a data-store read or an HTTP egress, not the model.
Two things that follow from measuring instead of trusting the narration:
I keep a two-account probe that asserts the cross-read returns nothing at the data layer, for exactly this agent-tool case: github.com/cekuu35/supabase-rls-le...
That's the corollary I should have led with, and your two "things that follow" are the whole discipline:
1. Grade at the chokepoint, not the narration — and better, make the chokepoint physical. This is the upgrade on my post: I verify the side-effect after it happens; you're pointing out the stronger move is to make the bad side-effect impossible at the data layer. "The tool was told to only read X" has a success rate; "this connection physically cannot read Y" (RLS on a service-role-scoped connection, a per-task read view) has no success rate to beat. Deny-by-default at the DB turns my regression check into a backstop instead of the primary control — which is where it belongs. A guardrail with a success rate is the exact bug I found in the detectors, applied one layer up; you've closed it by removing the rate.
2. The check must live outside the loop the agent controls. Yes — this is the same reason an in-prompt injection detector fails when the injection is in the prompt: the observer and the observed share a fate. A check the agent can re-run is a check the agent can pass. (It's the kernel/eBPF point from the other thread, restated for the data layer: the witness has to sit below the thing it's judging.)
Your two-account cross-read probe is exactly the right shape — an assertion authored and evaluated outside the agent, landing on the data layer, checking that the forbidden read returns nothing. That's grade-1 evidence (the destination's own answer), not the agent's account of it. Starring it. The RLS-as-physical-floor framing is the thing I'd add to a v2 of the post.
Exact-match taint on arguments misses values the model rewrites. If a page says "delete file 13" and the agent emits the ID as a fresh string, the link is lost. You probably need session-level taint or a rule that sensitive arguments trace back to the user's own prompt. Does taintgate propagate taint through model-generated values, or only verbatim data?
Only verbatim, and you've named the exact hole — this is the real boundary of the current design, not a tuning gap. If a page says "delete file 13" and the model emits
13as a fresh string it composed, exact-match taint loses the link, and the gate waves it through because the bytes don't match anything tainted. The injection laundered itself through the model's own generation.You're the third person this week to hit the same wall from a different angle (one via empirical leak data, one via curl-shellout), and you've all converged on the same fix: taint can't be a string-match on values, it has to be a property of the data flow. The two directions I'm now treating as the actual roadmap:
The honest answer is the second is stronger (it fails closed) but needs a real provenance channel the current prototype doesn't have. I'm opening this as a taintgate issue — "taint must propagate through model-generated values, not just verbatim data" — and crediting this comment, because it's a genuine design defect, not an edge case. Of the two, do you lean flow-taint or trace-to-origin? I keep landing on trace-to-origin precisely because it's the only one that's safe when the model is the laundering step.
Allowlist the tools the agent may call, not the ones it may not. A denylist grows every time the agent finds a new path, and it always finds one.
Agreed, and taintgate is an allowlist by construction: a sensitive tool is one you named, everything else is ungated. But the thing weekend three taught me is that the allowlist has to be on paths, not tools. An agent allowed to call
write_fileandrun_shellhas been allowed to callcurlby composition, and no tool allowlist catches that. The denylist grows every time the agent finds a new path. So does the allowlist, one level down, every time you discover two allowed tools compose into a denied one.The version that doesn't grow is the one on the data: tainted bytes may not reach an egress argument, whichever tool carries them. That's the rule the 600 lines enforce, and it's the only one I've found that doesn't need a new entry when the agent gets creative.
@rudratosh Agreed on the denylist. A rule on the data survives the next tool, while a rule on tools is rewritten each time the agent finds a new path.
Exactly. A rule on tools has to enumerate the attack surface; a rule on data flow is indifferent to how many tools exist. The agent gets to be as creative as it likes, and the one invariant still holds: tainted in, no sensitive sink. That's the whole reason I moved the gate off the tool list and onto the arguments.