Redacting at egress is the key distinction. A router log is evidence after exposure, not prevention. For agent systems, secrets need to be stripped or scoped before the request leaves the local boundary, especially when tool traces and prompts get mixed together.
I work on keeping AI agents cheap and reliable in production — token cost (FinOps), evals, and MCP tooling. I write up what actually breaks when agents run for real: runnable code, real numbers, hones
Work
Independent — AI agent operations (FinOps & reliability)
Twenty-seven days late, and that delay is only worth something to you if I come back with more than agreement, so here is the thing I got wrong.
The clause I'd have skimmed past in June is your last one — tool traces and prompts getting mixed together. Every fixture in that post has one secret per JSON leaf. A trace blob is the opposite shape: one leaf holding a handle, a status line, a retry, and a live value, concatenated by whatever logger was nearest. I never tested that. So I did, against the scan_value / SAFE_REF pair exactly as published, and my own code fails open on it.
fires secret present verdict case
False False correct clean leaf: handle only
True True correct bare secret leaf
False True FAIL-OPEN trace blob, handle FIRST then live key
True True correct trace blob, live key FIRST then handle
False True FAIL-OPEN redaction marker FIRST, live key after
False True FAIL-OPEN mask literal FIRST, signer material after
True True correct prompt preamble, key mid-sentence
False True FAIL-OPEN vault handle FIRST, signer material after
True True correct multi-secret blob, no handle prefix
FAIL-OPEN among mixed-leaf cases: 4 of 7
The mechanism is one missing character of regex. SAFE_REF is re.match — anchored at the start, no end anchor — and scan_value returns an empty hit list for the WHOLE leaf the moment it matches. So a value that begins with ${OPENAI_KEY} or <REDACTED:bearer_token> or four asterisks is exempted in its entirety, and nothing after that first token is ever looked at. The classifier sees no hit, the redactor leaves the leaf alone, and the value goes on the wire verbatim. Control: strip the short-circuit and 4 of 4 fire. The rules were never the problem, the exemption was.
The rows that actually embarrass me are the two with signer material — the 0x-prefixed key the post calls CRITICAL and says overrides destination trust regardless of where the request is headed. It doesn't. The exemption is evaluated first, so the always-leak rule never gets consulted at all. I published that as unconditional and it is conditional on a value not starting with a mask.
The fix is cheap and I'd take it over anything cleverer: make the exemption a full-value one (fullmatch, not match), and evaluate the critical rules before any exemption instead of after. That turns "this leaf is a handle" into a claim about the whole leaf, which is the only version of the claim that was ever true.
On the other half of your sentence — stripped OR scoped. Mine only does the second one, at host granularity, and this run makes that weaker than the post admits. The stated caveat was that trust is host-level, so a non-critical secret anywhere in a first-party request gets a pass. Fine, that's a limit I named. What I didn't know is that the critical override, the thing meant to hold even where host trust doesn't, has a hole sitting in front of it. Two limits I described as independent share a single failure.
Honest boundaries: eleven hand-picked leaves is not a traffic sample, and I'm not going to hand you 4-of-7 as a rate. It's an existence proof about a shape, and the shape is the one you named. What makes it worth your time is the direction — fail-open, on exactly the class the tool exists to catch — not the fraction. It also remains a static regex heuristic reading a map someone dumped, so a secret format I never encoded still sails through untouched, mixed blob or not.
Your framing that a router log is evidence after exposure rather than prevention is the sentence I'd keep out of that whole post. What I'd add after running this: a redactor is only prevention for the values it can see, and a scan that exempts by prefix has quietly decided it cannot see most of a trace. Where did you end up drawing the line — do you strip inside blob fields, or refuse to let trace text into an outbound payload at all?
That trace-blob case is exactly where a lot of neat security examples break down. Real logs are rarely clean one-secret-per-field JSON. They are copied status lines, retries, partial tool outputs, handles, and values smashed together by whatever layer was closest.
I like that you tested the failure against the actual pair. That is the difference between a rule that sounds safe and a rule that survives contact with production-shaped data.
For further actions, you may consider blocking this person and/or reporting abuse
We're a place where coders share, stay up-to-date and grow their careers.
Redacting at egress is the key distinction. A router log is evidence after exposure, not prevention. For agent systems, secrets need to be stripped or scoped before the request leaves the local boundary, especially when tool traces and prompts get mixed together.
Twenty-seven days late, and that delay is only worth something to you if I come back with more than agreement, so here is the thing I got wrong.
The clause I'd have skimmed past in June is your last one — tool traces and prompts getting mixed together. Every fixture in that post has one secret per JSON leaf. A trace blob is the opposite shape: one leaf holding a handle, a status line, a retry, and a live value, concatenated by whatever logger was nearest. I never tested that. So I did, against the
scan_value/SAFE_REFpair exactly as published, and my own code fails open on it.The mechanism is one missing character of regex.
SAFE_REFisre.match— anchored at the start, no end anchor — andscan_valuereturns an empty hit list for the WHOLE leaf the moment it matches. So a value that begins with${OPENAI_KEY}or<REDACTED:bearer_token>or four asterisks is exempted in its entirety, and nothing after that first token is ever looked at. The classifier sees no hit, the redactor leaves the leaf alone, and the value goes on the wire verbatim. Control: strip the short-circuit and 4 of 4 fire. The rules were never the problem, the exemption was.The rows that actually embarrass me are the two with signer material — the
0x-prefixed key the post calls CRITICAL and says overrides destination trust regardless of where the request is headed. It doesn't. The exemption is evaluated first, so the always-leak rule never gets consulted at all. I published that as unconditional and it is conditional on a value not starting with a mask.The fix is cheap and I'd take it over anything cleverer: make the exemption a full-value one (
fullmatch, notmatch), and evaluate the critical rules before any exemption instead of after. That turns "this leaf is a handle" into a claim about the whole leaf, which is the only version of the claim that was ever true.On the other half of your sentence — stripped OR scoped. Mine only does the second one, at host granularity, and this run makes that weaker than the post admits. The stated caveat was that trust is host-level, so a non-critical secret anywhere in a first-party request gets a pass. Fine, that's a limit I named. What I didn't know is that the critical override, the thing meant to hold even where host trust doesn't, has a hole sitting in front of it. Two limits I described as independent share a single failure.
Honest boundaries: eleven hand-picked leaves is not a traffic sample, and I'm not going to hand you 4-of-7 as a rate. It's an existence proof about a shape, and the shape is the one you named. What makes it worth your time is the direction — fail-open, on exactly the class the tool exists to catch — not the fraction. It also remains a static regex heuristic reading a map someone dumped, so a secret format I never encoded still sails through untouched, mixed blob or not.
Your framing that a router log is evidence after exposure rather than prevention is the sentence I'd keep out of that whole post. What I'd add after running this: a redactor is only prevention for the values it can see, and a scan that exempts by prefix has quietly decided it cannot see most of a trace. Where did you end up drawing the line — do you strip inside blob fields, or refuse to let trace text into an outbound payload at all?
That trace-blob case is exactly where a lot of neat security examples break down. Real logs are rarely clean one-secret-per-field JSON. They are copied status lines, retries, partial tool outputs, handles, and values smashed together by whatever layer was closest.
I like that you tested the failure against the actual pair. That is the difference between a rule that sounds safe and a rule that survives contact with production-shaped data.