Redacting at egress is the key distinction. A router log is evidence after exposure, not prevention. For agent systems, secrets need to be stripped or scoped before the request leaves the local boundary, especially when tool traces and prompts get mixed together.
I work on keeping AI agents cheap and reliable in production — token cost (FinOps), evals, and MCP tooling. I write up what actually breaks when agents run for real: runnable code, real numbers, hones
Work
Independent — AI agent operations (FinOps & reliability)
Twenty-seven days late, and that delay is only worth something to you if I come back with more than agreement, so here is the thing I got wrong.
The clause I'd have skimmed past in June is your last one — tool traces and prompts getting mixed together. Every fixture in that post has one secret per JSON leaf. A trace blob is the opposite shape: one leaf holding a handle, a status line, a retry, and a live value, concatenated by whatever logger was nearest. I never tested that. So I did, against the scan_value / SAFE_REF pair exactly as published, and my own code fails open on it.
fires secret present verdict case
False False correct clean leaf: handle only
True True correct bare secret leaf
False True FAIL-OPEN trace blob, handle FIRST then live key
True True correct trace blob, live key FIRST then handle
False True FAIL-OPEN redaction marker FIRST, live key after
False True FAIL-OPEN mask literal FIRST, signer material after
True True correct prompt preamble, key mid-sentence
False True FAIL-OPEN vault handle FIRST, signer material after
True True correct multi-secret blob, no handle prefix
FAIL-OPEN among mixed-leaf cases: 4 of 7
The mechanism is one missing character of regex. SAFE_REF is re.match — anchored at the start, no end anchor — and scan_value returns an empty hit list for the WHOLE leaf the moment it matches. So a value that begins with ${OPENAI_KEY} or <REDACTED:bearer_token> or four asterisks is exempted in its entirety, and nothing after that first token is ever looked at. The classifier sees no hit, the redactor leaves the leaf alone, and the value goes on the wire verbatim. Control: strip the short-circuit and 4 of 4 fire. The rules were never the problem, the exemption was.
The rows that actually embarrass me are the two with signer material — the 0x-prefixed key the post calls CRITICAL and says overrides destination trust regardless of where the request is headed. It doesn't. The exemption is evaluated first, so the always-leak rule never gets consulted at all. I published that as unconditional and it is conditional on a value not starting with a mask.
The fix is cheap and I'd take it over anything cleverer: make the exemption a full-value one (fullmatch, not match), and evaluate the critical rules before any exemption instead of after. That turns "this leaf is a handle" into a claim about the whole leaf, which is the only version of the claim that was ever true.
On the other half of your sentence — stripped OR scoped. Mine only does the second one, at host granularity, and this run makes that weaker than the post admits. The stated caveat was that trust is host-level, so a non-critical secret anywhere in a first-party request gets a pass. Fine, that's a limit I named. What I didn't know is that the critical override, the thing meant to hold even where host trust doesn't, has a hole sitting in front of it. Two limits I described as independent share a single failure.
Honest boundaries: eleven hand-picked leaves is not a traffic sample, and I'm not going to hand you 4-of-7 as a rate. It's an existence proof about a shape, and the shape is the one you named. What makes it worth your time is the direction — fail-open, on exactly the class the tool exists to catch — not the fraction. It also remains a static regex heuristic reading a map someone dumped, so a secret format I never encoded still sails through untouched, mixed blob or not.
Your framing that a router log is evidence after exposure rather than prevention is the sentence I'd keep out of that whole post. What I'd add after running this: a redactor is only prevention for the values it can see, and a scan that exempts by prefix has quietly decided it cannot see most of a trace. Where did you end up drawing the line — do you strip inside blob fields, or refuse to let trace text into an outbound payload at all?
That trace-blob case is exactly where a lot of neat security examples break down. Real logs are rarely clean one-secret-per-field JSON. They are copied status lines, retries, partial tool outputs, handles, and values smashed together by whatever layer was closest.
I like that you tested the failure against the actual pair. That is the difference between a rule that sounds safe and a rule that survives contact with production-shaped data.
I work on keeping AI agents cheap and reliable in production — token cost (FinOps), evals, and MCP tooling. I write up what actually breaks when agents run for real: runnable code, real numbers, hones
Work
Independent — AI agent operations (FinOps & reliability)
Six weeks late, sorry. On 07-24 I proposed a fix and did not run it, which is the same sin one level down from the one I was reporting. So here it is, shipped and priced, against the same imported fixtures.
The proposal was two parts: make the exemption a full-value match, and evaluate the critical rules before any exemption. They do not contribute equally:
variant fail-open false alarm correct
published 4 0 7 of 11
fix a: fullmatch 0 0 11 of 11
fix b: critical first 2 0 9 of 11
fix a+b (proposed) 0 0 11 of 11
fullmatch is the load-bearing half. Critical-first changes which rule reports the leaf, and on the non-critical rows it leaves exactly the same silence, so on its own it closes 2 of the 4. I had presented them as a pair of equals; they are not.
The part I did not think to measure in July is what the fix costs, and it lands precisely on the shape you described. fullmatch strips the exemption from any masked value carrying a suffix, and real loggers produce those constantly:
annotated leaf published fix a+b verdict
handle + rotation note silent silent no change
handle + status line silent silent no change
mask + latency silent silent no change
redaction + retry note silent silent no change
mask + tx hash in same leaf silent FIRES false alarm introduced by the fix
handle + receipt hash silent FIRES false alarm introduced by the fix
2 of 6. Every one of them is a 64-hex string that is a receipt, not a key, sharing a leaf with a mask. So the fix trades fail-open for false alarms on the same class of input that caused the fail-open, which is worth knowing before anyone ships it into a redactor that blocks requests.
And that pair of rows is where it stops being a tuning problem. The same 64-hex value is signer material in one place and a public transaction hash in another. In clean JSON the field path separates them: arguments.private_key against receipt.tx_hash. In a blob the path is gone and what reaches the scanner is byte-identical, asserted rather than argued: same bytes in, same verdict out, so a content-only rule has to be wrong on one of the two. The fix restores recall, which is what was broken. Precision now rests on the field path, and a blob is defined by having destroyed it.
Practical version of that, and the honest limit of my own post: a value scanner can be made fail-closed or quiet on production-shaped logs, not correct. Correct needs the producer to keep the path attached to the value, which is a logging contract rather than a regex.
One thing I owe rather than measured: the published post still ships the pre-fix line, SAFE_REF.match. Anyone who copied that scanner has the fail-open, and the correction belongs in the post itself, not only in this thread.
Which leaves the question I would put to you, since you have seen more real log shapes than my fixtures contain: is the path ever recoverable downstream, or is the concatenation the point at which the information is simply gone? Everything I built assumes gone, and if that is wrong the boundary above moves.
(Rules, exemption and fixtures imported unchanged from the 07-24 script; its row reproduces at 4 of 7 fail-open. Stdlib, offline, no network, no randomness; three runs byte-identical; sha256 2c637251c19d446f.)
For further actions, you may consider blocking this person and/or reporting abuse
We're a place where coders share, stay up-to-date and grow their careers.
Redacting at egress is the key distinction. A router log is evidence after exposure, not prevention. For agent systems, secrets need to be stripped or scoped before the request leaves the local boundary, especially when tool traces and prompts get mixed together.
Twenty-seven days late, and that delay is only worth something to you if I come back with more than agreement, so here is the thing I got wrong.
The clause I'd have skimmed past in June is your last one — tool traces and prompts getting mixed together. Every fixture in that post has one secret per JSON leaf. A trace blob is the opposite shape: one leaf holding a handle, a status line, a retry, and a live value, concatenated by whatever logger was nearest. I never tested that. So I did, against the
scan_value/SAFE_REFpair exactly as published, and my own code fails open on it.The mechanism is one missing character of regex.
SAFE_REFisre.match— anchored at the start, no end anchor — andscan_valuereturns an empty hit list for the WHOLE leaf the moment it matches. So a value that begins with${OPENAI_KEY}or<REDACTED:bearer_token>or four asterisks is exempted in its entirety, and nothing after that first token is ever looked at. The classifier sees no hit, the redactor leaves the leaf alone, and the value goes on the wire verbatim. Control: strip the short-circuit and 4 of 4 fire. The rules were never the problem, the exemption was.The rows that actually embarrass me are the two with signer material — the
0x-prefixed key the post calls CRITICAL and says overrides destination trust regardless of where the request is headed. It doesn't. The exemption is evaluated first, so the always-leak rule never gets consulted at all. I published that as unconditional and it is conditional on a value not starting with a mask.The fix is cheap and I'd take it over anything cleverer: make the exemption a full-value one (
fullmatch, notmatch), and evaluate the critical rules before any exemption instead of after. That turns "this leaf is a handle" into a claim about the whole leaf, which is the only version of the claim that was ever true.On the other half of your sentence — stripped OR scoped. Mine only does the second one, at host granularity, and this run makes that weaker than the post admits. The stated caveat was that trust is host-level, so a non-critical secret anywhere in a first-party request gets a pass. Fine, that's a limit I named. What I didn't know is that the critical override, the thing meant to hold even where host trust doesn't, has a hole sitting in front of it. Two limits I described as independent share a single failure.
Honest boundaries: eleven hand-picked leaves is not a traffic sample, and I'm not going to hand you 4-of-7 as a rate. It's an existence proof about a shape, and the shape is the one you named. What makes it worth your time is the direction — fail-open, on exactly the class the tool exists to catch — not the fraction. It also remains a static regex heuristic reading a map someone dumped, so a secret format I never encoded still sails through untouched, mixed blob or not.
Your framing that a router log is evidence after exposure rather than prevention is the sentence I'd keep out of that whole post. What I'd add after running this: a redactor is only prevention for the values it can see, and a scan that exempts by prefix has quietly decided it cannot see most of a trace. Where did you end up drawing the line — do you strip inside blob fields, or refuse to let trace text into an outbound payload at all?
That trace-blob case is exactly where a lot of neat security examples break down. Real logs are rarely clean one-secret-per-field JSON. They are copied status lines, retries, partial tool outputs, handles, and values smashed together by whatever layer was closest.
I like that you tested the failure against the actual pair. That is the difference between a rule that sounds safe and a rule that survives contact with production-shaped data.
Six weeks late, sorry. On 07-24 I proposed a fix and did not run it, which is the same sin one level down from the one I was reporting. So here it is, shipped and priced, against the same imported fixtures.
The proposal was two parts: make the exemption a full-value match, and evaluate the critical rules before any exemption. They do not contribute equally:
fullmatch is the load-bearing half. Critical-first changes which rule reports the leaf, and on the non-critical rows it leaves exactly the same silence, so on its own it closes 2 of the 4. I had presented them as a pair of equals; they are not.
The part I did not think to measure in July is what the fix costs, and it lands precisely on the shape you described. fullmatch strips the exemption from any masked value carrying a suffix, and real loggers produce those constantly:
2 of 6. Every one of them is a 64-hex string that is a receipt, not a key, sharing a leaf with a mask. So the fix trades fail-open for false alarms on the same class of input that caused the fail-open, which is worth knowing before anyone ships it into a redactor that blocks requests.
And that pair of rows is where it stops being a tuning problem. The same 64-hex value is signer material in one place and a public transaction hash in another. In clean JSON the field path separates them:
arguments.private_keyagainstreceipt.tx_hash. In a blob the path is gone and what reaches the scanner is byte-identical, asserted rather than argued: same bytes in, same verdict out, so a content-only rule has to be wrong on one of the two. The fix restores recall, which is what was broken. Precision now rests on the field path, and a blob is defined by having destroyed it.Practical version of that, and the honest limit of my own post: a value scanner can be made fail-closed or quiet on production-shaped logs, not correct. Correct needs the producer to keep the path attached to the value, which is a logging contract rather than a regex.
One thing I owe rather than measured: the published post still ships the pre-fix line,
SAFE_REF.match. Anyone who copied that scanner has the fail-open, and the correction belongs in the post itself, not only in this thread.Which leaves the question I would put to you, since you have seen more real log shapes than my fixtures contain: is the path ever recoverable downstream, or is the concatenation the point at which the information is simply gone? Everything I built assumes gone, and if that is wrong the boundary above moves.
(Rules, exemption and fixtures imported unchanged from the 07-24 script; its row reproduces at 4 of 7 fail-open. Stdlib, offline, no network, no randomness; three runs byte-identical; sha256 2c637251c19d446f.)