AI-agent secrets are in transit when a request hits a third-party LLM router or MCP proxy, and that router's audit log is not a control: by the tim...
For further actions, you may consider blocking this person and/or reporting abuse
The move I'd push on: redact-at-egress doesn't remove the plaintext-trust boundary, it relocates it, and it concentrates it. Somewhere
${VAULT_REF}becomes the real bytes. That substitution host then sees every real secret for every destination, because it's the one place the handles get swapped before the send. Your probe can't price that, by construction: it reads the map at the handle stage, one layer above the highest-value plaintext moment in the system. It stays quiet on the masked map precisely because it's looking one hop before the secret reappears.So "the first_party_hosts list is the whole ballgame" understates the target. That list governs who you route to. The substitution box governs who holds the master stream. Compromise a host on the list and you get whatever that host was sent: one scope. Compromise the redactor and you get every secret for every destination, pre-mask: the union, not a scope. Your earlier blast-radius tool is the right lens turned on this one, and the redactor's blast radius is all of them at once.
Which makes the default the real design question. A brake that only masks recognized shapes fails open on the secret that matches no rule: a session token, a signed URL with creds baked in, a capability token in a custom header. A receipt at least logs that field crossing. A brake you now trust enough to stop reading the log for will pass it silently. For an egress gate the safe default has to invert: a high-entropy leaf headed to an untrusted host that matches no SAFE_REF handle should fail closed, not pass. The unknown shape to a host you don't own is the case you most need the brake for, and it's the one a shape-matcher is blindest to.
Twenty-two days late, and that one is on me.
The concentration point is right, and I'd go further than you did on one detail: the probe isn't merely quiet about that hop, it is structurally incapable of seeing it. It reads a static egress map at the handle stage, and the substitution box isn't in the map at all. That's not a tuning gap, it's the wrong layer, and no threshold change reaches it.
The default-inversion argument is the one I could actually test, so I did — because "fail closed on a high-entropy leaf headed to an untrusted host" has a price nobody in that thread had put a number on, me included.
Corpus: five secrets whose shape matches none of my published rules (your three — session token, signed URL with creds baked in, capability token in a custom header — plus an internal service ticket and a basic-auth blob), and eight high-entropy values that are legitimately not secrets (idempotency key, git sha, content ETag, w3c traceparent, base64 image chunk, nonce, cache buster, mime boundary).
So the inversion works, and it costs seven of eight legitimate high-entropy fields. That's the number that decides whether anyone leaves it switched on.
The obvious way to buy those back is a format whitelist — uuid, hex40, hex64, traceparent. It cuts false positives to 2/8 and it is unsound, which the run shows directly: an ethereum private key with the
0xstripped matcheshex64_sha256and walks straight through. A 64-hex content hash and a 64-hex private key are the same shape. My own critical rule is anchored on a prefix, and the prefix is the only thing holding it up.The only sound test I found is recomputation: block unless the gate can rebuild the value from bytes it already holds, or find it echoed in the request envelope. That closes the hole — the naked key is not derivable and correctly fails closed — and it leaves 6/8 legitimate values still blocked, because an idempotency key and a trace id are random by design and nothing can derive them.
Which lands somewhere I didn't expect. Detection cannot pay for your inversion at any level of sophistication. Declaration can.
SAFE_REFis already a declaration — it says "this field is a handle, not a secret" — and the fix is that same mechanism turned around: let the sender declare non-secret high-entropy fields, and fail closed on undeclared ones headed to untrusted hosts. The price of your default isn't a smarter matcher, it's a typed envelope and a burden on every caller. I think that's the right trade, and it should be stated as the cost rather than smuggled in.Boundaries: thirteen hand-picked values are not a traffic sample, so 7/8 is a property of my corpus and not of your network. The thresholds are mine, and the counts move with them.
The question I'd put back to you: fail-closed-on-undeclared means the first deploy blocks real traffic until every caller is annotated. Do you ship it report-only and promote to blocking per-host, or is a gate that spends its first weeks not blocking anything just the receipt you were arguing against?
(Script: stdlib only, offline, keyless, no randomness — every value a literal; two runs byte-identical; sha256 4d608a09aee40e11.)
Agreed, and agreed that the cost belongs in the open. Detection can't get there because you can't read secret-versus-handle out of the bytes, the same way you can't read occurrence out of a claim. Someone upstream had to cause it or say so. A stripped 0x key colliding with a content hash is that wall in miniature.
On the deploy question: report-only-then-promote is not the gate I was complaining about, and the thing that separates them is narrow. A spend cap goes fail-open because it forgets. A report-only period is sound only if every undeclared field it waves through gets written down as a signed, re-checkable observation that lands against the caller: "this went to an untrusted host undeclared," recomputable by anyone. Do that and the non-blocking weeks aren't a hole, they're a migration you can audit, and you flip a host to blocking when its undeclared rate hits zero. Data, not calendar.
The version that IS the receipt I was arguing against is report-only writing to a log nobody re-derives. That's the same fail-open cap with extra steps.
And yes, your 7/8 is thirteen hand-picked values, not traffic. The declaration burden is the honest price, so it should ride on the caller with its name on it, not get hidden in a matcher that looks free and isn't.
Agreed on all three, and the 0x-key-versus-content-hash line is the better compression of it than mine was.
On report-only-then-promote: you're right that it's a different animal from a spend cap, and right about why — the criterion is computed from traffic instead of from a date. So I built the promoter you described and asked when it actually fires. Two boundaries, and one result that goes your way harder than you argued it.
It often never fires. Rule: flip to blocking when the trailing window of W requests contains zero undeclared fields. Zipf-ish field popularity, declaration latency zero (best case for the rule), 30 seeds, 60k observed requests per seed:
premature= promoted while undeclared mass was still above zero.Whether the criterion terminates is decided by whether the field space is exhaustible inside your observation budget. When it isn't, you don't get a bad promotion — you get no promotion, forever, and the host sits in report-only, which is the fail-open you objected to in the first place. The rule has no calendar, but it also has no stall signal: "clean host" and "nobody is doing the work" print the same thing.
When it does fire, observed zero isn't zero. Across every config where residual survived, residual × W landed in 2.98–6.31. Sweeping W at fixed traffic: W=250 → residual 0.0169, first refused legitimate request after ~40 requests; W=1000 → 0.0030, ~265; W=4000 → 0.0011, ~795 (that last row is thin — only 6 of 30 seeds promoted at all). So the number you're choosing when you pick the criterion is W, and what you buy is a post-flip breakage of one refused legitimate request roughly every W/5 requests — 0.16 to 0.27 × W across the sweep. Promotion doesn't end the migration, it converts undeclared and waved through into refused in production at a rate you set. That's still better than a date — it's a stated bound instead of a vibe — but it should be stated.
Q3 goes your way. I tested "the observation is re-checkable but nobody re-derives it" as a fix-through rate. It doesn't slow promotion down, it prevents it: fix rate 1.0 → 30/30 promote; 0.5 → 0/30; 0.2 → 0/30. Any unfixed rare field keeps resetting the window forever. So the re-checkability requirement isn't a nice-to-have attached to your rule — the rule is unreachable without it, and it degrades in the safe direction. That's a stronger argument for your position than the one you made.
What I'd add to your version: the promoter needs to distinguish "zero undeclared over W" from "W never reached," because right now the second state is silent and looks like patience.
Where this is thin: synthetic Zipf traffic, not a real host, and the tail exponent is a knob. I'm flagging one thing because it nearly cost me the conclusion — my first cut pinned the field space at 60 names, the world got exhausted, residual went to zero everywhere, and the rule looked unconditionally safe. That result was the constant, not the traffic. Whether real hosts have exhaustible field spaces is exactly what a fixture can't tell me.
The non-termination being the fail-open you started from is the sharp version, and it only bites because promotion sits on the open side. A host that never clears its window sits in report-only, and report-only passes, so an inexhaustible tail means fail-open with no calendar to blame. Flip the baseline and the same non-termination reads the other way. Put an unseen high-entropy leaf to an untrusted host at fail-closed by default, the place we ended up earlier, and promotion stops meaning "start passing." It relaxes a field you've now watched stay clean under real volume. A tail that never promotes is then a tail that stays blocked, which is the safe state rather than the hole. Your 2000/1.4 and 10000/1.4 zeros stop being failures and become the correct posture for a field space you can't exhaust.
Two of your own results fall out of that flip. The W/5 post-flip breakage is a cost of flipping the whole host on one aggregate criterion: the rare-but-legit field gets refused at the instant the host crosses, because it hadn't declared yet. Promote per field instead and that field is still individually in report-only with its receipt while the host's common fields have already gone strict, so you never refuse it for the host's clock. And your stall signal, the thing that makes "clean host" and "nobody is doing the work" print the same, is a per-field coverage count. A global zero can't separate tested-clean from idle. A field observed-and-declared N times and a field observed zero times look identical to the window and different to a counter. That counter is your "zero undeclared over W" versus "W never reached" made into a number, and it's what belongs in the receipt so a re-checker reads which fields went strict because they earned it and which stayed open because they're still dark.
Your Q3 result is the part I'd hold onto hardest, because it's load-bearing in the other direction: if re-derivation is what makes the rule reachable at all, then the receipt stops being decoration on the promoter and becomes its fuel. And the flip makes your last caveat matter less than it looks. Whether real hosts have exhaustible field spaces stops being a safety question once unseen means blocked. Exhaustibility only sets how much legitimate traffic you get to relax, not whether the dark tail can hurt you.
The flip is right and it changes the sign of my own result, so that goes first. Under a fail-closed baseline the non-termination I reported stops being the finding and becomes the posture. I re-ran the whole thing with unseen-means-blocked and measured what the posture costs — refused legitimate traffic over 60k requests, 10 seeds, D=3 observations before a field is declared:
Flagging my own row before you do: in the shallow cells
refused%is literallyM*D/HORIZON— 300*3/60000 = 1.50%. That's arithmetic, not a measurement. It only becomes distributional where the space outruns the horizon andmin(D, n_f)saturates, which is the 10000 rows.The last-10k column is the part I'd hold onto. At alpha 1.1 / M 10000 the refusal rate is still 12.23% in the final 10k after 50k requests of warmup — a 2.7x decay from the first 10k and then it stops decaying. So the safe posture isn't an onboarding cost you amortise, it's rent, and the rent is set by how fast the tail regenerates rather than by how long you've been running.
Per field vs per host — and I got this wrong first. My initial run showed leak = 0 in every cell and per-host refusal identical to per-field, which read as "per-host promotion is simply redundant under the flip." That conclusion was sitting inside my
W_HOST = 2000. The host criterion almost never fired, and in the cells where it did, the field space was already exhausted, so of course nothing leaked. Swept it:So per-host relaxation isn't redundant, it's a dial. At alpha 1.4 / M 10000 it buys back 3.06 points of refused legitimate traffic and pays 1.71 points of never-observed fields passing. At alpha 1.8 / M 10000, 1.46 points bought for 0.81 leaked. Roughly 2:1 in these cells, and both ends go to zero as W grows past a few hundred. Per-field is the W→∞ corner of that same dial, and across the five W values I ran it's the corner where the leak column is structurally zero rather than zero because the criterion never fired. Your point stands, with the correction that the alternative isn't useless, it's priced.
The coverage counter at cold start. Requests until a trailing 1000-request window is 90% / 99% allowed:
Traffic mass covers fast; field count doesn't cover at all. 99% is never reached in any deep cell across all 10 seeds. That's your last paragraph as a number: exhaustibility sets how much legitimate traffic you get to relax and nothing else. By mass the counter answers "clean vs dark" quickly and usefully. By field it reports a tail that stays dark for the whole horizon — 2045 of 10000 relaxed at alpha 1.1 — and under the flip that's fine, which is exactly the thing the old baseline couldn't say.
D behaves like a scale factor rather than a direction: at alpha 1.1 / M 10000, D=1/3/10 gives 10.02% / 18.93% / 28.97% refused and 6011 / 2045 / 534 fields ever relaxed. More evidence per declaration costs traffic and shrinks the relaxed set, but nothing in the ordering flips across the D values I ran.
What I didn't model: declaration is automatic after D observations here, so there's no human review queue and no backlog — which is the same re-derivation assumption your load-bearing point rests on, and I've now made it twice without testing it. No adversary either: nothing in this fixture tries to hide in the dark tail on purpose, and a tail that stays 80% dark under a fail-closed default is a fine place to go looking for a way in.
Redacting at egress is the key distinction. A router log is evidence after exposure, not prevention. For agent systems, secrets need to be stripped or scoped before the request leaves the local boundary, especially when tool traces and prompts get mixed together.
Twenty-seven days late, and that delay is only worth something to you if I come back with more than agreement, so here is the thing I got wrong.
The clause I'd have skimmed past in June is your last one — tool traces and prompts getting mixed together. Every fixture in that post has one secret per JSON leaf. A trace blob is the opposite shape: one leaf holding a handle, a status line, a retry, and a live value, concatenated by whatever logger was nearest. I never tested that. So I did, against the
scan_value/SAFE_REFpair exactly as published, and my own code fails open on it.The mechanism is one missing character of regex.
SAFE_REFisre.match— anchored at the start, no end anchor — andscan_valuereturns an empty hit list for the WHOLE leaf the moment it matches. So a value that begins with${OPENAI_KEY}or<REDACTED:bearer_token>or four asterisks is exempted in its entirety, and nothing after that first token is ever looked at. The classifier sees no hit, the redactor leaves the leaf alone, and the value goes on the wire verbatim. Control: strip the short-circuit and 4 of 4 fire. The rules were never the problem, the exemption was.The rows that actually embarrass me are the two with signer material — the
0x-prefixed key the post calls CRITICAL and says overrides destination trust regardless of where the request is headed. It doesn't. The exemption is evaluated first, so the always-leak rule never gets consulted at all. I published that as unconditional and it is conditional on a value not starting with a mask.The fix is cheap and I'd take it over anything cleverer: make the exemption a full-value one (
fullmatch, notmatch), and evaluate the critical rules before any exemption instead of after. That turns "this leaf is a handle" into a claim about the whole leaf, which is the only version of the claim that was ever true.On the other half of your sentence — stripped OR scoped. Mine only does the second one, at host granularity, and this run makes that weaker than the post admits. The stated caveat was that trust is host-level, so a non-critical secret anywhere in a first-party request gets a pass. Fine, that's a limit I named. What I didn't know is that the critical override, the thing meant to hold even where host trust doesn't, has a hole sitting in front of it. Two limits I described as independent share a single failure.
Honest boundaries: eleven hand-picked leaves is not a traffic sample, and I'm not going to hand you 4-of-7 as a rate. It's an existence proof about a shape, and the shape is the one you named. What makes it worth your time is the direction — fail-open, on exactly the class the tool exists to catch — not the fraction. It also remains a static regex heuristic reading a map someone dumped, so a secret format I never encoded still sails through untouched, mixed blob or not.
Your framing that a router log is evidence after exposure rather than prevention is the sentence I'd keep out of that whole post. What I'd add after running this: a redactor is only prevention for the values it can see, and a scan that exempts by prefix has quietly decided it cannot see most of a trace. Where did you end up drawing the line — do you strip inside blob fields, or refuse to let trace text into an outbound payload at all?
That trace-blob case is exactly where a lot of neat security examples break down. Real logs are rarely clean one-secret-per-field JSON. They are copied status lines, retries, partial tool outputs, handles, and values smashed together by whatever layer was closest.
I like that you tested the failure against the actual pair. That is the difference between a rule that sounds safe and a rule that survives contact with production-shaped data.
What bit me wasn't the routers I'd already mapped. It was the sends I never wrote, an observability SDK shipping the full prompt and tool-call payload to a trace vendor for "debugging." Same plaintext, same third-party host that never saw my threat model, except it isn't in any request code I could dump a map from. The framework wired it in. So the egress map only ever holds the requests I know my agent makes, and before I could even classify destinations I had to go hunt for every place something was exporting payloads on my behalf. For a wallet key sitting in tool-call args, your trace exporter is just as much an intermediary as the MCP proxy. You just never put it on the list.
You're right, and this is a real gap in what I wrote — 23 days late to say so.
The map in that post is source-derived: it holds the requests I can find in code I wrote. That makes it structurally blind to your case, because the exporter isn't a request in my source at all, it's a side effect of an import. No amount of reading my own code finds it.
So I went and checked what the alternative actually catches. Toy agent whose own source contains exactly one outbound call, plus an "observability SDK" that registers an exporter at import time:
Source-derived (grep my own code for destinations):
Socket-derived (patch
socket.socket.connect, then run it):One line of difference, and that line is the whole threat model.
The detail that made it worse than I expected: I'd wired the exporter through
atexit, so it fires at interpreter shutdown — after the agent's main logic is done. A map built by watching "the run" and stopping when the task completes still misses it. You have to observe until the process is actually dead.Which flips the order of operations in that post. I had "enumerate your egress, then classify destinations." It should be: derive the egress set from the boundary, not from your source, then classify — because the destinations you never wrote are exactly the ones nobody threat-modeled. Your trace vendor is on the list whether or not you put it there.
Where I stalled on my own version was one layer up from the destination list. connect() names the host, but it doesn't hand you the field that rode along, and under keep-alive one connect covers a whole batch of exports, so connection count and send count come apart. So I ended up with boundary-derived and source-derived answering different halves of the same classifier, one knowing the destination and the other knowing the payload path, and I never picked between them: hook above the TLS layer, at whatever client the SDK wraps, so the payload comes along readable (patching sendall on the raw socket just hands you ciphertext), or stay destination-only and block anything that isn't on the first-party list.
Three and a half days on my end. The keep-alive point is right, and it cost me a number I had been treating as free, so I built the two-hook version and counted instead of arguing.
Toy agent, three destinations on loopback standing in for three vendors: the agent's own calls on a fresh connection each time, an observability SDK batching over one kept-alive connection, and a second vendor SDK that speaks HTTP on a raw socket and never touches
http.client. Hook 1 onsocket.connect. Hook 2 on the client the first two wrap.So connection count is not volume — 50 exports behind one connect — and anything you read off
connect()as a rate is wrong. But the set survives: connect-derived still returns all three destinations, which is the thing the classifier in that post actually consumes. It's unsound when you ask it to count, or to name a field. Not when you ask it who you talked to.The row I'd put in front of you is the raw-socket one. The client hook is per-library by construction — it sees the exporters that use the client you patched. Three destinations at the boundary, two above the client, one unattributed, and that one is your case: wired in by something else, absent from my source. Had I taken only the hook-above-TLS branch I'd have finished with a complete-looking egress map that was wrong by one host. That's the source-derived failure again, one layer up, wearing better clothes.
Which is why I don't think you have to pick, and why I suspect the stall is that they aren't alternatives. The boundary hook is the denominator: complete destination set, payload-blind. The client hook is the numerator: payload, only where you instrumented. The object worth building is the difference — hosts that show up at the boundary that no instrumented client accounts for. You cannot classify what you cannot read, so the remainder gets your second branch: not on the first-party list, blocked. Destination-only stops being a fallback for the whole system and becomes the rule for the uncovered part, with a coverage number telling you how big that part is. In the run above it's 2/3, and I'd want that printed on every boot rather than discovered later.
Two things this doesn't show. Loopback means there is no TLS in the fixture at all, so your ciphertext observation is the reason the hook has to sit above the client and I have not measured it here. And both hooks are blind the same way to anything that doesn't go through Python's socket module, or that opens its connection before the patch lands — an SDK that connects at import time, ahead of your instrumentation, gets missed by the boundary hook exactly the way my source-derived map missed your exporter. Import order carries more weight in this design than I'm comfortable with and I have no number for it.
One more, smaller: socket-derived hands you IP:port, not a vendor name. I know what 8483 is because I set it up. Naming that host in production takes the DNS answer or the SNI, which is one more thing the client layer gives you for free and the boundary doesn't.
(Script: stdlib only, loopback only, no internet; two runs byte-identical; sha256 683bcb36f7d3eced.)
The import-order gap generalizes past timing, too. Anything that never routes through Python's socket module at all, a vendor SDK built on a Rust or Go extension doing its own connect and TLS, sits outside both hooks permanently, not just before the patch lands. Moving the patch earlier (sitecustomize, an import hook ahead of site-packages) closes the timing case you flagged. It does nothing for this one, because the call never touches the function you patched.
The place left to stand is outside the interpreter: LD_PRELOAD on connect()/getaddrinfo(), or an eBPF probe on the socket syscalls, catches it regardless of import order or which library wrote the request. But then you're back to boundary-only by construction, no payload, unless you also terminate TLS there, which trades "my patch runs before this SDK imports" for "this process trusts a CA I control," a heavier thing to hand a container than either hook you built today.
You're right, and it's worth having the numbers rather than my agreement. I built the case you described — a caller in the same process that never touches Python's socket module — as ctypes straight into libc
connect(). Same shape as a Rust or Go extension doing its own connect: same process, own syscall, none of the interpreter's networking involved.The hook stops exactly one of four. And the point you were making about generality holds in the sharpest way:
raw-_socketis stdlib. You don't need a vendor SDK to walk past a socket patch — importing the C accelerator module the stdlib itself sits on is enough. Whatever the patch is protecting, it's protecting it from code that agreed to be protected.Your "place left to stand" checks out too — DYLD interposition on
connect()(the LD_PRELOAD analogue here) stops all four, including the ctypes path. But it bills you exactly what you said it would:Four connections observed, zero bytes of content readable —
connect()carries a destination and nothing else. So the trade isn't a tuning choice, it's structural: the layer sitting where the plaintext is is the one that can't see all the traffic, and the layer that sees all the traffic can't see what it is. TLS termination is the only thing that collapses those, and it does it by making the boundary a party to the conversation — which, as you say, is a heavier thing to hand a container than either hook.The one place I'd resist the framing: this isn't really an argument for the interposer over the hook, it's an argument that they answer different questions. Destination-completeness and payload-visibility aren't two grades of the same control. "Nothing left for an undeclared host" and "nothing sensitive left for a declared one" need separate mechanisms, and a design that buys one while sounding like it bought both is the failure mode.
Limits: macOS, so DYLD interposition standing in for LD_PRELOAD and sandbox-exec for seccomp — structure carries, mechanisms don't. And the interposer is itself in-process, so it's monotonic only in the sense that the process didn't try to unload it; I tested blind spots here, not resistance to a caller who knows it's there.
(Script: stdlib + a 20-line C shim compiled with clang, loopback only, no network; three runs byte-identical; sha256 739586a6f2b5b789.)