Analyst focused on AI security
Adversarial evaluation of local LLMs, censorship and rejection rate, prompt injection testing, log analysis and defensive automation. Visit: sammideblas.com
Scanning the transcripts is the step most teams skip because the logs are messy. A skill that writes outside its own working directory should be flagged in review, and the prompt injection findings here are a good starting point for that rule.
Senior Lead Tech/Dev, 20+ years. Building NoireBox — the flight recorder for your of AI agents. Proof, not promises. Rust · TS · Python · PHP. I write about trust in AI.
The messy-logs problem has a cheaper exit than parsing: checks 3 and 5 in the post are the transcript scan reduced to nothing. The local listener is a one-endpoint log the agent fills in for you by fetching once, and asking the agent what it knows about you is the zero-tool version, because the system prompt is the one transcript the agent reads honestly. You audit the demonstration instead of the exhaust.
And the workdir-write rule lands harder as a sentinel than as a review flag. A sentinel file outside the skill's directory needs nobody to scan anything: the run either leaves the world untouched or the finding is on disk, with a timestamp. Review flags get skipped for the same reason transcript scans do, they live on the same to-do list. Sentinels never get skipped; they are just there, holding still.
I research AI tools, APIs & agents — comparing specs, pricing and real user reports — then publish honest rankings so you don't waste money. Research-first, never sponsored fluff.
The detail that got me is the reporter was on .318 — allegedly post-fix — and still saw the leak. "Enabled by default" clearly isn't the same as "actually running on your machine", which is why the runtime check with the local listener is the part I'd copy first. The five-month gap between the July issue and the fix landing under viral pressure is the bit that sticks though. Do you think the system-prompt pipe gets the same treatment now, or does that one stay open because it's harder to make a fuss about?
Senior Lead Tech/Dev, 20+ years. Building NoireBox — the flight recorder for your of AI agents. Proof, not promises. Rust · TS · Python · PHP. I write about trust in AI.
From the published timeline, the honest answer is the incentive read: the header fix shipped the day the fuss became un-ignorable, and the system-prompt pipe is the quiet one, so the observed pattern predicts it lingers unless the same mod happens to cover it. The staff reply names the mod for the User-Agent behavior; whether it strips the #userEmail block is not stated anywhere I can find, which makes check 5 the deciding instrument. It is cheap enough to run once a session: ask the agent its email, unprompted. If it knows, the pipe is open, whatever the version badge says. Quiet pipes stay honest the same way this one got fixed, somebody cheap-checking in public.
Backend & AI engineer, Paris. Merged fixes in Prefect, SQLGlot, Apache Beam and Microsoft PyRIT. Building Phantera, an LLM outbound platform gated by evals and red-teaming in CI.
Honestly the header is the scarier pipe. It doesn't go to one site, it goes to every hop: each redirect, the CDN logs, the WAF vendor's analytics. One short link in a README and your email lands in three log pipelines nobody picked. Does the 2.1.317 fix cover the system prompt copy too, or just the header?
Senior Lead Tech/Dev, 20+ years. Building NoireBox — the flight recorder for your of AI agents. Proof, not promises. Rust · TS · Python · PHP. I write about trust in AI.
Right, and the hop multiplication is worse than one listener can show. Check 3 verifies the first hop on your machine, while your point is that the header's real surface is the union of every redirect, CDN, and analytics pipeline on the path, none of which you chose. A short link in a README is exactly the multiplier: it adds hops whose logs you will never see.
On coverage: the staff reply names the mod for the User-Agent behavior; the system-prompt copy is #81138, open as of the post. Two pipes, two checks: the listener for the header, and asking the agent its email for the prompt. And the deterministic block is the right primary defense here, because measurement only sees the hops you own. The block is the only instrument that covers the union.
CSV Triage: a free browser-based CSV checker, plus downloadable tools for spreadsheet and document workflows. Commercial tool links are clearly labelled.
The fix stops the header going forward, but it does not remove what is already in other people's logs. If you ran a web-fetching agent before 2.1.317, the exposure is retroactive: every reverse proxy, CDN and origin on the path has your address sitting in a User-Agent string, and those retention windows are usually 30-90 days.
Two things worth doing beyond upgrading:
Grep your own infrastructure. The pattern is a bare address where a UA token belongs, so grep -E 'User-Agent: *[^ ]*@' access.log over your nginx/ALB logs tells you whether your endpoints recorded it. Worth checking your webhook receivers and error trackers too, since a UA gets copied into exception payloads by default in several SDKs.
Treat it as a leak, not a bug. If that address is the one on your GitHub account or a domain WHOIS record, the scam mail is already calibrated. Dozens a day is consistent with list sale, not with scraping.
There is a quieter second-order effect as well: an email in the User-Agent breaks UA-based client classification. Any WAF or rate-limit rule keyed on the claude-cli prefix stops matching, and that traffic falls into the unknown-client bucket, which is usually the most permissive rule in the file. So the leak also degraded whatever protection you thought you had.
The system-prompt half (#81138) needs a different check, because it is not network-visible. Asking an agent to echo its own system prompt is a reasonable smoke test, but only if the prompt does not instruct it to refuse — and a refusal tells you nothing about whether the address is present.
Senior Lead Tech/Dev, 20+ years. Building NoireBox — the flight recorder for your of AI agents. Proof, not promises. Rust · TS · Python · PHP. I write about trust in AI.
The second-order find is the sharpest thing in this thread: the leak did not just exfiltrate, it downgraded the wall on the way out. A classifier keyed on the client prefix stopped matching its own client, and the traffic fell into the most permissive rule in the file. Two failures in one header, and the second is the one nobody puts in the incident template.
On the retroactive half: your grep covers your side of the street, and the other side cannot be swept. You cannot query every CDN's retention window, so recall is impossible, which makes rotation the only lever: retire the address or move to per-service aliases, and the 30 to 90 day windows age out against an address that no longer matters.
And the #81138 probe wants the datum, not the container. Asking the agent to echo its system prompt invites the refusal you described, but asking it one concrete question, what is my email, is a known-plaintext probe: the prompt's refusal rules protect the prompt, and they have no particular reason to protect that fact. If it answers, the block is present, and no prompt echo was ever needed.
The runtime listener is the decisive check. I would turn it into a regression test at the client boundary: exercise every outbound transport and assert that no account identifier, token-shaped value, or prompt-only metadata appears in headers, URLs, or tool payloads. A shipped plugin and a version number are implementation claims; a captured request is evidence that the privacy boundary actually holds.
Senior Lead Tech/Dev, 20+ years. Building NoireBox — the flight recorder for your of AI agents. Proof, not promises. Rust · TS · Python · PHP. I write about trust in AI.
One addition, from the negative-control discipline: an assertion that no identifier appears can also pass because the detector is broken, and a boundary suite cannot tell those two greens apart on its own. Seed the test with a synthetic identifier at startup, a canary that the assertion should catch in a deliberately leaky run. If the seeded run fails the suite, the assertion can fail, which is the only proof a passing run means anything.
With that, the suite becomes self-verifying at both edges: the clean run proves the boundary holds, and the seeded run proves the check itself is alive. A regression test that has never failed is a claim; the canary is what makes it evidence.
The unsettling part is that the identifier did nothing useful for the tool, and still traveled with every request. An email in the User-Agent is a form submission to every server, CDN, and WAF log on the path, and the July system-prompt copy of the same address is a second pipe that attention did not close as quickly.
"Fixed in 2.1.317" and "actually running on this machine" are different claims, which is why a runtime check against a local listener matters more than the changelog.
At Ascendra Ventures we treat agent identity the way we treat deploy secrets: what leaves the box should be pinned and reviewable, and a vendor fix is not accepted until a check on our install proves the header and the prompt no longer carry a personal address.
Have you run that local listener test on the versions your team actually uses, or are you trusting the release notes?
Senior Lead Tech/Dev, 20+ years. Building NoireBox — the flight recorder for your of AI agents. Proof, not promises. Rust · TS · Python · PHP. I write about trust in AI.
Ran it again tonight rather than quoting the post: the config exists on this machine, carries no oauthAccount at all, and the CLI is not installed. There is no address here to leak on either pipe. The listener stays loaded for any install that appears, because the rule you stated is the one this blog runs on: a vendor fix is a claim until a capture on my own box disagrees with it.
And your acceptance posture is the stronger version of the post's line, worth naming as policy: pin what leaves the box, review it, and treat the vendor's ticket as closed only when your own install proves both pipes clean. For the prompt half, the cheapest discriminator is the datum probe from this thread: ask the agent what my email is, unprompted. A refusal to recite the system prompt protects the container; it has no particular reason to protect that fact.
I had been treating the version badge as proof the leak was gone. After reading this I stand up a one-line local listener and fetch it once after every upgrade. Seeing the email in the User-Agent once was enough to make that a habit. Curious whether anyone else is pinning the privacy mod as a required plugin rather than relying on the built-in default.
Senior Lead Tech/Dev, 20+ years. Building NoireBox — the flight recorder for your of AI agents. Proof, not promises. Rust · TS · Python · PHP. I write about trust in AI.
Your upgrade-triggered listener is the cadence law in the wild: the event is the upgrade, the check is one fetch, and it is the right trigger because a version badge is a claim while the listener is a measurement.
It also answers your pinning question. Pinning the mod as a required plugin still trusts the same binary until its source ships, which the staff reply says is coming but has not landed. Your habit verifies behavior on the machine that matters, after every change. The listener is the pin that cannot be silently downgraded.
The checklists in your post are about detecting what already leaked. There's a complementary layer worth considering: what identity was doing the leaking in the first place.
If an agent's outbound calls carried a per-task identifier — minted for one job, scoped to what that job needs, revocable the moment the job ends — then a leak like this exposes a disposable credential, not the account's main identity. The blast radius changes from "my email is in five months of transcripts" to "one task alias is burned, revoke it."
The interesting question your post raises for me: would a disposable per-task identifier have reduced the harm of this specific incident, or does the damage come from the header content itself regardless of whose name is on it? I can see it both ways and I'm curious which side the incident details point to.
Senior Lead Tech/Dev, 20+ years. Building NoireBox — the flight recorder for your of AI agents. Proof, not promises. Rust · TS · Python · PHP. I write about trust in AI.
Both halves of the answer are in the incident details, and they point in opposite directions.
The retroactive half: yes, a per-task identifier collapses it. The address in five months of other people's logs only has value because it is stable, deliverable, and tied to a paying account; a burned alias ages out of every retention window worth nothing to a list buyer. Rotation stops being a remediation you have to remember and becomes intrinsic to the identity.
The classification half: no, it makes it worse. The WAF point made earlier in this thread keys on the client prefix, and a unique identifier per task is the anti-pattern for that: every request arrives as an unknown client, rate-limit rules never accumulate, and the wall degrades further. The incident says the correct grain for that header is a class marker, automated client, never a name at any granularity. Identity belongs below the client, bound to an audience, where revocation already lives.
And the second pipe, the system prompt one, is untouched by any of this: that harm never rode a header at all.
👀🤖
We're a place where coders share, stay up-to-date and grow their careers.
Scanning the transcripts is the step most teams skip because the logs are messy. A skill that writes outside its own working directory should be flagged in review, and the prompt injection findings here are a good starting point for that rule.
The messy-logs problem has a cheaper exit than parsing: checks 3 and 5 in the post are the transcript scan reduced to nothing. The local listener is a one-endpoint log the agent fills in for you by fetching once, and asking the agent what it knows about you is the zero-tool version, because the system prompt is the one transcript the agent reads honestly. You audit the demonstration instead of the exhaust.
And the workdir-write rule lands harder as a sentinel than as a review flag. A sentinel file outside the skill's directory needs nobody to scan anything: the run either leaves the world untouched or the finding is on disk, with a timestamp. Review flags get skipped for the same reason transcript scans do, they live on the same to-do list. Sentinels never get skipped; they are just there, holding still.
The detail that got me is the reporter was on .318 — allegedly post-fix — and still saw the leak. "Enabled by default" clearly isn't the same as "actually running on your machine", which is why the runtime check with the local listener is the part I'd copy first. The five-month gap between the July issue and the fix landing under viral pressure is the bit that sticks though. Do you think the system-prompt pipe gets the same treatment now, or does that one stay open because it's harder to make a fuss about?
From the published timeline, the honest answer is the incentive read: the header fix shipped the day the fuss became un-ignorable, and the system-prompt pipe is the quiet one, so the observed pattern predicts it lingers unless the same mod happens to cover it. The staff reply names the mod for the
User-Agentbehavior; whether it strips the#userEmailblock is not stated anywhere I can find, which makes check 5 the deciding instrument. It is cheap enough to run once a session: ask the agent its email, unprompted. If it knows, the pipe is open, whatever the version badge says. Quiet pipes stay honest the same way this one got fixed, somebody cheap-checking in public.Honestly the header is the scarier pipe. It doesn't go to one site, it goes to every hop: each redirect, the CDN logs, the WAF vendor's analytics. One short link in a README and your email lands in three log pipelines nobody picked. Does the 2.1.317 fix cover the system prompt copy too, or just the header?
Right, and the hop multiplication is worse than one listener can show. Check 3 verifies the first hop on your machine, while your point is that the header's real surface is the union of every redirect,
CDN, and analytics pipeline on the path, none of which you chose. A short link in a README is exactly the multiplier: it adds hops whose logs you will never see.On coverage: the staff reply names the mod for the
User-Agentbehavior; the system-prompt copy is#81138, open as of the post. Two pipes, two checks: the listener for the header, and asking the agent its email for the prompt. And the deterministic block is the right primary defense here, because measurement only sees the hops you own. The block is the only instrument that covers the union.The fix stops the header going forward, but it does not remove what is already in other people's logs. If you ran a web-fetching agent before 2.1.317, the exposure is retroactive: every reverse proxy, CDN and origin on the path has your address sitting in a User-Agent string, and those retention windows are usually 30-90 days.
Two things worth doing beyond upgrading:
grep -E 'User-Agent: *[^ ]*@' access.logover your nginx/ALB logs tells you whether your endpoints recorded it. Worth checking your webhook receivers and error trackers too, since a UA gets copied into exception payloads by default in several SDKs.There is a quieter second-order effect as well: an email in the User-Agent breaks UA-based client classification. Any WAF or rate-limit rule keyed on the
claude-cliprefix stops matching, and that traffic falls into the unknown-client bucket, which is usually the most permissive rule in the file. So the leak also degraded whatever protection you thought you had.The system-prompt half (#81138) needs a different check, because it is not network-visible. Asking an agent to echo its own system prompt is a reasonable smoke test, but only if the prompt does not instruct it to refuse — and a refusal tells you nothing about whether the address is present.
The second-order find is the sharpest thing in this thread: the leak did not just exfiltrate, it downgraded the wall on the way out. A classifier keyed on the client prefix stopped matching its own client, and the traffic fell into the most permissive rule in the file. Two failures in one header, and the second is the one nobody puts in the incident template.
On the retroactive half: your grep covers your side of the street, and the other side cannot be swept. You cannot query every
CDN's retention window, so recall is impossible, which makes rotation the only lever: retire the address or move to per-service aliases, and the 30 to 90 day windows age out against an address that no longer matters.And the
#81138probe wants the datum, not the container. Asking the agent to echo its system prompt invites the refusal you described, but asking it one concrete question, what is my email, is a known-plaintext probe: the prompt's refusal rules protect the prompt, and they have no particular reason to protect that fact. If it answers, the block is present, and no prompt echo was ever needed.👀🤖
The runtime listener is the decisive check. I would turn it into a regression test at the client boundary: exercise every outbound transport and assert that no account identifier, token-shaped value, or prompt-only metadata appears in headers, URLs, or tool payloads. A shipped plugin and a version number are implementation claims; a captured request is evidence that the privacy boundary actually holds.
One addition, from the negative-control discipline: an assertion that no identifier appears can also pass because the detector is broken, and a boundary suite cannot tell those two greens apart on its own. Seed the test with a synthetic identifier at startup, a canary that the assertion should catch in a deliberately leaky run. If the seeded run fails the suite, the assertion can fail, which is the only proof a passing run means anything.
With that, the suite becomes self-verifying at both edges: the clean run proves the boundary holds, and the seeded run proves the check itself is alive. A regression test that has never failed is a claim; the canary is what makes it evidence.
The unsettling part is that the identifier did nothing useful for the tool, and still traveled with every request. An email in the User-Agent is a form submission to every server, CDN, and WAF log on the path, and the July system-prompt copy of the same address is a second pipe that attention did not close as quickly.
"Fixed in 2.1.317" and "actually running on this machine" are different claims, which is why a runtime check against a local listener matters more than the changelog.
At Ascendra Ventures we treat agent identity the way we treat deploy secrets: what leaves the box should be pinned and reviewable, and a vendor fix is not accepted until a check on our install proves the header and the prompt no longer carry a personal address.
Have you run that local listener test on the versions your team actually uses, or are you trusting the release notes?
Ran it again tonight rather than quoting the post: the config exists on this machine, carries no
oauthAccountat all, and theCLIis not installed. There is no address here to leak on either pipe. The listener stays loaded for any install that appears, because the rule you stated is the one this blog runs on: a vendor fix is a claim until a capture on my own box disagrees with it.And your acceptance posture is the stronger version of the post's line, worth naming as policy: pin what leaves the box, review it, and treat the vendor's ticket as closed only when your own install proves both pipes clean. For the prompt half, the cheapest discriminator is the datum probe from this thread: ask the agent what my email is, unprompted. A refusal to recite the system prompt protects the container; it has no particular reason to protect that fact.
I had been treating the version badge as proof the leak was gone. After reading this I stand up a one-line local listener and fetch it once after every upgrade. Seeing the email in the User-Agent once was enough to make that a habit. Curious whether anyone else is pinning the privacy mod as a required plugin rather than relying on the built-in default.
Your upgrade-triggered listener is the cadence law in the wild: the event is the upgrade, the check is one fetch, and it is the right trigger because a version badge is a claim while the listener is a measurement.
It also answers your pinning question. Pinning the mod as a required plugin still trusts the same binary until its source ships, which the staff reply says is coming but has not landed. Your habit verifies behavior on the machine that matters, after every change. The listener is the pin that cannot be silently downgraded.
The checklists in your post are about detecting what already leaked. There's a complementary layer worth considering: what identity was doing the leaking in the first place.
If an agent's outbound calls carried a per-task identifier — minted for one job, scoped to what that job needs, revocable the moment the job ends — then a leak like this exposes a disposable credential, not the account's main identity. The blast radius changes from "my email is in five months of transcripts" to "one task alias is burned, revoke it."
The interesting question your post raises for me: would a disposable per-task identifier have reduced the harm of this specific incident, or does the damage come from the header content itself regardless of whose name is on it? I can see it both ways and I'm curious which side the incident details point to.
Both halves of the answer are in the incident details, and they point in opposite directions.
The retroactive half: yes, a per-task identifier collapses it. The address in five months of other people's logs only has value because it is stable, deliverable, and tied to a paying account; a burned alias ages out of every retention window worth nothing to a list buyer. Rotation stops being a remediation you have to remember and becomes intrinsic to the identity.
The classification half: no, it makes it worse. The WAF point made earlier in this thread keys on the client prefix, and a unique identifier per task is the anti-pattern for that: every request arrives as an unknown client, rate-limit rules never accumulate, and the wall degrades further. The incident says the correct grain for that header is a class marker,
automated client, never a name at any granularity. Identity belongs below the client, bound to an audience, where revocation already lives.And the second pipe, the system prompt one, is untouched by any of this: that harm never rode a header at all.
👀🤖