I've been collecting the disclosed cases of LLM apps leaking data, and the thing that struck me isn't that they happen — it's how identical they are. Different companies, different products, same exact shape. If you build LLM apps, this is the pattern worth burning into memory, because it's not going away.
Here are three that are publicly disclosed and patched:
EchoLeak (Microsoft 365 Copilot, disclosed as CVE-2025-32711, CVSS 9.3) — a single incoming email with a hidden instruction got Copilot to pull internal context out, with zero clicks from the user.
CamoLeak (GitHub Copilot Chat, disclosed as CVE-2025-59145, CVSS 9.6) — an invisible markdown comment in a pull request got Copilot to fetch secrets from private repos and encode them out.
GitLab Duo (patched in duo-ui!52) — a hidden instruction in a merge request description, commit, or even source code got Duo (built on Claude) to leak private source.
The shape they share
Strip away the vendor and each one is the same five beats:
The AI ingests content from outside the trust boundary — an email, a PR comment, a doc, source code.
That content has an instruction hidden in it, often literally invisible to a human reviewer (CamoLeak used GitHub's invisible-comment syntax; GitLab Duo researchers used Unicode smuggling and KaTeX white-on-white text).
The AI follows that instruction using its own legitimate access.
It pulls a secret it's allowed to see and places it into its output.
The user did nothing wrong. Often they asked something completely mundane.
This is a classic confused deputy: the AI is the deputy, it has real authority, and an attacker who can't reach the secret directly tricks the deputy into fetching it. The novelty isn't the confused-deputy idea — that's decades old. The novelty is that the instruction is natural language buried in content, so none of the usual defenses fire. There's no malformed payload for input validation to reject. There's no schema that says "this paragraph is data, not a command." The model reads everything in its context window as potentially-actionable, and the context window mixes trusted and untrusted content on the same plane.
Why this is genuinely hard to defend
The honest version: you can't fully schema-validate natural language, and you can't easily stop a model from treating retrieved content as instructions, because treating retrieved content as meaningful is the entire point of the assistant. The vendors above didn't ship sloppy code — they shipped useful features whose usefulness was the attack surface. Microsoft patched server-side; GitHub disabled image rendering in Copilot Chat; GitLab blocked unsafe HTML rendering to external domains. All reactive, all after disclosure.
(I'm deliberately not detailing the exfiltration channels — the image-URL tricks, the proxy abuse, the CSP bypass. Those are real but they're a web-rendering layer, and they're not the part builders can generalize from. The generalizable part is upstream: a secret the model should never emit showed up in its output.)
Where this connects to what I'm building
This pattern is exactly why I went deterministic with rojaprove instead of asking an LLM to judge whether an app is safe.
If the failure is "a secret that should never appear, appeared," then you don't need a judgment call — you need a canary. Plant a string in your system prompt that has no legitimate reason to ever surface in a response. Send the kind of inputs these attacks use. Then check, by exact string match, whether the canary came back out. It did, or it didn't. No model interpreting another model's output, no probabilities.
I want to be precise about scope, because it matters: rojaprove today detects system-prompt leakage (OWASP LLM07) — the slice where the leaked thing is the prompt itself. The full indirect-injection-to-exfil chain in EchoLeak/CamoLeak/Duo is broader than what I detect today; those sit on my roadmap, not in my "tested" column. And there's a neighboring class I deliberately don't touch — multi-tenant access control ("can user A read user B's row") — because there's no canary for it: both records are real, nothing should-never-appear, so a leak-shaped check can't see it. I'd rather be narrow and deterministic than broad and hand-wavy.
But the GitLab Duo detail is the clearest illustration of why the canary approach works: the researchers hid prompts with base16 encoding, Unicode smuggling, KaTeX white text. All of that hiding is invisible to a human and to most filters — but a planted canary doesn't care how the instruction was hidden. It only cares whether the secret surfaced. That's the whole appeal of a ground-truth oracle: the attacker's cleverness upstream doesn't change the yes/no downstream.
It's free and open source, BSL 1.1.
→ github.com/ghkfuddl1327-wq/rojaprove
If you ship LLM apps: have you tested for this shape pre-launch, or is it the kind of thing you'd only find out about from an incident report? And which part of the chain (prompt leakage, indirect injection, exfil) would actually be useful to catch first? Genuinely asking — I'd rather build the slice people need.
Top comments (0)