If you ship an LLM feature, you probably put a redaction layer in front of the model to strip sensitive data. Good. There's already solid, independent work measuring how well these tools catch PII — academic benchmarks like LeakageBench and REDACT, vendor comparisons on datasets like ai4privacy. If PII is your worry, that ground is covered.
Here's what isn't measured: secrets. The API keys, tokens, connection strings and private keys that end up in prompts not because a user typed them, but because someone pasted a stack trace, an env dump, or a "why isn't this config working?" blob. Those are arguably worse to leak than a name — and nobody's independently testing whether your redactor stops them.
So I built a small, zero-dependency, vendor-neutral tool — LeakProbe — that fires evasive secret (and PII) payloads at any redactor and reports what survives. Then I ran it against a real, popular OSS library: redact-pii.
Give the library its due
redact-pii is genuinely good at the job it advertises. Against my corpus it redacted 100% of the classic PII — emails, credit cards, IBANs, US SSNs — including spaced and lightly obfuscated variants that a naive regex misses. If you picked it to scrub PII, it earns its place.
Now point it at secrets
Here it gets interesting — the coverage is a patchwork:
| Secret | Result |
|---|---|
| Google API key, Slack token, GitHub fine-grained PAT | Caught |
| DB / Mongo connection strings (password in the URL) | Caught |
| Private keys — RSA, EC, OpenSSH, even well-formed PEM | Leaked (100%) |
| Stripe live key, SendGrid key, AWS secret access key | Leaked |
| AWS access key id, OpenAI key, classic GitHub PAT | Leaked |
| JWT / Bearer token inside a JSON blob | Leaked |
| Moroccan phone / national ID (CIN) | Leaked |
This is sneakier than "PII tools are blind to secrets." redact-pii catches some credentials — Google, Slack, GitHub fine-grained tokens, even passwords inside connection strings. That partial coverage is exactly what lulls you into trusting it. Then your private keys, your Stripe live key, and your AWS secret access key sail straight through, and you cannot eyeball which half you got. (Overall on my corpus: 20 of 38 payloads leaked, 52.6%.)
It's not that the library is broken — it's a PII tool, and a private key isn't PII. The failure is the deployment: assuming one redactor covers everything sensitive going into your prompts.
The point isn't this one library
Two takeaways:
- "We have redaction" is not a measurement. Coverage is specific. A tool that nails PII can pass 100% of your secrets, and you won't know until one shows up in a log, a trace, or a model provider's retained prompt.
- If your users aren't US/EU, a lot of "PII redaction" quietly doesn't apply to them. The Moroccan number and CIN were invisible here.
And these aren't hypothetical — the gaps mirror real, documented issues: streamed-output splits (presidio #2312), silently-dropped locale entities (presidio #2256), and secrets hiding in tool-call arguments (litellm #41265). LeakProbe ships a test for each.
Honest caveats
- Testing a PII tool on secrets is unfair to the tool — but it's exactly fair to how these libraries get used in front of LLMs.
- This corpus is v1 and not exhaustive.
- Leak detection is verbatim-survival; partial masking isn't scored yet.
- Independent PII benchmarks exist and are good — LeakProbe isn't claiming to be the first benchmark. Its focus is the uncovered slice: secret leakage, tested against your own redactor, in CI.
Run it on your own stack
LeakProbe is open source, zero dependencies. Write a five-line adapter around whatever you use today and get a per-category leak report.
git clone https://github.com/roshcompanylabs/leakprobe && cd leakprobe
node src/cli.js run --target ./your-adapter.js --threshold 0 # fail CI on any secret that leaks
If you run it and something surprising leaks, I'd genuinely like to hear what.
Top comments (1)
The deployment point is the one that matters: "we have redaction" says nothing until someone checks which categories it covers. Secrets and PII really need different detectors: most secrets have a known prefix or format, while PII needs checksums, context words and, for names, a model. Tool-call arguments are a good case to keep in the corpus, since an agent's tool calls carry whatever it has read.
Disclosure: I'm the author of Latet, a local layer between AI agents and the model, built on that split: gitleaks' default rules for secrets, Presidio's recognizers (patterns, context words, checksums) for PII, and the real values restored in responses, streams and tool-call arguments. Your locale point applies to it too: national ID and tax numbers are covered for 15 countries, not everywhere. I'd like to run LeakProbe against it and share what survives.
github.com/Latet-AI/latet