Your PII redactor probably leaks tool-call arguments

Most "redact PII before the LLM" tools scan the chat message text and stop there. That was fine when an LLM call was one string in, one string out. It is not fine for agents.

An agentic request carries user data in places that are not the message text:

• Tool-call arguments. The model emits send_email({"to": "[e-mail]", "name": "Marie Dupont"}). The email and name live inside a JSON string under tool_calls[].function.arguments, not in content. A flat-text scrubber never looks there.
• Multimodal parts. content is a list of text and image parts, not a string, so a scrubber that expects a string skips it. (The Responses API input list has the same shape problem.)

So I built a reproducible benchmark to measure how much PII survives in these places, and how the common drop-in approach (a vanilla Presidio pass over the message text, which is what most guardrails ship) compares to a structured-aware one.

The measurement

120 real documents from the open AI4Privacy pii-masking-200k dataset (458 PII items, labeled by 10 independent auditor agents and cross-checked against the dataset's own mask), across DE/EN/FR/IT, plus 14 clean documents (code, prose) to measure false positives. Each PII item is also embedded inside a tool-call argument and a multimodal text part to measure structured leakage. Everything is reproducible: privaite-bench.

Approach	Recall (span)	Recall (strict)	False positives	PII surviving in tool-call args
LiteLLM Presidio guardrail	70.3%	65.3%	3 / 14	99.1%
LLM Guard (Anonymize scanner)	76.9%	74.9%	5 / 14	99.1%
PrivAiTe, light preset	62.4%	57.9%	3 / 14	37.6%
PrivAiTe, onnx preset (default)	84.5%	80.6%	2 / 14	15.5%

Every tool here is configured at its genuine best (full entity coverage, per-document language for Presidio; LLM Guard at its own defaults with entities expanded). LLM Guard's DeBERTa model actually out-recalls LiteLLM's Presidio guardrail on flat text (76.9% vs 70.3%). And yet the number that matters is the last column, not the first: both leak 99% of the same PII the moment it rides inside a tool-call argument, because they scan message text and never parse the tool-call JSON. Better flat-text recall does not close that gap.

How the structured-aware path works

• Two detectors that complement each other. Presidio (regex + checksums + spaCy NER) owns the structured types: emails, phones, credit cards (Luhn), IBANs, IPs, SSNs. The open-source openai/privacy-filter model (run locally via ONNX) owns what regex is weak at: names in any casing, addresses, and secrets/passwords. Their spans are merged as a union.
• Reversible by default. Detected PII becomes numbered placeholders (<PERSON_1>, <EMAIL_ADDRESS_1>), so the provider only ever sees placeholders. The response is de-anonymized back to the real values, streaming and non-streaming, including inside restored tool-call arguments and the model's reasoning trace. (mask and redact modes are irreversible on purpose.)
• JSON-aware tool-call scrubbing. Arguments are parsed and scrubbed value by value, so object keys and the function name stay intact and the tool still receives valid JSON. A card number sent as a bare JSON number is caught too.
• Streaming restoration uses a trie buffer that holds back a placeholder split across SSE chunks until it resolves, per choice index.

One caveat, stated plainly: this is pseudonymization plus transfer minimization, not guaranteed anonymization. Detection is best-effort ML and patterns, so treat it as defense-in-depth, not a compliance checkbox. And LLM Guard's model is fine-tuned on the same AI4Privacy family the benchmark draws from, so its recall above is, if anything, optimistic, while PrivAiTe's default model is independent.

If you run agents against cloud models, the useful question for whatever redaction layer you already run is whether it parses tool-call arguments at all. The benchmark lets you check. Run PrivAiTe as a standalone proxy (docker compose up -d), a LiteLLM guardrail, or an Open WebUI filter.

crp4222 / PrivAiTe

Drop-in self-hosted LLM proxy that reversibly redacts PII before OpenAI, ChatGPT and Ollama calls, including inside tool-call arguments and multimodal content. OpenAI-compatible, zero telemetry.

PrivAiTe

Self-hosted PII redaction proxy for LLM APIs.

A drop-in LLM proxy that reversibly replaces PII before it reaches the provider, including inside tool-call arguments and multimodal content, with zero telemetry.

Keep personal data out of your LLM calls. PrivAiTe is a local proxy that sits between your app and the model provider. It finds names, emails, phone numbers, cards, IBANs, secrets and more, swaps them for stand-ins before anything leaves your machine, and puts the real values back in the reply. Most tools scan only the plain message text and miss the rest, which is the gap PrivAiTe closes. Detection runs on your machine and nothing phones home. By default it runs the full ONNX suite, so it also catches secrets and passwords, not just the easy regex entities. It exposes an OpenAI-compatible API you can point any existing client at, and the same engine runs three ways…

View on GitHub

crp4222 / privaite-bench

Benchmark suite for PrivAiTe PII detection

privaite-bench

PII detection benchmark suite for PrivAiTe.

Measures detection rate, false positive rate, and latency across languages and entity types.

Comparative benchmark (multiple solutions)

COMPARISON.md scores several PII solutions on the same real documents: PrivAiTe (light and onnx) and a vanilla Microsoft Presidio baseline the engine behind LiteLLM's Presidio guardrail and most drop-in PII proxies. The corpus is real AI4Privacy documents whose ground-truth PII was labeled by 10 independent auditor agents and cross-checked against the dataset's own mask. See solutions/ for the framework and how to add another tool.

Headline: privaite-onnx leads on recall (84.5%) and removes PII from tool-call arguments (100% protection) where the flat-text baseline leaks about 99%, the gap that matters for agentic and multimodal traffic.

Latest results

Precision / Recall / F1 (entity-level, 64 documents, 5 languages)

We measure at the entity level:

• TP = entity correctly flagged (matches expected)
• FP = entity flagged but…

View on GitHub