Read this first: You don't need a frontier model to redact PII
Asked to redact a PII-dense document, llama3.1:8b sometimes answers: "I cannot provide information that could be used to identify an individual." That is a 200 response containing a well-formed English sentence. Write it into your destination field and the original document, every name intact, carries on downstream. The record looks processed.
That was the pattern behind most of the problems we hit while benchmarking six redaction approaches. A parser that fails throws. A schema validator that fails rejects. A redaction pass that fails returns a plausible-looking string in the expected shape, and nothing objects.
The fixes below are less about detection quality than about manufacturing the error signal the system does not produce on its own.
The ordering mistake
The obvious pipeline runs the deterministic pass first and hands the result to the model. Rules catch what rules can, the model handles the rest. It reads like defense in depth.
It quietly makes things worse. When a rule-based pass replaces "Klaus Bauer" with a PERSON tag before the model sees the text, the model hits a token pattern that does not occur in natural prose. Offsets stop lining up with anything it was trained on. It gets confused about character offsets, and starts wrapping the existing tag in a second one, emitting numbered variants nobody asked for, or dropping the clause around it.
None of that raises an exception. The output still contains tags, which is what you were expecting to see.
Run both passes over the original text instead, independently, and reconcile afterwards:
Because the structural pass ran against the original, its offsets are still valid. Because the substitution is a literal string match against the model's output, it cannot corrupt a redaction the model already made correctly. The injection step then catches what the model missed: a partially redacted card number, an IBAN in an unusual format, an IP address.
One filter matters here. Presidio flags "quarterly" and "last year" as dates at 85% confidence, so DATE_TIME detections should be dropped before injection and left to the model. Inject them and you ship a document with tags where ordinary words used to be, which is a worse failure than a missed date and just as silent.
The markup mistake
Enterprise documents carry markup. CRM ticket exports, patient record extracts, consent forms, email bodies. Hand any of that to a model as-is and a div tag reads as a signal that the payload is structured data to be parsed rather than prose to be rewritten.
The output comes back garbled, and garbled HTML still parses. Downstream services accept it. Several of our real-world test cases sat at 0% recall for exactly this reason. Garbled HTML still parses, so nothing downstream rejects it.
Extract the text nodes, redact those, substitute back:
# Input: "<div>Policyholder: Stas al-Sendi</div>"
# Extract: "Policyholder: Stas al-Sendi"
# Redact: "Policyholder: <PERSON>"
# Result: "<div>Policyholder: <PERSON></div>"
That change took those cases from 0% to above 80%. The model had been capable of finding the name all along. The wrapper was stopping it from trying.
Handling the refusal
Back to the refusal from the opening. The model has read redaction as disclosure, which is a defensible mistake about a document consisting entirely of names and account numbers. Neither the Claude nor the Nova models did this on our test set; llama3.1:8b did it occasionally.
Detect the refusal pattern in the first tokens of the response and fall back to the structural pass. Then log the fallback, because a refusal rate climbing on one document type is information about your prompt that you would otherwise never receive.
Fail-open
When the redaction call times out, does your gateway pass the raw text through or reject the request?
Most teams never make this decision explicitly, which means they have made it: the exception handler logs a warning and returns the input. Fail-open on a PII layer is a data leak that returns HTTP 200. Nobody gets paged. The document lands in the index with every name in it and you find out during an audit, if you find out.
Fail closed. Treat the structural pass as the degraded path rather than the bypass, and mark that specific document as degraded so a partial run stays distinguishable from a clean one. A degraded-run counter gives you something to alert on.
Prompt drift
Editing a prompt changes recall. Nothing records that it happened.
Log the model ID and a hash of the prompt alongside every redaction. Without them, this quarter's detection rates are not comparable to last quarter's, and when someone asks why the numbers moved in March there is no way to answer. A prompt is a deployed artifact with a measurable effect on a compliance control, and it deserves to be versioned like one.
While you are in there: log the PII type, the count, and the character span. Do not log the values.
A redaction log holding what it redacted is a second copy of the data you just protected, usually behind weaker access controls than the original and usually retained longer. The values sit right there in the detection object, so logging them is the path of least resistance.
The mapping table that relocates your exposure
Collapsing every name to a single PERSON tag destroys the relationships in the text. "Klaus emailed Petra about Petra's contract" becomes unreadable to whatever consumes it next. Numbered placeholders preserve co-reference, so the downstream model can still follow who did what to whom.
The cost is a mapping table, and that table is PII carrying the same obligations as the source. This is the quiet one, because it feels like progress: you built a redaction layer, your documents now contain tags, and your exposure appears to have shrunk. It has moved into a smaller, more concentrated, more attractive store.
Decide reversibility before you build. If the output returns to a human who needs real names, you need the map, and it needs the access controls, retention policy, and encryption of the original data. If the output is a classification, a summary, or an aggregate that never rejoins the record, do not keep the map: store counts and types for the audit trail and discard the values. Keeping the map is the default in most example code, so this needs to be a deliberate choice rather than an omission.
Precision, invisible by construction
Every redaction metric you are likely to have is recall: did the PII disappear. None of them answer how much clean text was destroyed on the way, because production traffic has no labels and a mangled document does not announce itself.
So manufacture the signal. Keep a fixed set of documents containing no PII at all: an earnings report, a product catalogue, a changelog, a few pages of your own docs. Run it on every deploy and every prompt change. Any tag appearing in that set is a regression.
It is a cheap check for the failure mode that does the most damage. A missed IBAN is invisible but survivable. A layer that eats product names or ticket references produces complaints from the service owners feeding it, and that is how a redaction layer gets switched off.
At platform scale, stop picking a model
Once redaction is a service every internal team calls, the single-model question dissolves. Nothing wins on every axis and your callers do not share constraints: one team's data cannot leave the VPC, another needs a tighter latency budget than a model call allows, a third is German-only. Build one interface and route inside it.
| Caller constraint | Route |
|---|---|
| Default | Nova Micro |
| Tight latency budget | Amazon Comprehend, 100ms against Nova's 400ms |
| Residency-restricted | mistral:7b container in your own VPC |
| German financial documents | Nova Pro |
| Any route fails or times out | Presidio, with the degradation logged |
Those routes come out of a benchmark across two languages: Nova Micro reaches Comprehend's German accuracy at roughly a twentieth of the cost, Nova Pro leads on German financial identifiers by eleven points, and a 4.1GB local model matches both when data cannot leave your network. The numbers are in the companion piece.
The structural pass runs on every request regardless of route. It is deterministic, adds 0.1 seconds, costs nothing, and needs no network call, which is what qualifies it as a fallback.
Two implementation notes if you push your prompt past 1,024 tokens to add few-shot examples, which is worth doing because caching makes a long prompt cost about what a short one costs. The cached prefix has to be byte-identical between calls, so the document goes after the cache point and never inside the prefix: get that wrong and you miss on every request with nothing in the response to tell you. And the cache TTL is five minutes, resetting on each hit, so a steady pipeline stays warm while a nightly batch pays full price on the first call of each window. One upside worth knowing: cache read tokens do not count against Bedrock's tokens-per-minute quotas, which makes caching a throughput lever as much as a cost one.
Build order
- Extract text out of any markup wrapper before anything else touches it.
- Run the semantic pass and the structural pass independently, both against that extracted text.
- Filter DATE_TIME out of the structural results.
- Inject remaining structural detections into the semantic output by literal string match.
- Substitute the redacted text back into the original markup structure.
- Detect refusals in the first tokens of the model response, fall back, and mark as degraded.
- Log type, count, span, model ID, prompt hash. Never values.
Then build the canary set and run it on every change to any of those seven steps. A redaction layer will not tell you when it stops working. The canary set is you telling yourself.
Findings from a PII redaction benchmark across Presidio, BERT NER, open-weight models via Ollama, Amazon Comprehend, Amazon Nova, and Claude, on 15 curated cases and 100 real-world cases per language.
Top comments (0)