Testing masking recall one PII type at a time, instead of as one average, found two real detection gaps a single number would have hidden, including a credit card recognizer that was never active for Bulgarian text at all.
Key takeaways
- Planting around 40 synthetic examples per PII type and measuring recall separately showed a 30 to 100 percentage point gap between the best and worst covered type, a spread no single aggregate recall number reveals.
- The same canary caught an injected regression immediately: disabling one pattern recognizer dropped recall for its type by 75 points while every other type's recall stayed exactly unchanged.
- The clearest finding was not in either registered hypothesis: a credit card recognizer was wired up for English only, so a card number mentioned in a Bulgarian sentence passed through completely undetected, with no error anywhere.
- A second, narrower gap hit English too: card numbers with a length outside the recognizer's expected range were missed regardless of language.
- Every example is synthetic, generated from a frozen, seeded script, not drawn from any annotated dataset or real registry.
Why measure recall per type instead of as one number?
A reader comment on an earlier note, the masking layer matters more than the model, made a point I had not stated clearly: a bug in restoring a masked value gets caught fast, because the output looks visibly wrong. A detector that quietly fails to flag a phone number in the first place looks like nothing at all. The reader's suggestion was direct: stop reporting one recall number and start continuously running synthetic, "planted" PII through the pipeline, checking recall per type.
That is a fair correction. An aggregate recall of, say, 0.85 can describe a system that is uniformly decent, or one that is excellent at emails and blind to credit cards. The two situations call for different fixes, and the average cannot tell them apart. This note is the experiment that comment asked for.
What does a planted-PII canary actually look like?
The method is small on purpose. A seeded script generates synthetic values per entity type, given name, surname, email, phone, card number, date, city, street, and a checksum-valid but entirely fake Bulgarian personal number, then drops each one into a neutral carrier sentence such as "Please contact {name} about the renewal." The exact character span is recorded at generation time, so no manual labelling happens afterward.
Nothing here comes from an annotated dataset or a real registry; every value is either library-generated or built from a public formula, the same discipline as the recall measurements in why I pre-register a kill-gate for every experiment. The set is run through the same ensemble of standard open components measured there, Presidio orchestration, an XLM-RoBERTa NER model, and pattern recognizers for formatted identifiers, unchanged, except for one deliberate change described below.
What did the per-type recall actually show?
Across 240 English and 280 Bulgarian examples, recall by type ranged from 1.000 (email, both languages) down to 0.700 (credit card, English) and 0.000 (credit card, Bulgarian). The spread between the best and worst type was 30 points in English and 100 points in Bulgarian, well past the 20-point bar registered before the run. A system report built from one recall figure would have shown something in the comfortable middle of that range and called it a day.
Why did Bulgarian credit card detection fail completely?
Not a partial miss, a complete one, so I checked the recognizer registry directly rather than trusting the number. The credit card recognizer was registered only under the English language configuration. Several country-specific identity recognizers in the same stack are explicitly duplicated across every supported language; this one was not. A card number sitting in a Bulgarian sentence never reaches a recognizer that could have caught it, no exception thrown, no failing test, because nothing had tried that exact combination before. That is precisely the shape of failure the original comment described: invisible until someone specifically looks for it.
Does a canary like this actually catch a regression?
To find out, I reran the same planted set with the phone recognizer deliberately switched off, simulating an accidental regression during a future change. Phone recall dropped 75 points in both languages. Every other type's recall moved by exactly zero. The canary isolates the break instead of producing generic noise, which is the property that makes it worth running again, not just once for an article.
What's next?
The plan, the generator script, the raw numbers, and the evaluation code are published alongside the earlier experiments at github.com/divanski/kill-gate. "Continuously" above still means a reproducible, seeded script rather than a job on a schedule; turning it into one is the next step, now that the one-off version has already found something worth watching for.
FAQ
Is this the same masking stack as earlier articles?
Yes, the same ensemble of standard open components measured in the kill-gates article, unchanged except for deliberately removing one recognizer to test whether the canary would notice.
Why only English and Bulgarian?
Those are the two languages already covered by the earlier recall measurements on this stack. The method applies to any supported language; this run simply did not extend further.
Is 40 examples per type enough to trust these numbers?
Enough to catch a complete gap like 0% recall with confidence. Not enough to pin down a precise recall figure for a type that is merely mediocre rather than absent; a larger run is the natural next step.
Could other languages have the same kind of gap?
Possibly, and this experiment did not check. It tested the two languages already in scope, found one clear miss, and stopped there rather than claiming a broader audit it did not do.
What triggered this experiment?
A comment from a reader on the masking layer article, arguing that visible restoration bugs get more attention than silent detection misses. The argument was right, and this note is the direct answer to it.
Top comments (0)