So, in my last piece I talked about how your de-identification is only as good as your detection. Detection is the first building block of de-identifying data in a way that keeps its context. It also opens a whole new can of worms. What are the approaches, how do they work, and why do they so often fail?
It can look straightforward. Categorise identifying data, build a set of rules to find that data, and highlight every example. But categorising identifiers is hard in itself. Our founding researcher likes to point out how difficult names are: the same word can be a proper noun in a name and a common noun anywhere else. “Hannah Grey” has the shape of a full name, but “grey” in lower case reads as a colour.
Approaches
Regex is fast and predictable, and it’s strongest when paired with checksum verification. Machine learning models, including named entity recognition (NER), learn from training data to predict whether a span is sensitive. LLMs bring broad pretraining to the same job and can read context.
Teams often pick one of these and stretch it across a whole dataset. But the better approach is to look at what kind of data you’re treating, and what it needs for the purpose you’re using it for. Data treated with attention to its meaning and context performs very differently from data treated in bulk.
Rules
Some identifiers carry their own proof. Card numbers end in a Luhn check digit. An IBAN is only valid if a mod-97 calculation leaves 1. UK NHS numbers use mod-11, and a result of 10 means the number can’t exist. For these, a rule plus a checksum is fast, cheap and quite robust.
But a checksum is a filter, not a proof. Roughly one in ten random digit strings of the right length passes Luhn, so false positives grow with your data. Presidio, for example, had to fix a case where 13-digit timestamps were being read as card numbers.
Some identifiers have no checksum at all. US Social Security numbers have been randomly assigned since 2011, so you can only reject a handful of impossible values. UK National Insurance numbers have a fixed shape but no check digit. Both need context before you can trust a match, and that means needing a second approach. Rules also can’t find what has no fixed shape; names and free-text addresses defeat regex entirely.
Models
Models can read context, and they rely on it. In OpenAI’s Privacy Filter model card, overall recall rises from 58% when a value appears on its own to 86% when a cue such as “this is my home landline number” comes before it.
Headline scores need care too. Privacy Filter reports a token-level F1 of 0.96 on PII-Masking-300k, a synthetic benchmark. On SPY, a separate synthetic benchmark of legal questions and medical consultations, scored with exact span matching, five open PII models, including Privacy Filter averaged between 0.37 and 0.47 F1. How you score matters as much as what you score. On a credentials dataset, Privacy Filter’s own card reports 0.84 F1 counted by tokens, but 0.62 counted by exact spans, because partly detected spans get the boundaries wrong.
In combination?
The answer could be to “use both”. In a 2026 stress test, researchers at Halmstad University ran spaCy, Presidio and a small LLM over deliberately messy text, and each failed differently. Presidio’s regex caught every SSN and IP address. spaCy found 95% of names. The LLM coped best with typos, but still labelled nearly half of the card numbers as SSNs. The researchers settled on recommending running all three and merging the results.
A public CV benchmark from this year shows what using in combination buys you, and where it stops helping. Default Presidio, which pairs regex with a spaCy model, found 65% of the personal details and leaked something in 39 of 40 CVs. Swapping spaCy for GLiNER inside Presidio barely moved that. The biggest single gain came from a policy layer written specifically for CVs: the best stack, three GLiNER-family models plus that layer, reached 0.97 recall but still leaked details in 10 of 40 CVs.
The same pattern shows up on the Text Anonymization Benchmark (TAB), built from European Court of Human Rights judgments. TAB separates direct identifiers from quasi-identifiers, so it tests contextual reasoning rather than pattern matching. A Longformer model fine-tuned on TAB’s training data caught every direct identifier in the test set. Presidio caught under half, partly because nothing in its rules or its general-purpose model covered court case numbers, which identify the applicant.
OpenAI saw the same effect with Privacy Filter on SPY. Out of the box it scored 0.55 F1. After training on 10% of SPY’s training data, it scored 0.96.
The real divide isn’t approach versus approach, but whether the detector has seen and been trained on labelled examples of your kind of data and been able to train.
But labelling is slow and expensive, and not many teams have a few thousand labelled examples to hand. It also means people reading the very data you’re trying to protect, and the result can work well on one domain without necessarily transferring to the next.
Regex and checksums hold up for a different reason. They behave the same way every time, they can be audited, and they only change when you change them. A rule can be explained to a compliance lead and tested against known examples until you know where its edges are. A predictable rule that finds card numbers in log files is worth more than a few points of F1. When the data belongs to someone else, knowing where your pipeline leaks matters more than its average score.
Hybrid doesn’t always win. It helps when each part suits your data: rules for identifiers with a fixed shape, and models trained on text like yours for everything else. The only score that tells you how a detector will behave on your text is one measured on a labelled sample of it.
Sources:
Mind the Gap: Robustness Risks in PII Detection Systems (arXiv, Sep 2026)
CV-PII-Bench (GitHub)
OpenAI Privacy Filter model card
GLiNER2-PII (arXiv, May 2026)
The Text Anonymization Benchmark (TAB)
Bootstrapping Text Anonymization Models with Distant Supervision (TAB test results)
Presidio release 2.2.359
Top comments (0)