DEV Community

Origin Data Partners
Origin Data Partners

Posted on AI-assisted

Best data anonymization tools for dataset licensing (2026)

The best open-source stack for preparing a dataset to license covers six jobs, and no single tool does all of them. For personal information in text, use Presidio with spaCy (or scrubadub for something smaller); for tables, ARX; for faces in video, deface; for secrets in code, gitleaks or TruffleHog, then git-filter-repo to remove them from history; for third-party code, ScanCode; for near-duplicates, datasketch or text-dedup; and for documentation, a Hugging Face dataset card or a datasheet. Every one of them is free, and every detection tool misses things, so a human still reviews a sample.

What does preparing a dataset for licensing involve?

It means removing what the buyer must not receive and describing what they will. In practice that is six jobs: strip personal information, blur faces, remove credentials, separate code you don't own, collapse duplicates, and write down what the dataset is and how it was made.

Licensing raises the bar compared with internal use. Records leave your control, and the buyer's lawyers will ask how you de-identified them. Our guide to de-identifying company data before licensing covers the method. This post covers the tools.

Which tools are best for each job?

We checked each project's license and status on its official repository on 7 November 2026. Licenses and maintenance change, so re-check before you adopt one.

Tool Job License What it does
Presidio PII in text, images, tables MIT Detects names, card numbers, phone numbers and more with NER, regex, rules and checksums, then masks or replaces them
spaCy Named entity recognition MIT The NLP engine Presidio loads by default (en_core_web_lg)
scrubadub PII in free text Apache-2.0 Small Python library: names, emails, phones, addresses (US, GB, CA), card numbers, US SSNs
ARX Tables Apache-2.0 k-anonymity, l-diversity, t-closeness, differential privacy and re-identification risk analysis
deface Faces in video and photos MIT Detects faces frame by frame and applies blur, solid boxes or mosaic
gitleaks Secrets in git history MIT Scans every patch through git log -p, plus directories and stdin
TruffleHog Secrets in git history AGPL-3.0 700+ detectors, with optional live verification of each credential
detect-secrets Stopping new secrets Apache-2.0 Baseline file plus pre-commit hook, so only new secrets fail a commit
git-filter-repo Removing data from history MIT Rewrites history to delete files or replace strings in every commit
ScanCode Toolkit Third-party code Apache-2.0 Detects licenses, copyrights and package manifests in source and binaries
datasketch Near-duplicate text MIT MinHash and LSH building blocks for your own pipeline
text-dedup Near-duplicate text Apache-2.0 Ready scripts for MinHash, SimHash, suffix array and Bloom filter dedup
dedupe Duplicate records MIT Fuzzy matching and entity resolution for structured rows
Hugging Face dataset cards Documentation Format A README with YAML metadata: license, languages, structure, collection
Datasheets for Datasets Documentation Paper A question set on motivation, composition, collection and intended uses
Croissant Machine-readable metadata Apache-2.0 MLCommons format built on schema.org that describes files, structure and semantics

Which PII tool should you start with?

Start with Presidio unless your data is tiny. It handles free text, images (including DICOM medical images) and structured data, and you can add your own recognizers for internal IDs such as customer or ticket numbers, which no generic model knows about.

One change matters as of November 2026: Presidio is no longer a Microsoft project. It moved to a community-governed GitHub organization, Data Privacy Stack, stays MIT-licensed, and new Docker images now ship from GitHub Container Registry instead of Microsoft's registry. Old links redirect, but update mcr.microsoft.com/presidio-* image references.

A minimal run looks like this:

from presidio_analyzer import AnalyzerEngine
from presidio_anonymizer import AnonymizerEngine

text = "Ticket 4411: call Dana Ruiz on 212-555-0143 about the refund."
findings = AnalyzerEngine().analyze(text=text, language="en")
print(AnonymizerEngine().anonymize(text=text, analyzer_results=findings).text)
Enter fullscreen mode Exit fullscreen mode

The default operator replaces each finding with its entity type, so a name becomes a PERSON placeholder. The README itself warns that automated detection cannot guarantee it finds everything. Treat its output as a first pass and review a random sample by hand.

scrubadub is simpler and fine for a quick script, but its last release (v2.0.1) dates from September 2023. For tables of customers or transactions, ARX is the better fit, because the risk there is rare combinations of ordinary fields (postcode, age, job title), not names in sentences.

How do you clean secrets out of a codebase?

Scan the full history, rotate what you find, then rewrite. A key deleted in a later commit is still in the old one, and a code buyer receives the whole history.

  • gitleaks is the fast default. gitleaks git -v --log-opts="--all" . scans every branch.
  • TruffleHog adds verification: it tests each candidate against the provider's API and reports whether it is live. That is useful for triage, but it means the tool contacts those services with your credentials. --no-verification turns it off. The AGPL-3.0 license matters if you plan to embed it in a product rather than run it.
  • git-filter-repo does the rewrite. Since version 2.47 it has a --sensitive-data-removal flag that also reports which commits changed and what to clean up elsewhere.

How do you handle faces, duplicates and third-party code?

Use deface for faces, a MinHash tool for duplicates and ScanCode for code you don't own. Each solves one narrow problem well.

deface blurs faces only. Number plates, screens and paperwork in frame need another pass. Its output leaves out the audio track unless you pass --keep-audio, which is the right default for workplace footage.

For duplicates, DeepMind's Genie paper is a useful reminder of why curation matters: a curated set of about 30,000 hours, roughly a tenth of the raw video, trained a better model than the full set (Genie, arXiv). Near-duplicate support tickets and email threads inflate your volume count without adding new examples, so dedupe before you describe the dataset to anyone.

For codebases, ScanCode lists vendored libraries and their licenses so you can exclude or document them. Our guide to selling source code for AI training explains why ownership gaps end code deals.

What should the documentation say?

Write a dataset card even if the buyer never asks. Cover what the records are, the years they span, how they were collected, what you removed and with which tools, and what the buyer may not do with them. Datasheets for Datasets gives the questions; a Hugging Face card or Croissant file gives the format.

The card also forces an honest inventory. If you can't say where a field came from, a buyer can't license it. Records you never kept can't be documented at all, which is why a retention policy that keeps data sellable comes before any of these tools.

Common questions

Do these tools make data anonymous?
Not on their own. They remove identifiers, but under the GDPR pseudonymised data is still personal data, and rare combinations of facts can identify people after names are gone. Anonymity is a result you test, by sampling and by asking whether someone with outside knowledge could re-identify a person, not a setting you switch on.

Is gitleaks or TruffleHog better?
You can run both. gitleaks is MIT-licensed, fast and easy to put in CI. TruffleHog's verification tells you which findings are live credentials, which helps you rotate the right keys first. The two use different detector sets, so a second scan on a long history is cheap insurance.

Can I use an LLM to redact PII instead?
You can, but sending raw records to a hosted model is itself a disclosure of the data you are trying to protect. Local tools such as Presidio keep the records on your machines. If you add a model, run it locally or under a data processing agreement, and still review a sample.

Who does the clean-up, the seller or the buyer?
Usually the seller, or a processor acting on the seller's instructions, before records leave its control. The licensing agreement should say who is responsible. A buyer can describe the format and quality it needs, but raw records with personal information should not be sent out to be cleaned.

If you want to know whether your records are worth this preparation, our free estimate of what your company data is worth takes six questions and no files.

Origin Data Partners is an independent data partner that helps companies get their data bought. A buyer may pay Origin when a deal closes.

Top comments (0)