DEV Community: Felipe Carvajal Brown

PII masking in Polars: MaskOps 2.0, and two metrics that lied to me

Felipe Carvajal Brown — Thu, 11 Jun 2026 04:27:39 +0000

MaskOps 2.0 shipped this week. Before I told anyone, I looked at my own numbers. Two of them were lying to me, in opposite directions.

MaskOps is a Rust plugin for Polars that does PII masking inside the dataframe: RUT, CPF, credit cards, IBANs, and twenty-odd more families, air-gapped, with no network call, ever. If you have reached for Microsoft Presidio and found it carries no Latin American identifiers, that is the gap MaskOps fills. It does check-digit-validated RUT, CPF, and CURP detection alongside the EU, US, and APAC families, as a native Polars expression. Version 2.0 is the enterprise line: configurable patterns, structured extraction, an audit pass that counts what it masked, and format-preserving encryption (GDPR Art. 4(5) pseudonymization) for the reversible cases. That part I was sure of. The numbers around it, less so.

The first number lied against me: the benchmark

The last thing I checked was the benchmark table in my own README. It said MaskOps ran at 0.4× to 0.7× the speed of plain Python re. Slower than the language I wrote it to replace.

I almost opened the profiler. Instead I read the benchmark harness. I should have read it first.

Here is what it did. For every family, "Credit Card", "EU", "LatAm", it ran the full masker. All thirty-five pattern families at once. Then it compared the time against a Python baseline that ran one regex for that family.

So the "Credit Card" row timed MaskOps scanning for cards, phones, IBANs, Korean RRNs, and thirty others, against Python scanning for cards. The proof sat in the table the whole time: every MaskOps row took the same 2.3 seconds regardless of family, because it always did all the work. Only the Python column moved.

I was timing my engine doing thirty-five times the work and calling it slow.

Two fixes. The first was the benchmark, not the code: compare like-for-like. When the row says "Credit Card", mask credit cards, the same job the baseline does. MaskOps already supports selection, so it was one argument: mask_pii("text", patterns=["credit_card"]).

The second was real. Most rows in real data contain no PII. Every pattern MaskOps detects needs a digit, or an @. A row with neither cannot match anything. So before any regex, walk the bytes:

pub fn has_pii_candidate(value: &str) -> bool {
    value.bytes().any(|b| b.is_ascii_digit() || b == b'@')
}

If false, return the string untouched. On clean text this skips all thirty-five scans for the price of one pass over the bytes. Output does not change. The same 394 tests pass.

PII masking in Polars, measured fairly. One million rows, median of three, against a pure-Python re baseline with matching coverage:

Data profile	Speedup vs Python
clean (no PII)	11×–163×
mixed (50% PII)	1.2×–3.2×
dense (every row)	1.3×–2.7×

One family still loses on dense data. The European ID set runs four separate regex passes, and a single combined Python regex edges it out, 0.9×. I left that in the README. A table with no losses is a table someone tuned until it lied.

The second number lied for me: the downloads

The other number was downloads. I shipped the 1.7 through 2.0 releases in one short burst, and the PyPI counter jumped from about ten a day to three and a half thousand on release day. A hundredfold, overnight.

It would be easy to write "downloads are exploding." It would also be false.

That spike sits exactly on the days I pushed releases. It is CI building wheels across the OS and Python matrix, mirrors syncing, bots crawling each new version. PyPI counts all of it. None of it is a person deciding to use the thing. Strip the release days and the real line is flat and small. Single digits, which is the honest state of a young project.

So I am not going to tell you adoption is taking off. The download number is real and it is mostly noise, and pretending otherwise insults anyone who can open the same pypistats page I did.

What I take from this

Two instruments. One read low because it measured the wrong thing. One read high because it counted the wrong things. A metric is not a verdict. It is a measurement, and a measurement can be miscalibrated in your favor or against it, and you owe it to yourself to know which.

Read the harness before the flame graph. Strip the release days before you celebrate the downloads. Then trust what is left.

MaskOps is open source, MPL-2.0, on PyPI. It does PII masking inside Polars, air-gapped, with check-digit validation so a random nine-digit number is not mistaken for an ID. It does not do named-entity recognition. The source and the benchmark code are on GitHub. Run it. If your machine disagrees with mine, I want to know.

I dropped this for three months. Here's what I added when I came back.

Felipe Carvajal Brown — Mon, 08 Jun 2026 16:28:44 +0000

I dropped this for three months. Here's what I added when I came back.

I started MaskOps in March. It masks PII in Polars DataFrames using Rust — no Python per row, no NLP models, just regex running on Arrow buffers.

Then I got hired. Cencosud S.A. The project sat untouched until last week.

Coming back to it, I had a backlog. I shipped the one I kept thinking about at work: mask_pii_audit.

The problem with masking alone

Masking answers "is this field safe to store?" It doesn't answer "what kind of PII just came through, and how much of it?"

Compliance teams need both. They need the masked value and a count of what was found — by family — without running the column twice.

What mask_pii_audit does

It returns a nested Struct: the masked text, plus a count for each of the 33 PII families.

import polars as pl
import maskops

df = pl.DataFrame({"notes": [
    "Call me at 555-123-4567. SSN: 123-45-6789.",
    "IBAN: DE89370400440532013000",
    "Nothing here.",
]})

result = (
    df
    .with_columns(maskops.mask_pii_audit("notes").alias("audit"))
    .unnest("audit")
)

print(result.select("masked", "counts"))

┌───────────────────────────────┬────────────────────────────┐
│ masked                        ┆ counts                     │
╞═══════════════════════════════╪════════════════════════════╡
│ Call me at ***-***-****. SSN… ┆ {"phone": 1, "ssn": 1, …} │
│ IBAN: DE89******************  ┆ {"iban": 1, …}             │
│ Nothing here.                 ┆ {"phone": 0, "ssn": 0, …} │
└───────────────────────────────┴────────────────────────────┘

Same masked output as mask_pii. Zero fields mean no match.

One pass

The counting happens inside the existing replace_all call. A Cell<u32> in the closure increments on each validated match. No second scan, no cloned strings.

pub fn replace_counted<F>(re: &Regex, s: &str, render: F) -> (String, u32)
where F: Fn(&Captures) -> Option<String> {
    let count = Cell::new(0u32);
    let out = re.replace_all(s, |caps: &Captures| match render(caps) {
        Some(masked) => { count.set(count.get() + 1); masked }
        None => caps[0].to_string(),
    }).into_owned();
    (out, count.get())
}

A daily audit pattern

(
    df
    .with_columns(maskops.mask_pii_audit("free_text").alias("audit"))
    .unnest("audit")
    .select(
        pl.col("counts").struct.field("ssn").sum().alias("ssn_total"),
        pl.col("counts").struct.field("credit_card").sum().alias("cc_total"),
        pl.col("counts").struct.field("iban").sum().alias("iban_total"),
    )
)

Run this at ingest. Log the totals. Alert if a family appears that shouldn't.

Where it stands

v1.6.0. 33 PII families: EU IDs, US healthcare, LATAM nationals, APAC. Asterisk masking and FF3-1 format-preserving encryption. Polars lazy and streaming supported.

pip install maskops

Source: github.com/fcarvajalbrown/MaskOps

Happy to answer questions.

MaskOps 0.1.0: A Native Polars Plugin for High-Speed PII Masking in Python

Felipe Carvajal Brown — Mon, 09 Mar 2026 03:55:40 +0000

TL;DR: I built a Rust-powered Polars plugin that masks GDPR-sensitive data (IBAN, EU VAT) at up to 16 million rows per second — no NLP models, no spaCy, no Presidio overhead. pip install maskops.

The Problem

If you work with financial data, healthcare records, or any GDPR-regulated dataset in Python, you've likely hit the same wall: de-identifying structured data at scale is painfully slow.

The go-to solution is Microsoft Presidio. It's powerful, but it's built for unstructured text — it spins up a full spaCy NLP pipeline to find a phone number in a CSV column. For structured DataFrames where you already know which columns contain PII, that's enormous overhead:

Presidio with spaCy NER: ~1,000–5,000 rows/s
Presidio with regex-only recognizers: ~10,000–50,000 rows/s
Pure Python re module: ~1,100,000 rows/s

None of these integrate natively with Polars, the fastest DataFrame library in Python.

The Solution: maskops

maskops is a native Polars expression plugin written in Rust. It extends Polars with two new expressions — mask_pii() and contains_pii() — that run directly on Arrow memory buffers with zero Python overhead per row.

import polars as pl
import maskops

df = pl.read_csv("payments.csv")

# Mask all PII in a column
df.with_columns(maskops.mask_pii("notes"))
# "Transfer to DE89370400440532013000" → "Transfer to DE89******************"

# Boolean detection — filter rows containing PII
df.filter(maskops.contains_pii("free_text"))

That's it. No model downloads, no engine initialization, no spaCy.

Benchmarks

Tested on 1,000,000 rows, Intel i-series CPU, Python 3.14, Windows.

maskops throughput

Profile	Expression	Time	Rows/s	MB/s
clean (no PII)	`mask_pii`	0.404s	2,477,599	54.5
clean (no PII)	`contains_pii`	0.169s	5,915,970	130.2
dense (all PII)	`mask_pii`	1.385s	722,104	15.9
dense (all PII)	`contains_pii`	0.059s	16,987,879	373.7
mixed (50/50)	`mask_pii`	0.760s	1,315,407	28.9
mixed (50/50)	`contains_pii`	0.133s	7,498,315	165.0

vs pure Python regex (same machine)

Profile	maskops `mask_pii`	Python `re`	Speedup
clean	0.404s	0.925s	2.3×
dense	1.385s	1.653s	1.2×
mixed	0.760s	1.337s	1.8×

On clean and mixed data maskops is consistently faster. On dense data (every row is a full IBAN) both are regex-bound — the bottleneck is the pattern itself, not Python overhead.

vs Microsoft Presidio (estimated)

Presidio processes structured DataFrames via presidio-structured, which runs a spaCy NLP pipeline per row. Based on community reports and the architecture:

Tool	Throughput (structured data)	Requires NLP model
maskops	~700K–17M rows/s	No
Presidio (regex-only recognizers)	~10–50K rows/s*	No
Presidio (spaCy NER)	~1–5K rows/s*	Yes (250MB+)

* Estimated from community benchmarks and Presidio's own documentation noting it is "not optimized for bulk structured data." Microsoft confirmed no official throughput benchmarks exist.

maskops is purpose-built for structured data pipelines where Presidio's NLP overhead is unnecessary.

How It Works

The key is the Polars expression plugin system, introduced in Polars 0.20. It allows you to register custom Rust functions that Polars calls directly on Arrow ChunkedArray buffers — bypassing Python entirely for the hot loop.

The architecture is three layers:

Python (user code)
    ↓  register_plugin_function()
Polars expression engine
    ↓  Arrow ChunkedArray
Rust (maskops core)
    ↓  regex::Regex on &str slices

Each PII type lives in its own Rust module (iban.rs, vat.rs) with a compiled once_cell::Lazy<Regex> — the regex is compiled once at startup, not per row.

// Rust side — called directly by Polars on each string slice
#[polars_expr(output_type=String)]
fn mask_pii(inputs: &[Series]) -> PolarsResult<Series> {
    let ca = inputs[0].str()?;
    let out: StringChunked = ca.apply(|opt_val: Option<&str>| {
        opt_val.map(|s| std::borrow::Cow::Owned(mask_all(s)))
    });
    Ok(out.into_series())
}

Supported PII Patterns (v0.1.0)

Pattern	Coverage	Example
IBAN	All 36 SEPA countries	`DE89370400440532013000` → `DE89******************`
EU VAT	All 27 EU member states	`DE123456789` → `DE*********`

Tested against Faker-generated data in 8 EU locales: DE, FR, ES, IT, NL, PL, PT, SE.

Why Not Just Use Polars `.str.replace()`?

You could write pl.col("x").str.replace_all(pattern, "****") directly in Polars. The problem:

You need one expression per PII type — maskops applies all patterns in a single pass.
No detection — Polars has no contains_pii() equivalent without writing the regex yourself.
No masking logic — mask_pii preserves the IBAN country code and check digits, which is standard practice for audit trails. A raw str.replace_all would wipe everything.

Roadmap

v0.1.1: Email, phone number, IP address patterns
v0.1.2: Format-Preserving Encryption (FPE/FF3-1) for reversible masking + PyPI publish
v0.2.0: Latin American IDs (Chilean RUT, Brazilian CPF, Mexican CURP)

Install & Getting Started

pip install maskops

import polars as pl
import maskops

df = pl.DataFrame({
    "transaction": [
        "Payment from DE89370400440532013000",
        "Invoice VAT: DE123456789",
        "No PII here"
    ]
})

result = df.with_columns([
    maskops.mask_pii("transaction").alias("masked"),
    maskops.contains_pii("transaction").alias("has_pii")
])

print(result)

Output:

┌─────────────────────────────────────┬──────────────────────────────────┬─────────┐
│ transaction                         ┆ masked                           ┆ has_pii │
╞═════════════════════════════════════╪══════════════════════════════════╪═════════╡
│ Payment from DE89370400440532013000 ┆ Payment from DE89*************** ┆ true    │
│ Invoice VAT: DE123456789            ┆ Invoice VAT: DE*********         ┆ true    │
│ No PII here                         ┆ No PII here                      ┆ false   │
└─────────────────────────────────────┴──────────────────────────────────┴─────────┘

Source code: github.com/fcarvajalbrown/MaskOps

Built with Rust, pyo3-polars, and maturin. Contributions welcome.

Tags: #rust #python #polars #gdpr #dataengineering #privacy #pii #opensource

DEV Community: Felipe Carvajal Brown

PII masking in Polars: MaskOps 2.0, and two metrics that lied to me

The first number lied against me: the benchmark

The second number lied for me: the downloads

What I take from this

I dropped this for three months. Here's what I added when I came back.

I dropped this for three months. Here's what I added when I came back.

The problem with masking alone

What mask_pii_audit does

One pass

A daily audit pattern

Where it stands

MaskOps 0.1.0: A Native Polars Plugin for High-Speed PII Masking in Python

The Problem

The Solution: maskops

Benchmarks

maskops throughput

vs pure Python regex (same machine)

vs Microsoft Presidio (estimated)

How It Works

Supported PII Patterns (v0.1.0)

Why Not Just Use Polars .str.replace()?

Roadmap

Install & Getting Started

Why Not Just Use Polars `.str.replace()`?