DEV Community: Felipe Carvajal Brown

Format-preserving encryption for PII in Polars: FF3-1 vs FF1 for RUT, CPF, and DNI

Felipe Carvajal Brown — Fri, 03 Jul 2026 00:43:18 +0000

You need to hand a dataset of Chilean RUTs to an outside analytics team. They will join it against other tables by identifier, run the cohort analysis, and hand back a model. They do not need to know, and should never learn, who any of these people are. Asterisk the RUT column and the join dies on contact: **********-K matches every other asterisked RUT in the file. Not almost every one. Every one. You need the same input to reappear as the same output, shaped like a real, check-digit-valid identifier the rest of your schema still recognizes, and eight weeks later, when a fraud investigator needs the original RUT back for one row, you need to be able to give it to them.

Irreversible masking cannot do any of this. Hashing gets you consistency but not the format, and never the value back. What you need is format-preserving encryption: run a digit string through a cipher and get out another digit string, same length, same shape, that decrypts to the original under the key you hold. Nothing else.

What FPE actually does

MaskOps exposes this as mask_pii_fpe. It masks digit-based PII, cards, phones, RUT, CPF, Argentine DNI, in place, and gives back something the same length and shape:

import maskops
import secrets

key = secrets.token_bytes(32)     # AES-256, client holds this
tweak = secrets.token_bytes(7)    # per-column/per-dataset context

df.with_columns(maskops.mask_pii_fpe("rut_column", key, tweak))

76.354.771-K becomes some other RUT-shaped, check-digit-valid string of the same length, under this key and tweak. Run it back through with the same key and tweak and it decrypts. Non-digit PII, IBAN, VAT, email, IP, EU national IDs, gets none of this. It always asterisks. There is no clean digit domain to encrypt into, so MaskOps does not pretend there is.

The key never touches MaskOps' output. The client generates it, holds it, and passes it in at call time, and because MaskOps makes no network call and keeps no storage layer, there is nowhere for that key to leak to even if someone wanted it to. That separation is not a nice-to-have sitting on top of the design. It is the design. GDPR Article 4(5) defines pseudonymization as data that cannot be re-attributed to a person "without the use of additional information," provided that information "is kept separately." The key is the additional information. Keep it separate and the column is pseudonymized. Store it next to the data, and it is just encryption wearing a compliance label.

Say it plainly, because it gets glossed over in every vendor deck that sells this as a compliance checkbox: FPE output is still personal data. It is reversible by construction, so anyone holding the key can undo it, and that makes it pseudonymization under GDPR, not anonymization. Is that a weaker claim than "anonymized"? Yes. It is also the true one. Anonymous data falls outside GDPR's scope entirely. Pseudonymized data does not. If you need the stronger claim, use asterisk masking instead, and give up the join.

Two ciphers, one decision NIST is still making

mask_pii_fpe takes a mode argument: "ff3", the default, or "ff1". Both are Feistel-network constructions over AES-256. Both are reversible. Both preserve length and format. The difference between them is not cryptographic strength today. It is which one NIST still stands behind tomorrow.

Give FF3-1 its due first. NIST SP 800-38G approved it in 2016 alongside FF1. A later draft narrowed it to FF3-1 by shrinking the tweak from 64 bits to 56 and raising the minimum domain. It is what MaskOps has supported the longest, and it is not broken today. Then, in the second public draft of SP 800-38G Revision 1, published in February 2025, NIST proposed removing FF3 and FF3-1 outright, citing a 2021 cryptanalytic result by Beyne. That draft has not been finalized as of this writing, so FF3-1 has not been formally withdrawn. But the direction is not a rumor. FF1 is the mode NIST intends to keep, and FF3-1 is the mode NIST intends to retire.

What does that attack actually require? Not a break of AES, and not key recovery. It is a distinguishing and message-recovery attack against the FF3-1 Feistel structure, and it needs a large number of known or chosen plaintext and ciphertext pairs, all under one fixed key and tweak, over one small domain. Shrink the domain or grow the volume under a single tweak, and the attack gets cheaper. Encrypt a handful of RUTs under a dedicated tweak, and you are nowhere near its reach. Encrypt millions of rows of a six-digit field under one tweak forever, and you are exactly the case NIST is warning about.

MaskOps' position is the one a security team doing diligence wants stated out loud, not discovered in a footnote: FF3-1 is not broken. It is scheduled, and FF1 is where anything new should land. rekey_pii_fpe exists to move a column from one to the other, decrypting under the old key and mode and re-encrypting under the new one in a single pass, so the plaintext never has to sit still as a materialized column while you switch.

Why the domain minimum is not a formality

Both FF1 and FF3-1 inherit a minimum-domain rule from SP 800-38G: radix raised to the minimum length must be at least 1,000,000. In base 10, why six digits and not five? Because below that floor, the space of possible values is small enough that FPE stops being a meaningfully strong cipher, and small domains are exactly what makes the Beyne attack cheap. MaskOps enforces this in the Rust core, not in a comment somebody will forget to update:

pub(crate) const MIN_LEN: usize = 6;
pub(crate) const MAX_LEN: usize = 30;

Feed it a five-digit value and it refuses, for both modes, rather than handing back weak ciphertext and letting you find out the hard way.

Here is where the LatAm identifier set earns its keep as a worked example, because the three families do not hit that floor the same way. A Chilean RUT is a seven-or-eight-digit body plus one check digit (76.354.771-K, where K is the check digit), and MaskOps encrypts only the body, leaving the check digit in cleartext so the output still validates as a RUT. An Argentine DNI (12.345.678) carries no separate check digit, so the whole seven-or-eight-digit run gets encrypted. A Brazilian CPF (529.982.247-25) is eleven digits encrypted whole. All three clear the floor comfortably. A four-digit PIN or a short internal sequence number would not, and that is exactly the case MIN_LEN exists to refuse, not to work around.

What this buys you

mask_pii_fpe answers one narrow, common question: how to anonymize PII in Python pipelines without breaking joins across tables or downstream format validation, and without losing the ability to recover the original value with a key you control. Call it PII masking in Polars, not a script bolted on from outside: the cipher runs inside the same expression engine already scanning the dataframe. If you have looked at Presidio for this job, know what it is and is not. Presidio does named-entity recognition over free text. It does not run natively as a Polars expression, and it does not ship check-digit-validated RUT, CPF, or CNPJ detection at all. MaskOps is a Presidio alternative for structured columns with schema-defined PII: RUT / CPF detection with real Módulo 11 validation, running as a native Rust expression with no network call anywhere in the path.

None of this replaces a lawyer or a data protection officer telling you your legal basis for processing personal data. It replaces the part of the compliance story that was always a data-engineering problem to begin with: getting a reversible, honestly-labeled pseudonymization pass onto a column without shipping the plaintext anywhere. FF3-1 is not broken. It is scheduled. Encrypt what is new under FF1, and decide once, not twice.

MaskOps is open source, MPL-2.0, on PyPI and GitHub.

PII masking in Polars: MaskOps 2.0, and two metrics that lied to me

Felipe Carvajal Brown — Thu, 11 Jun 2026 04:27:39 +0000

MaskOps 2.0 shipped this week. Before I told anyone, I looked at my own numbers. Two of them were lying to me, in opposite directions.

MaskOps is a Rust plugin for Polars that does PII masking inside the dataframe: RUT, CPF, credit cards, IBANs, and twenty-odd more families, air-gapped, with no network call, ever. If you have reached for Microsoft Presidio and found it carries no Latin American identifiers, that is the gap MaskOps fills. It does check-digit-validated RUT, CPF, and CURP detection alongside the EU, US, and APAC families, as a native Polars expression. Version 2.0 is the enterprise line: configurable patterns, structured extraction, an audit pass that counts what it masked, and format-preserving encryption (GDPR Art. 4(5) pseudonymization) for the reversible cases. That part I was sure of. The numbers around it, less so.

The first number lied against me: the benchmark

The last thing I checked was the benchmark table in my own README. It said MaskOps ran at 0.4× to 0.7× the speed of plain Python re. Slower than the language I wrote it to replace.

I almost opened the profiler. Instead I read the benchmark harness. I should have read it first.

Here is what it did. For every family, "Credit Card", "EU", "LatAm", it ran the full masker. All thirty-five pattern families at once. Then it compared the time against a Python baseline that ran one regex for that family.

So the "Credit Card" row timed MaskOps scanning for cards, phones, IBANs, Korean RRNs, and thirty others, against Python scanning for cards. The proof sat in the table the whole time: every MaskOps row took the same 2.3 seconds regardless of family, because it always did all the work. Only the Python column moved.

I was timing my engine doing thirty-five times the work and calling it slow.

Two fixes. The first was the benchmark, not the code: compare like-for-like. When the row says "Credit Card", mask credit cards, the same job the baseline does. MaskOps already supports selection, so it was one argument: mask_pii("text", patterns=["credit_card"]).

The second was real. Most rows in real data contain no PII. Every pattern MaskOps detects needs a digit, or an @. A row with neither cannot match anything. So before any regex, walk the bytes:

pub fn has_pii_candidate(value: &str) -> bool {
    value.bytes().any(|b| b.is_ascii_digit() || b == b'@')
}

If false, return the string untouched. On clean text this skips all thirty-five scans for the price of one pass over the bytes. Output does not change. The same 394 tests pass.

PII masking in Polars, measured fairly. One million rows, median of three, against a pure-Python re baseline with matching coverage:

Data profile	Speedup vs Python
clean (no PII)	11×–163×
mixed (50% PII)	1.2×–3.2×
dense (every row)	1.3×–2.7×

One family still loses on dense data. The European ID set runs four separate regex passes, and a single combined Python regex edges it out, 0.9×. I left that in the README. A table with no losses is a table someone tuned until it lied.

The second number lied for me: the downloads

The other number was downloads. I shipped the 1.7 through 2.0 releases in one short burst, and the PyPI counter jumped from about ten a day to three and a half thousand on release day. A hundredfold, overnight.

It would be easy to write "downloads are exploding." It would also be false.

That spike sits exactly on the days I pushed releases. It is CI building wheels across the OS and Python matrix, mirrors syncing, bots crawling each new version. PyPI counts all of it. None of it is a person deciding to use the thing. Strip the release days and the real line is flat and small. Single digits, which is the honest state of a young project.

So I am not going to tell you adoption is taking off. The download number is real and it is mostly noise, and pretending otherwise insults anyone who can open the same pypistats page I did.

What I take from this

Two instruments. One read low because it measured the wrong thing. One read high because it counted the wrong things. A metric is not a verdict. It is a measurement, and a measurement can be miscalibrated in your favor or against it, and you owe it to yourself to know which.

Read the harness before the flame graph. Strip the release days before you celebrate the downloads. Then trust what is left.

MaskOps is open source, MPL-2.0, on PyPI. It does PII masking inside Polars, air-gapped, with check-digit validation so a random nine-digit number is not mistaken for an ID. It does not do named-entity recognition. The source and the benchmark code are on GitHub. Run it. If your machine disagrees with mine, I want to know.

I dropped this for three months. Here's what I added when I came back.

Felipe Carvajal Brown — Mon, 08 Jun 2026 16:28:44 +0000

I dropped this for three months. Here's what I added when I came back.

I started MaskOps in March. It masks PII in Polars DataFrames using Rust — no Python per row, no NLP models, just regex running on Arrow buffers.

Then I got hired. Cencosud S.A. The project sat untouched until last week.

Coming back to it, I had a backlog. I shipped the one I kept thinking about at work: mask_pii_audit.

The problem with masking alone

Masking answers "is this field safe to store?" It doesn't answer "what kind of PII just came through, and how much of it?"

Compliance teams need both. They need the masked value and a count of what was found — by family — without running the column twice.

What mask_pii_audit does

It returns a nested Struct: the masked text, plus a count for each of the 33 PII families.

import polars as pl
import maskops

df = pl.DataFrame({"notes": [
    "Call me at 555-123-4567. SSN: 123-45-6789.",
    "IBAN: DE89370400440532013000",
    "Nothing here.",
]})

result = (
    df
    .with_columns(maskops.mask_pii_audit("notes").alias("audit"))
    .unnest("audit")
)

print(result.select("masked", "counts"))

┌───────────────────────────────┬────────────────────────────┐
│ masked                        ┆ counts                     │
╞═══════════════════════════════╪════════════════════════════╡
│ Call me at ***-***-****. SSN… ┆ {"phone": 1, "ssn": 1, …} │
│ IBAN: DE89******************  ┆ {"iban": 1, …}             │
│ Nothing here.                 ┆ {"phone": 0, "ssn": 0, …} │
└───────────────────────────────┴────────────────────────────┘

Same masked output as mask_pii. Zero fields mean no match.

One pass

The counting happens inside the existing replace_all call. A Cell<u32> in the closure increments on each validated match. No second scan, no cloned strings.

pub fn replace_counted<F>(re: &Regex, s: &str, render: F) -> (String, u32)
where F: Fn(&Captures) -> Option<String> {
    let count = Cell::new(0u32);
    let out = re.replace_all(s, |caps: &Captures| match render(caps) {
        Some(masked) => { count.set(count.get() + 1); masked }
        None => caps[0].to_string(),
    }).into_owned();
    (out, count.get())
}

A daily audit pattern

(
    df
    .with_columns(maskops.mask_pii_audit("free_text").alias("audit"))
    .unnest("audit")
    .select(
        pl.col("counts").struct.field("ssn").sum().alias("ssn_total"),
        pl.col("counts").struct.field("credit_card").sum().alias("cc_total"),
        pl.col("counts").struct.field("iban").sum().alias("iban_total"),
    )
)

Run this at ingest. Log the totals. Alert if a family appears that shouldn't.

Where it stands

v1.6.0. 33 PII families: EU IDs, US healthcare, LATAM nationals, APAC. Asterisk masking and FF3-1 format-preserving encryption. Polars lazy and streaming supported.

pip install maskops

Source: github.com/fcarvajalbrown/MaskOps

Happy to answer questions.

MaskOps 0.1.0: A Native Polars Plugin for High-Speed PII Masking in Python

Felipe Carvajal Brown — Mon, 09 Mar 2026 03:55:40 +0000

TL;DR: I built a Rust-powered Polars plugin that masks GDPR-sensitive data (IBAN, EU VAT) at up to 16 million rows per second — no NLP models, no spaCy, no Presidio overhead. pip install maskops.

The Problem

If you work with financial data, healthcare records, or any GDPR-regulated dataset in Python, you've likely hit the same wall: de-identifying structured data at scale is painfully slow.

The go-to solution is Microsoft Presidio. It's powerful, but it's built for unstructured text — it spins up a full spaCy NLP pipeline to find a phone number in a CSV column. For structured DataFrames where you already know which columns contain PII, that's enormous overhead:

Presidio with spaCy NER: ~1,000–5,000 rows/s
Presidio with regex-only recognizers: ~10,000–50,000 rows/s
Pure Python re module: ~1,100,000 rows/s

None of these integrate natively with Polars, the fastest DataFrame library in Python.

The Solution: maskops

maskops is a native Polars expression plugin written in Rust. It extends Polars with two new expressions — mask_pii() and contains_pii() — that run directly on Arrow memory buffers with zero Python overhead per row.

import polars as pl
import maskops

df = pl.read_csv("payments.csv")

# Mask all PII in a column
df.with_columns(maskops.mask_pii("notes"))
# "Transfer to DE89370400440532013000" → "Transfer to DE89******************"

# Boolean detection — filter rows containing PII
df.filter(maskops.contains_pii("free_text"))

That's it. No model downloads, no engine initialization, no spaCy.

Benchmarks

Tested on 1,000,000 rows, Intel i-series CPU, Python 3.14, Windows.

maskops throughput

Profile	Expression	Time	Rows/s	MB/s
clean (no PII)	`mask_pii`	0.404s	2,477,599	54.5
clean (no PII)	`contains_pii`	0.169s	5,915,970	130.2
dense (all PII)	`mask_pii`	1.385s	722,104	15.9
dense (all PII)	`contains_pii`	0.059s	16,987,879	373.7
mixed (50/50)	`mask_pii`	0.760s	1,315,407	28.9
mixed (50/50)	`contains_pii`	0.133s	7,498,315	165.0

vs pure Python regex (same machine)

Profile	maskops `mask_pii`	Python `re`	Speedup
clean	0.404s	0.925s	2.3×
dense	1.385s	1.653s	1.2×
mixed	0.760s	1.337s	1.8×

On clean and mixed data maskops is consistently faster. On dense data (every row is a full IBAN) both are regex-bound — the bottleneck is the pattern itself, not Python overhead.

vs Microsoft Presidio (estimated)

Presidio processes structured DataFrames via presidio-structured, which runs a spaCy NLP pipeline per row. Based on community reports and the architecture:

Tool	Throughput (structured data)	Requires NLP model
maskops	~700K–17M rows/s	No
Presidio (regex-only recognizers)	~10–50K rows/s*	No
Presidio (spaCy NER)	~1–5K rows/s*	Yes (250MB+)

* Estimated from community benchmarks and Presidio's own documentation noting it is "not optimized for bulk structured data." Microsoft confirmed no official throughput benchmarks exist.

maskops is purpose-built for structured data pipelines where Presidio's NLP overhead is unnecessary.

How It Works

The key is the Polars expression plugin system, introduced in Polars 0.20. It allows you to register custom Rust functions that Polars calls directly on Arrow ChunkedArray buffers — bypassing Python entirely for the hot loop.

The architecture is three layers:

Python (user code)
    ↓  register_plugin_function()
Polars expression engine
    ↓  Arrow ChunkedArray
Rust (maskops core)
    ↓  regex::Regex on &str slices

Each PII type lives in its own Rust module (iban.rs, vat.rs) with a compiled once_cell::Lazy<Regex> — the regex is compiled once at startup, not per row.

// Rust side — called directly by Polars on each string slice
#[polars_expr(output_type=String)]
fn mask_pii(inputs: &[Series]) -> PolarsResult<Series> {
    let ca = inputs[0].str()?;
    let out: StringChunked = ca.apply(|opt_val: Option<&str>| {
        opt_val.map(|s| std::borrow::Cow::Owned(mask_all(s)))
    });
    Ok(out.into_series())
}

Supported PII Patterns (v0.1.0)

Pattern	Coverage	Example
IBAN	All 36 SEPA countries	`DE89370400440532013000` → `DE89******************`
EU VAT	All 27 EU member states	`DE123456789` → `DE*********`

Tested against Faker-generated data in 8 EU locales: DE, FR, ES, IT, NL, PL, PT, SE.

Why Not Just Use Polars `.str.replace()`?

You could write pl.col("x").str.replace_all(pattern, "****") directly in Polars. The problem:

You need one expression per PII type — maskops applies all patterns in a single pass.
No detection — Polars has no contains_pii() equivalent without writing the regex yourself.
No masking logic — mask_pii preserves the IBAN country code and check digits, which is standard practice for audit trails. A raw str.replace_all would wipe everything.

Roadmap

v0.1.1: Email, phone number, IP address patterns
v0.1.2: Format-Preserving Encryption (FPE/FF3-1) for reversible masking + PyPI publish
v0.2.0: Latin American IDs (Chilean RUT, Brazilian CPF, Mexican CURP)

Install & Getting Started

pip install maskops

import polars as pl
import maskops

df = pl.DataFrame({
    "transaction": [
        "Payment from DE89370400440532013000",
        "Invoice VAT: DE123456789",
        "No PII here"
    ]
})

result = df.with_columns([
    maskops.mask_pii("transaction").alias("masked"),
    maskops.contains_pii("transaction").alias("has_pii")
])

print(result)

Output:

┌─────────────────────────────────────┬──────────────────────────────────┬─────────┐
│ transaction                         ┆ masked                           ┆ has_pii │
╞═════════════════════════════════════╪══════════════════════════════════╪═════════╡
│ Payment from DE89370400440532013000 ┆ Payment from DE89*************** ┆ true    │
│ Invoice VAT: DE123456789            ┆ Invoice VAT: DE*********         ┆ true    │
│ No PII here                         ┆ No PII here                      ┆ false   │
└─────────────────────────────────────┴──────────────────────────────────┴─────────┘

Source code: github.com/fcarvajalbrown/MaskOps

Built with Rust, pyo3-polars, and maturin. Contributions welcome.

Tags: #rust #python #polars #gdpr #dataengineering #privacy #pii #opensource

DEV Community: Felipe Carvajal Brown

Format-preserving encryption for PII in Polars: FF3-1 vs FF1 for RUT, CPF, and DNI

What FPE actually does

Two ciphers, one decision NIST is still making

Why the domain minimum is not a formality

What this buys you

PII masking in Polars: MaskOps 2.0, and two metrics that lied to me

The first number lied against me: the benchmark

The second number lied for me: the downloads

What I take from this

I dropped this for three months. Here's what I added when I came back.

I dropped this for three months. Here's what I added when I came back.

The problem with masking alone

What mask_pii_audit does

One pass

A daily audit pattern

Where it stands

MaskOps 0.1.0: A Native Polars Plugin for High-Speed PII Masking in Python

The Problem

The Solution: maskops

Benchmarks

maskops throughput

vs pure Python regex (same machine)

vs Microsoft Presidio (estimated)

How It Works

Supported PII Patterns (v0.1.0)

Why Not Just Use Polars .str.replace()?

Roadmap

Install & Getting Started

Why Not Just Use Polars `.str.replace()`?