Mask, Hash, Tokenize, or Encrypt? Choosing the Right PII Protection for Your Pipeline

In Part 1 you built a map: every column that holds PII, tagged with a tier. Feels like the hard part is over. Then someone in #data-platform asks the reasonable question:

"OK so... do we hash the emails? Or mask them? I feel like hashing. Let's hash them."

"Hash it" is the two-word answer that's almost always wrong in an interesting way. Not because hashing is bad — because the question wasn't "which algorithm," it was "what do we still need this data to do?" This article is about answering that question deliberately. All code is in the companion repo — every claim below has a runnable file behind it.

One map before the weapons

Every technique fits somewhere on one axis that matters more than people admit: can the original value be recovered?

The rule of thumb hiding in this diagram: if a key or vault can recover the person, you are still holding personal data. Encryption and tokenization buy you access control. Masking, hashing, and generalization buy you distance. Sometimes you need one, sometimes the other, sometimes both on the same column in different zones. Let's earn that sentence.

Masking: for when nobody needs the original

# src/pii/masking.py
def mask_email(value: str) -> str:
    """[e-mail] -> j***@example.com (irreversible, display-safe)."""
    user, _, domain = value.partition("@")
    return f"{user[:1]}***@{domain}" if user and domain else "***"

Masking is display logic. Partial masks keep just enough for humans to recognize ("yeah that's my email"), and the original value is gone — there is no key. Static masking rewrites the data before it lands somewhere broad (analytics, test databases); dynamic masking evaluates per-user at query time (Snowflake masking policies, BigQuery column-level security — the warehouse layer, not your Python).

Great for: support agents verifying identity by last-4 digits, dashboards, test data.
Wrong for: anything downstream that needs to join, count distinct, or reach back out to the person.

Hashing: the trap everyone walks into

Hashing feels like privacy alchemy: deterministic, irreversible, join-friendly. And for high-entropy secrets (passwords) it genuinely is. PII is not high-entropy. A US phone number has 10 billion possibilities — precomputable. An SSN: a billion. A gender-plus-ZIP combo: forget it. When the input space is small, an attacker doesn't reverse your hash; they hash the whole dictionary and look it up.

This is not hypothetical — the repo demonstrates it (uv run python -m pii.hashing_demo):

leaked: 5 phone numbers, each appearing twice (as if in two tables)
attacker: precomputed dictionary of 20,000 candidate phones

1. sha256(phone)
   cracked by dictionary : 5/5
   same phone, same digest (joins work): YES — for the attacker too

2. sha256(salt + phone)
   cracked by dictionary : 0/10
   same phone, same digest (joins work): NO — broken for everyone

3. hmac(key, phone)
   cracked by dictionary : 0/5 (key unknown)
   same phone, same digest (joins work): YES — attacker needs the key

Three lines, one entire decision framework.

The join-key dilemma

Here's the tension that actually decides your architecture:

Analytics needs deterministic identifiers — same email in the events table and the orders table must produce the same key, or joins, distinct counts, and funnels all die. Security needs outputs nobody can precompute. Plain hashing gives you the first and forfeits the second. The textbook fix — a random salt per row — gives you the second and forfeits the first: the same phone now hashes differently everywhere it appears, and COUNT(DISTINCT user) quietly becomes nonsense.

HMAC: the workhorse

# src/pii/masking.py
def pseudonymize(value: str, key: bytes) -> str:
    """HMAC-SHA256, truncated. Deterministic (joins survive), keyed."""
    return hmac.new(key, value.encode(), hashlib.sha256).hexdigest()[:16]

HMAC is hashing with a secret key baked into the function. Same input + same key → same digest, so your joins live. But an attacker with a dictionary and no key gets nothing — their precomputed table is useless against a keyed construction. It's not magic, it's a trade-off with one new obligation: key management. The key lives in a secrets manager, reaches the pipeline as PII_HMAC_KEY, and — the part teams forget — needs a rotation story. Version your tokens (v2:a1b2c3...) so rotating the key doesn't orphan every join key you've ever minted. The trade: anyone holding the key can map every pseudonym back — so the key's blast radius is your privacy posture.

For most analytics pipelines this is the default answer for direct identifiers: deterministic, irreversible without the key, and good enough to build on.

Tokenization: when the business needs to reach back

Sometimes "irreversible" is a deal-breaker — support genuinely needs to email the customer back. Tokenization swaps the PII for a token and stores the mapping in a vault:

token          email
──────────────────────────────────────
0138f3a7b71b…  [e-mail]     ← vault.duckdb, max lock-down

The pipeline and analytics only ever see the token; reversal is an explicit, audited vault lookup behind an authorization boundary. Tokens can be deterministic (same input → same token → joins survive, same trade as HMAC) or random (unlinkable, but unjoinable). And this is the shape PCI-DSS expects for card data, because "we never store the PAN, only the token" is a sentence auditors enjoy.

The honest cost: the vault becomes your crown-jewel database. Compromise it and every token in every table becomes plaintext. Guard, audit, and back it up like it's the one file with everyone's names in it — because it is.

Encryption: access control, not anonymity

Encryption is non-negotiable and overrated, in the same way locks are: essential, but nobody claims a locked house contains no furniture. If a key can decrypt the column — and at query time something usually can — then GDPR still sees personal data, your breach blast radius still includes those columns, and your erasure obligations still apply. Encrypt everything at rest and in transit, use column-level encryption for the spicy fields, and file it under securing PII, not reducing it. The techniques above reduce; encryption just locks the door.

The legal line: pseudonymized vs. anonymous

The distinction that ends arguments in incident reviews:

Under GDPR, a hashed email is pseudonymous, not anonymous — it's still personal data, because possession of the key (or a matching hash elsewhere) re-links it. Retention limits and right-to-erasure apply in full. The only road to "anonymous" runs through the quasi-identifiers from Part 1: generalize zip → 941, dob → 1974, suppress rare values, until no combination singles anyone out (k-anonymity). That's why Part 1's tier tags mattered: direct identifiers get HMAC or tokens; quasi-identifiers get generalization. Different tiers, different weapons.

One caveat worth knowing before you treat k-anonymity as a finish line: it protects against singling someone out, not against learning something about them. If every one of the k records sharing a generalized ZIP+birth-year bucket happens to have the same sensitive attribute — the same diagnosis, say — an attacker doesn't need to identify the individual to learn the fact, because the whole bucket shares it (the textbook "homogeneity attack"). k-anonymity is the regulatory bar GDPR actually recognizes, and it's the right target for this series, but if a quasi-identifier bucket is going to sit next to a Tier 3 sensitive attribute, it's worth checking the bucket isn't accidentally uniform on that attribute — that's what extensions like l-diversity exist to catch.

The decision table

Screenshot this one:

Requirement driving the choice	Masking	HMAC / hashing	Tokenization	Encryption	Generalization
Analytics must join / count distinct on it	no	yes	yes (deterministic tokens)	no	partially
Business must recover the original	no	no	yes (vault)	yes (key)	no
Survives without key/vault management	yes	no	no	no	yes
Format must stay valid (sort keys, regex)	partial	no	yes (FPE tokens)	no	no
Removes GDPR obligations on the dataset	no	no	no	no	yes (k-anonymity)
Typical home	display & serving layer	direct identifiers in analytics	payment & operational data	storage & transit	quasi-identifiers

Most real pipelines are a composition: encrypt the raw zone, HMAC the direct identifiers into curated, tokenize what support must reach, mask at the serving layer, generalize the quasi-identifiers. Part 3 builds exactly that.

Takeaways

• "Hash it" is a question about what the data still needs to do, not an algorithm choice.
• Low-entropy PII (phones, SSNs, ZIPs) falls to dictionary attacks unless the hash is keyed — use HMAC.
• Salts protect passwords, not join keys. Per-row salts and analytics joins are mutually exclusive.
• Reversibility is access control, not anonymity. The key and the vault are personal data by proxy.
• GDPR calls everything except k-anonymized data "personal data." Plan retention and erasure accordingly.

Next in the series: we take the tier tags from Part 1 and the technique choices from Part 2, and build the actual pipeline — raw zone, curated zone, vault, role-based serving views, GDPR erasure across all of them, and the CI test that fails your build when PII leaks. All runnable, all local.