DEV Community

Alessio
Alessio

Posted on Edited on Originally published at py-forge.com

Mask PII in Logs, CSV and JSON with Python (and Keep the Data Joinable)

#privacy #security #logging #python

Somebody needs a copy of production data to debug an issue, or a support ticket export has to go to a contractor, or your application logs are about to be shipped to a third-party service. Each time, emails, card numbers and phone numbers go along unless you strip them first. DataMask is a single Python file, standard library only, that masks that kind of PII in plain text, CSV and JSON. With a secret key it can replace values with stable tokens, so the masked data can still be joined and counted.

Everything below was run locally with the shipped datamask.py. The outputs are copied as-is.

What it detects

The built-in rules are regular expressions for:

Type Example input Default output
Email jane.doe@example.com j******e@example.com
Card (16 digits, with or without separators) 4111 1111 1111 1111 ****-****-****-1111
US SSN 123-45-6789 [SSN_REDACTED]
North American phone 415-555-0132 (415) ***-0132
IPv4 192.168.10.24 192.168.0.0
key=value secrets (api_key, secret, token) with a value of 16+ chars api_key=sk_test_51Hx… [GENERIC_SECRET_REDACTED]

Masking free text (logs, tickets, notes) 

printf 'Contact jane.doe@example.com or call +1 (415) 555-0132.\nCard 4111 1111 1111 1111, SSN 123-45-6789, IP 192.168.10.24\n' > in.txt
python3 datamask.py -i in.txt -k demo-key
Enter fullscreen mode Exit fullscreen mode 
Contact j******e@example.com or call +(415) ***-0132.
Card ****-****-****-1111, SSN [SSN_REDACTED], IP 192.168.0.0
Enter fullscreen mode Exit fullscreen mode

The defaults keep enough to stay useful: the email domain, the last four digits of the card, the phone's area code and the /16 of the IP. To mask everything with one strategy, pass --strategy:

--strategy REDACT
Contact [EMAIL_REDACTED] or call +[PHONE_REDACTED].
Card [CREDIT_CARD_REDACTED], SSN [SSN_REDACTED], IP [IPV4_REDACTED]

--strategy TOKENIZE
Contact TOK_16978AC475DBF78A or call +TOK_407080C9ACCD2486.
Card TOK_7B8FCD949B6A4A3A, SSN TOK_39D12AE6849AA1EB, IP TOK_5052DA002CECEAC6
Enter fullscreen mode Exit fullscreen mode

It also reads stdin and writes stdout, so it fits in a pipe:

tail -f app.log | python3 datamask.py -k "$DATAMASK_KEY" >> app.masked.log
Enter fullscreen mode Exit fullscreen mode

Tokens you can still join on

TOKENIZE isn't random. It's an HMAC-SHA256 of the value under your key, truncated to 16 hex characters:

def tokenize(self, value: str) -> str:
    token = hmac.new(self.secret_key, value.encode("utf-8"), hashlib.sha256).hexdigest()
    return f"TOK_{token[:16].upper()}"
Enter fullscreen mode Exit fullscreen mode

The same input with the same key always gives the same token, whether you pass the key with -k or through the DATAMASK_KEY environment variable:

echo "ada.park@example.com" | python3 datamask.py -k team-secret --strategy TOKENIZE
TOK_58D0726A2F40DC76
echo "ada.park@example.com" | DATAMASK_KEY=team-secret python3 datamask.py --strategy TOKENIZE
TOK_58D0726A2F40DC76
Enter fullscreen mode Exit fullscreen mode

So an analyst can still count orders per customer, or join the masked users table to the masked orders table, without ever seeing an address. Without a key, DataMask generates a random one for each run, so tokens won't match across runs. Keep the key in your secrets manager, not next to the data.

CSV with per-column rules

For structured files you choose a strategy per column in a small JSON file:

{"email": "TOKENIZE", "name": "REDACT", "card": "PARTIAL", "phone": "PARTIAL"}
Enter fullscreen mode Exit fullscreen mode 
cat users.csv
id,name,email,phone,card,notes
1,Ada Park,ada.park@example.com,415-555-0101,4242 4242 4242 4242,VIP since 2021
2,Luis Ortega,luis@example.org,212-555-0199,5555-5555-5555-4444,called from 10.0.4.17

python3 datamask.py -f csv -i users.csv --rules rules.json -k team-secret
Enter fullscreen mode Exit fullscreen mode 
id,name,email,phone,card,notes
1,[CONFIDENTIAL],TOK_58D0726A2F40DC76,(415) ***-0101,****-****-****-4242,VIP since 2021
2,[CONFIDENTIAL],TOK_9E2DBA2E53D1D11F,(212) ***-0199,****-****-****-4444,called from 10.0.0.0
Enter fullscreen mode Exit fullscreen mode

Columns without a rule still go through the text scanner, which is why the IP buried in notes came out as 10.0.0.0. The CSV is processed row by row with csv.DictReader, so large files don't need to fit in memory.

JSON, at any depth 

echo '{"user":{"id":7,"email":"ada.park@example.com","note":"reach me at 415-555-0101"},"orders":[{"card":"4242424242424242"}]}' > ev.json
python3 datamask.py -f json -i ev.json -k team-secret --rules rules.json
Enter fullscreen mode Exit fullscreen mode 
{
  "user": {
    "id": 7,
    "email": "TOK_58D0726A2F40DC76",
    "note": "reach me at (415) ***-0101"
  },
  "orders": [
    {
      "card": "****-****-****-4242"
    }
  ]
}
Enter fullscreen mode Exit fullscreen mode

The rules match on key names at any depth, and every other string value is scanned. Numbers and booleans are left alone. Notice that the email token is identical to the one in the CSV: same key, same value, same token.

Using it from Python 

from datamask import DataMaskEngine, MaskingStrategy

engine = DataMaskEngine(hmac_key="team-secret")
print(engine.mask_text("refund to jane.doe@example.com"))
# refund to j******e@example.com
clean = engine.mask_structure(payload, {"email": MaskingStrategy.TOKENIZE})
Enter fullscreen mode Exit fullscreen mode

That makes it easy to wire into a logging.Filter or a serializer before data leaves the process.

How it works

Each rule is a compiled regex, a default strategy and an optional format-preserving replacer. mask_text applies them in order:

for rule in self.rules:
    strategy = strategy_override or rule.default_strategy
    if strategy == MaskingStrategy.PARTIAL and rule.custom_replacer:
        sanitized = rule.pattern.sub(rule.custom_replacer, sanitized)
    elif strategy == MaskingStrategy.TOKENIZE:
        sanitized = rule.pattern.sub(lambda m: self.tokenize(m.group(0)), sanitized)
    ...
Enter fullscreen mode Exit fullscreen mode

mask_structure walks dicts and lists recursively. Keys with a rule get that strategy; other strings go through mask_text.

Limitations

  • It matches patterns, it doesn't understand language. Names and street addresses in free text aren't detected. "Ada Park" stays as it is unless it sits in a column or key you've given a rule. Use field rules for structured data.
  • The phone and SSN rules are North American formats. Other countries' numbers and 15-digit Amex cards aren't covered by the default rules.
  • The secret rule looks for api_key=, secret: and token= style pairs and Bearer <token> values (for example Authorization: Bearer … headers), with tokens of 16+ characters. Other credential formats need a custom rule.
  • HASH is a plain SHA-256 with no key. For low-entropy values such as phone numbers, use TOKENIZE (keyed) instead.

FAQ

Is this GDPR/HIPAA compliance? No tool makes you compliant by itself. DataMask helps with data minimisation and pseudonymisation; whether that's sufficient depends on your data and your process.

Does it need dependencies? No, it's standard library only.

Can I add my own patterns? Yes. The rules are a list of PIIRule(name, sensitivity, pattern, default_strategy, custom_replacer) objects in the source, so you can append your own (an internal customer-ID format, for example).

DataMask is a one-time purchase that includes the source and README: get DataMask ($19). More details are on the DataMask product page.

Top comments (0)