DEV Community

Cover image for Data De-identification for LLMs: Protecting Privacy While Preserving Meaning
Nico Bistolfi
Nico Bistolfi

Posted on

Data De-identification for LLMs: Protecting Privacy While Preserving Meaning

#ai #discuss #go #showdev

TL;DR: deidentify is a zero-dependency Go library that removes PII from your data while preserving format and meaning. Same input = same output, making it perfect for LLM preprocessing. Star it on GitHub if you find it useful!

The Hidden Risk in Your AI Pipeline

Every time you send customer data to an LLM, you're making a trust decision. Whether it's GPT-4, Claude, or your company's internal model, that data leaves your control. And if it contains names, emails, SSNs, or credit card numbers, you're one data breach away from a nightmare.

Imagine the following scenario: you're looking to use LLMs to analyze customer support tickets. Great idea, until legal asks: "What happens to the customer data we're sending to OpenAI?"

Why Traditional Redaction Falls Short

The obvious solution is to redact everything. Replace all names with [REDACTED], all emails with [EMAIL], and so on. But here's the problem:

Original: "John Smith emailed john.smith@company.com about his order"
Redacted: "[NAME] emailed [EMAIL] about his order"
Enter fullscreen mode Exit fullscreen mode

The LLM loses critical context. Was it the same person? Different people? The relationships between entities disappear, making analysis nearly useless.

Preserving Meaning While Protecting Privacy

What you need is consistent, deterministic replacement. Same input should always produce the same output, so relationships stay intact:

Original: "John Smith emailed john.smith@company.com, then Jane Doe replied"
Better: "Robert Johnson emailed robert.j@demo.com, then Sarah Miller replied"
Enter fullscreen mode Exit fullscreen mode

Now the LLM understands that the email belongs to the first person, not the second. The data remains useful for analysis while protecting actual identities.

Enter deidentify: A Go Solution

After facing this problem repeatedly, we built deidentify - a Go library that handles this automatically. What makes it special:

  1. Zero dependencies - Just Go's standard library. No supply chain risks.
  2. Deterministic - Same secret key + same input = same output, always.
  3. Format preserving - Phone numbers look like phone numbers, emails like emails.
  4. Context aware - Uses column names to prevent correlation across fields.

Here's what it looks like in practice:

secretKey, _ := deidentify.GenerateSecretKey()
d := deidentify.NewDeidentifier(secretKey)

text := "Contact Alice at alice@startup.com or 555-123-4567"
safe, _ := d.Text(text)
// Output: "Contact Taylor at member4921@demo.co or 555-642-8317"
Enter fullscreen mode Exit fullscreen mode

Real-World Example: Customer Support Analysis

At my current company, we process thousands of support tickets through LLMs for sentiment analysis and categorization. Before deidentify:

Ticket: "Hi, I'm Bob Wilson (bob@example.com). My SSN 123-45-6789
was exposed when your system crashed. Please call me at 555-0123."

Risk: Bob's entire identity is sent to an external API.
Enter fullscreen mode Exit fullscreen mode

After deidentify:

Ticket: "Hi, I'm Michael Davis (user7823@demo.org). My SSN 847-92-3651
was exposed when your system crashed. Please call me at 555-7492."

Result: LLM can still analyze the severity (SSN exposure) without seeing real data.
Enter fullscreen mode Exit fullscreen mode

Why Go Matters Here

You might wonder why we built this in Go. Three reasons:

  1. Performance - De-identifying gigabytes of data needs to be fast
  2. Deployment - Single binary, no runtime dependencies
  3. Safety - Strong typing catches PII-type mismatches at compile time

The fact that it uses only Go's standard library means you can audit the entire codebase without chasing dependencies. Critical for security-conscious teams.

Implementation Tips

When integrating de-identification into your LLM pipeline:

  1. De-identify early - Before data hits your message queue or API
  2. Keep your key safe - The secret key is what makes replacements consistent
  3. Test with production-like data - PII patterns vary by industry

Example pipeline:

// Step 1: Load customer data
data := loadCustomerData()

// Step 2: De-identify
safe := d.Table(data)

// Step 3: Send to LLM
response := llm.Analyze(safe)

// Step 4: Process results (no re-identification needed)
Enter fullscreen mode Exit fullscreen mode

Take Advantage

We made deidentify open source because privacy tools should be transparent. You can inspect every line of code, understand exactly how it works, and even contribute improvements.

The library handles:

  • Names, emails, phone numbers
  • SSNs, credit cards, addresses
  • Structured data (CSV, database exports)
  • International formats (100+ address patterns)

If you're sending any data to LLMs, check out deidentify on GitHub. And if it saves you from a data breach, consider giving it a star 🌟 - it helps others find the tool.

Top comments (1)