GDPR vs CCPA vs India DPDP: Building Multi-Jurisdiction Privacy Pipelines

multi-jurisdiction privacy pipelines are what you build when the same row of personal data must be legal in three places at once — an EU customer, a California consumer, and an Indian data principal all land in the same warehouse, and each is governed by a different statute with a different vocabulary, a different legal trigger for processing, and a different deadline for honouring a deletion request. The naive answer is to fork the platform per region. The engineering answer is to parameterise one platform by jurisdiction, so that consent, policy, subject rights, and de-identification are data-driven rather than region-specific code.

This guide is a contrast, not a single-law walkthrough. The EU's GDPR, California's CCPA (as amended by the CPRA), and India's Digital Personal Data Protection Act of 2023 agree on the goal — give individuals control over their data — and disagree on almost every mechanism: opt-in consent versus opt-out of sale, a "data subject" versus a "consumer" versus a "data principal", a one-month response window versus forty-five days. A data engineer who hard-codes GDPR everywhere ships a system that is simultaneously over-compliant in California and non-compliant in India. The four ideas an interviewer will actually probe — a unified consent and subject registry, per-jurisdiction policy tags with purpose limitation, the access and erasure pipelines, and tokenization versus pseudonymization — are the load-bearing walls of a platform that satisfies all three at once.

When you want hands-on reps immediately after reading, drill the access-control practice library →, rehearse de-identification on the tokenization practice set →, and harden your validation logic on the data-quality practice set →.

---

On this page

• Privacy as a data-engineering problem, not a legal memo
• The unified consent & subject registry
• Policy tags, data mapping & purpose limitation
• Subject-rights pipelines — access & erasure tombstones
• Tokenization & pseudonymization
• Cheat sheet — multi-jurisdiction privacy recipes
• Frequently asked questions
• Practice on PipeCode

---

1. Privacy as a data-engineering problem, not a legal memo

Three statutes, one physical platform — the differences are the specification, not trivia

The one-sentence invariant: you cannot fork the warehouse per country, so every difference between GDPR, CCPA, and DPDP must become a column, a tag, or a policy the pipeline reads at runtime. A privacy pipeline is not a lawyer's summary rendered in a wiki; it is the set of data structures and jobs that make "is this processing allowed?" and "erase this person" answerable by code. To build it you have to know precisely where the three regimes diverge.

The vocabulary maps, but does not match.

• The individual. GDPR calls them a data subject; CCPA calls them a consumer (a California resident); DPDP calls them a data principal. Same human, three keys — which is exactly why you need one canonical subject_id.
• The accountable party. GDPR has a controller (decides purpose and means) and a processor; CCPA has a business and a service provider; DPDP has a data fiduciary and a data processor, plus the higher-duty Significant Data Fiduciary (SDF).
• The oversight role. GDPR can require a Data Protection Officer (DPO); DPDP requires an SDF to appoint a DPO based in India; CCPA has no DPO mandate but the CPRA created the California Privacy Protection Agency (CPPA) as a regulator.

The legal trigger for processing is where they diverge most.

• GDPR — a lawful basis, opt-in by default. You may only process personal data if you can name one of six lawful bases (Article 6): consent, contract, legal obligation, vital interests, public task, or legitimate interests. Consent, when used, must be freely given, specific, informed, and unambiguous.
• CCPA/CPRA — notice and opt-out, not consent. California does not require a lawful basis to collect. It requires notice at collection and gives the consumer a right to opt out of the sale or sharing of personal information (the "Do Not Sell or Share" link and the Global Privacy Control signal). Opt-in consent is only mandated for minors and for "sensitive personal information" limits.
• DPDP — consent or a "legitimate use", plus itemised notice. India's default is consent (free, specific, informed, unconditional, unambiguous, with a clear affirmative action), backed by a plain-language notice that must be available in English and the scheduled Indian languages. A short list of "legitimate uses" (e.g. the principal voluntarily provides data for a requested service) can substitute.

The subject rights rhyme, but the deadlines and shapes differ.

• Access. GDPR's Data Subject Access Request (DSAR, Article 15) → 1 month. CCPA's right to know → 45 days (extendable to 90). DPDP's right to access a summary of processing → within a period the rules set, via the fiduciary's grievance mechanism.
• Erasure. GDPR's right to erasure ("right to be forgotten", Article 17) is broad. CCPA's right to delete has enumerated exceptions. DPDP ties erasure to withdrawal of consent and to the purpose being fulfilled — there is no standalone "right to be forgotten" clause, but withdrawal must be as easy as giving consent.
• Correction and portability. GDPR grants rectification and portability; CPRA added a right to correct; DPDP grants correction and completion. Portability is a first-class GDPR right and absent from DPDP.

What interviewers listen for.

• Do you say "I parameterise the platform by jurisdiction" rather than "I build a GDPR system"? — senior signal.
• Do you distinguish opt-in consent (GDPR/DPDP) from opt-out of sale (CCPA) without prompting? — the single most common trap.
• Do you treat erasure as propagation across every downstream copy, not a DELETE on one table? — the whole engineering point.
• Do you know that DPDP has no portability right and CCPA needs no lawful basis, so a one-size GDPR build is wrong in two directions? — depth signal.

Worked example — the same person under three regimes

Detailed explanation. Before any code, internalise that one physical record carries three legal states simultaneously. A single user, Priya, who is an EU resident who later moved to California and holds Indian citizenship, could plausibly trigger all three regimes depending on where the data was collected and where she resides. The platform must answer "what may I do with row X?" differently per jurisdiction, from the same stored row.

Question. For one stored profile row, list what each regime requires before you may use it for marketing analytics, and what a deletion request means under each.

Input.

attribute	value
subject_id	s_88f1
email	[e-mail]
collected_in	EU (2023), then CA (2025)
purpose_requested	marketing analytics

Code.

REGIMES = {   # a jurisdiction is a policy object, not a country string in a WHERE clause
    "GDPR": {"needs": "lawful_basis", "marketing_ok_if": "consent",
             "erasure": "broad", "access_sla_days": 30},
    "CCPA": {"needs": "notice_at_collection", "marketing_ok_if": "not_opted_out",
             "erasure": "with_exceptions", "access_sla_days": 45},
    "DPDP": {"needs": "consent_or_legitimate_use", "marketing_ok_if": "consent",
             "erasure": "on_withdrawal", "access_sla_days": 30},
}

Step-by-step explanation. Under GDPR, marketing analytics on personal data needs a lawful basis; for marketing that is almost always consent, so you may proceed only if Priya opted in, and a deletion request is a broad erasure. Under CCPA, you needed no consent to collect — only notice — so marketing is allowed unless she opted out of sale/sharing, and deletion is honoured subject to statutory exceptions. Under DPDP, marketing requires consent (or a listed legitimate use) plus a valid notice id, and "deletion" is triggered when she withdraws that consent. Same row, three gates.

Output.

regime	may run marketing analytics if…	deletion means
GDPR	consent on record (opt-in)	broad erasure across copies
CCPA	consumer has not opted out	delete with enumerated exceptions
DPDP	consent + valid notice id	erase on consent withdrawal

Rule of thumb. If a single answer ("delete the user") is the same across all three regimes in your design, you have almost certainly hard-coded one law and mis-modelled the other two.

---

2. The unified consent & subject registry

One canonical subject_id and an append-only consent ledger are the foundation everything else stands on

The feature that makes a multi-jurisdiction platform tractable is that every regional identity resolves to one canonical subject_id, and every legality fact about that subject is an event in an append-only ledger. Without the single id you cannot honour an erasure request that spans an EU login and a California account; without the append-only ledger you cannot prove what consent existed at the moment a job ran, which is the question a regulator actually asks.

Identity resolution first.

• One subject, many keys. An EU customer id, a California consumer id, and an Indian principal id may all be the same human. A resolution step (deterministic on verified email/phone, probabilistic only as a fallback) maps them to one subject_id. This id, never raw PII, is the join key across the platform.
• Why not just email. Emails change, are shared, and are themselves PII. The subject_id is an opaque surrogate so that analytics can join on identity without touching a regulated attribute.

The consent ledger is event-sourced, not a mutable flag.

• Per-jurisdiction, per-purpose rows. A ledger row records (subject_id, jurisdiction, purpose, basis, state, notice_id, ts, source). GDPR rows carry a lawful_basis; CCPA rows carry an opt_out state for sale/sharing; DPDP rows carry consent plus the notice_id the principal saw.
• Withdrawal is a new event. You never UPDATE consent=false. You append a withdrawn event with a timestamp. The current state is a fold over history, so the ledger can answer "was consent valid at 14:03 last Tuesday?" — required for audits and for defending a past processing decision.
• Versioned notices. Because DPDP (and GDPR transparency) tie validity to the exact notice text shown, the notice_id references an immutable, versioned notice record. Re-consent is required when a material purpose changes.

The gate that reads the ledger.

• is_allowed(subject_id, purpose, jurisdiction) folds the ledger to a current state and returns a boolean plus a reason. Every job that touches personal data calls it — ingestion, transformation, activation.
• Deny-by-default. No matching allow event means not allowed. This is what makes DPDP's "consent required" and GDPR's "lawful basis required" the same code path, while CCPA's "allowed unless opted out" is expressed as a default-allow with an opt-out event.

Worked example — folding a consent ledger to a decision

Detailed explanation. The everyday operation is: given a subject, a purpose, and a jurisdiction, fold the event history into a yes/no. The fold encodes the regime difference — GDPR/DPDP require a positive consent event, CCPA requires the absence of an opt-out event — so the same function serves all three.

Question. Given the ledger below, may you run marketing analytics for s_88f1 under GDPR, and under CCPA?

Input.

subject_id	jurisdiction	purpose	event	ts
s_88f1	GDPR	marketing	consent_granted	2026-01-10
s_88f1	GDPR	marketing	consent_withdrawn	2026-03-01
s_88f1	CCPA	sale	opt_out	2026-02-15

Code.

from collections import defaultdict

def is_allowed(events, subject_id, purpose, jurisdiction):
    # fold history -> current state; deny-by-default for opt-in regimes
    rows = [e for e in events
            if e["subject_id"] == subject_id
            and e["jurisdiction"] == jurisdiction
            and e["purpose"] in (purpose, "sale")]
    rows.sort(key=lambda e: e["ts"])            # chronological fold
    state = None
    for e in rows:
        state = e["event"]
    if jurisdiction in ("GDPR", "DPDP"):
        return state == "consent_granted"       # positive consent required
    if jurisdiction == "CCPA":
        return state != "opt_out"               # allowed unless opted out
    return False

Step-by-step explanation. For GDPR the fold sees consent_granted then consent_withdrawn; the last state is consent_withdrawn, and since GDPR needs a positive consent_granted, the answer is no. For CCPA the relevant sale events fold to opt_out, and because CCPA is allowed-unless-opted-out, state == "opt_out" returns no as well. The identical function returns the right answer for opposite legal models because the fold rule branches on jurisdiction.

Output.

jurisdiction	folded state	marketing allowed
GDPR	consent_withdrawn	no
CCPA	opt_out	no

Rule of thumb. Store consent as an immutable event stream and compute the boolean on read — a mutable consent column throws away the history that every audit and every "was this legal at the time?" question depends on.

Privacy-engineering interview question on the consent registry

Question. An interviewer gives you three source systems — an EU billing DB keyed by eu_customer_id, a California app keyed by ca_user_id, and an Indian app keyed by in_principal_id — plus a verified-contact table. Design the resolution so a single erasure request deletes the person everywhere, and show the code that produces the canonical subject_id and rejects an unverified match.

Solution Using deterministic identity resolution on verified contacts

Code.

import hashlib

def canonical_subject_id(identity, verified_contacts):
    # Only *verified* email/phone may bridge identities across regimes.
    key = None
    for contact in ("email", "phone"):
        val = identity.get(contact)
        if val and verified_contacts.get((contact, val)):   # must be verified
            key = f"{contact}:{val.lower().strip()}"
            break
    if key is None:
        # No verified bridge -> do NOT merge; mint a region-local surrogate.
        return "local:" + identity["source"] + ":" + identity["source_id"]
    return "s_" + hashlib.sha256(key.encode()).hexdigest()[:12]

Step-by-step trace.

source	source_id	verified contact	resolves to
EU billing	eu_1001	email [e-mail] ✓	s_88f1c2a4e9b0
CA app	ca_7742	email [e-mail] ✓	s_88f1c2a4e9b0
IN app	in_5590	phone unverified ✗	local:IN:in_5590

1. Each incoming identity is looked up against the verified_contacts set — a contact only bridges regimes if it was actually confirmed (double opt-in, OTP), never on a raw string match.
2. The EU and CA rows both carry a verified [e-mail], so both hash to the same subject_id; a later erasure on that id reaches both systems.
3. The IN row's phone is unverified, so it does not merge — merging on an unverified identifier would risk deleting or exposing the wrong person, a worse failure than an unmerged duplicate.
4. The hash is deterministic and one-way, so the surrogate is stable across runs yet never leaks the underlying email.

Output:

canonical subject_id	linked source records	mergeable
s_88f1c2a4e9b0	eu_1001, ca_7742	yes (verified email)
local:IN:in_5590	in_5590	no (unverified)

Why this works — concept by concept:

• Deterministic resolution — hashing a verified, normalised contact yields the same subject_id on every run, so identity is reproducible and joinable without central coordination.
• Verify before you merge — bridging only on confirmed contacts prevents a mis-merge that would let one person's erasure or access request hit another's data — the highest-severity privacy bug.
• Surrogate over PII — the platform joins on subject_id, never on email, so analytics never touches a regulated attribute and the join key itself is not personal data.
• Fail-safe default — an unverified match stays a separate region-local id; a duplicate is recoverable, a wrong merge is a breach.
• Cost — resolution is O(1) per identity (a hash and a set lookup), so it scales linearly with ingest volume and adds no join-time overhead.

Access control
Topic — access-control
Consent-gate and access-control problems

Practice →

Data quality Topic — data-quality Identity-resolution and dedup problems

Practice →

---

3. Policy tags, data mapping & purpose limitation

Tags on the schema turn compliance into a query, not a code review

The move that scales privacy across hundreds of tables is policy-as-code tags attached to every column, so the pipeline enforces jurisdiction, purpose, and retention by reading the tag instead of by hand-written rules per table. All three regimes demand a form of this: GDPR's Article 30 requires a Record of Processing Activities (RoPA) — a living map of what data you hold, why, and for how long; DPDP requires an itemised notice inventory; CCPA requires you to catalogue categories of personal information and their purposes. The data map is that record made queryable.

Tag the columns, not the docs.

• pii_class. Marks a column as direct PII (email, phone, national id), indirect/quasi-identifier (ip, device, zip), sensitive (health, biometric, financial — GDPR Article 9 special category, CPRA "sensitive PI", DPDP has no separate sensitive tier but treats children's data specially), or non_pii.
• jurisdiction. Which regimes the column's rows fall under, derived from the subject registry, so a query can filter to EU-only rows when a rule is EU-specific.
• purpose_allowed. The closed set of purposes this column may be used for — billing, fraud, analytics, marketing. This is the machine-readable form of purpose limitation (GDPR Article 5(1)(b)): data collected for one purpose may not be silently repurposed.
• retention. A duration after which the value must be tombstoned — the operational form of GDPR's storage limitation and DPDP's "erase when the purpose is served".

The data map is a catalog you can query.

• Who / what / why / where / how long. Each dataset entry answers the RoPA questions in structured fields, so "which tables hold EU sensitive data used for marketing?" is a SELECT, not a week of interviews.
• Drift detection. New columns without tags fail CI. An untagged PII column is a compliance hole, so the catalog is enforced at schema-change time, the same way a data contract is.

Purpose limitation is enforced at read time.

• A purpose token on every job. A job declares the purpose it runs for (purpose="fraud"); the access layer intersects that with each column's purpose_allowed and strips or blocks columns that do not match.
• Repurposing requires a new basis. Using marketing-tagged data for a fraud model is not a bug you catch in review — the token mismatch blocks it, and lifting the block requires a new consent/basis record, which is exactly the legal reality.

Worked example — a purpose-limited column projection

Detailed explanation. The clearest way to see purpose limitation is a function that, given a job's purpose, returns only the columns that purpose is allowed to read. The tags live with the schema; the gate is a set intersection. This is how you stop a marketing job from quietly training on billing data.

Question. A fraud job asks to read the transactions table. Given the column tags below, which columns does it receive?

Input.

column	pii_class	purpose_allowed
amount	non_pii	billing, fraud, analytics
card_token	indirect	fraud
email	direct	billing, marketing
device_ip	indirect	fraud, analytics

Code.

def project_for_purpose(schema, purpose):
    # Return only columns whose purpose_allowed set includes this job's purpose.
    return [
        col["column"]
        for col in schema
        if purpose in col["purpose_allowed"]
    ]

visible = project_for_purpose(transactions_schema, purpose="fraud")

Step-by-step explanation. The gate walks the column tags and keeps a column only if "fraud" is in its purpose_allowed set. amount, card_token, and device_ip all list fraud, so they pass. email lists only billing and marketing, so the fraud job never sees it — even though the column physically sits in the same table. Purpose limitation becomes a projection, applied uniformly to every job.

Output.

column	visible to fraud job
amount	yes
card_token	yes
email	no
device_ip	yes

Rule of thumb. If the only thing standing between a marketing job and PII is a reviewer noticing the wrong SELECT, purpose limitation is not enforced — put the rule on the column and make the platform apply it.

Privacy-engineering interview question on retention and the data map

Question. GDPR storage limitation and DPDP's "erase when the purpose is served" both demand that data not outlive its purpose. You have thousands of tables. How do you drive retention off the data map so that expired rows are tombstoned automatically, and prove it happened for an audit?

Solution Using retention tags to schedule tombstones

Code.

from datetime import datetime, timedelta

def rows_to_tombstone(catalog, today):
    # Emit (table, column, cutoff) work items from the data map's retention tags.
    work = []
    for entry in catalog:                       # catalog == queryable RoPA
        for col in entry["columns"]:
            days = col.get("retention_days")
            if days is None:
                continue
            cutoff = today - timedelta(days=days)
            work.append({
                "table": entry["table"],
                "column": col["column"],
                "cutoff": cutoff,               # tombstone rows older than this
                "basis": "storage_limitation",
            })
    return work

Step-by-step trace.

table	column	retention_days	cutoff (today=2026-09-15)
events	device_ip	90	2026-06-17
profiles	email	730	2024-09-16
ledger	amount	(none)	skipped

1. The scheduler reads the data map, not the tables — retention lives with the catalog entry, so one job covers every dataset.
2. For each column carrying a retention_days tag it computes a cutoff date; rows whose event time predates the cutoff are due for a tombstone.
3. amount has no retention tag, so it is skipped — financial records often have a legal-obligation basis to be retained, which the absence of a tag encodes.
4. Each emitted work item names a basis (storage_limitation), so the resulting tombstone log is a self-describing audit record: what was erased, from where, and under which legal rationale.

Output:

work items emitted	audit fields per item
2 (device_ip, email)	table, column, cutoff, basis, run_ts

Why this works — concept by concept:

• Data map as driver — putting retention_days on the catalog means one scheduler enforces storage limitation across thousands of tables, instead of a cron per table that drifts.
• Tag absence is a decision — an untagged column is deliberately retained (legal obligation), so the model distinguishes "keep forever on purpose" from "nobody set a rule", which CI catches.
• Basis on every action — stamping each tombstone with its legal basis turns the erasure log into the evidence an auditor and a DPO ask for.
• Uniform across regimes — the same cutoff mechanism serves GDPR storage limitation and DPDP purpose-fulfilment because both reduce to "erase rows older than the tagged horizon".
• Cost — building the work list is O(columns) over the catalog, and the actual erase is O(expired rows) per table, decoupled from total table size via a time index.

Data quality
Topic — data-quality
Data-contract and retention-validation problems

Practice →

Access control Topic — access-control Purpose-limited access and column-projection problems

Practice →

---

4. Subject-rights pipelines — access & erasure tombstones

One router, three SLAs — access fans out and assembles, erasure writes a tombstone that propagates

Subject rights are where a privacy platform earns its keep, and the design that satisfies all three regimes is a single request router that branches into an access pipeline and an erasure pipeline, both driven by the data map and both leaving an audit trail. The regimes differ in deadline and scope — GDPR's DSAR is one month, CCPA's requests are forty-five days, DPDP routes through the fiduciary's grievance mechanism — but the engineering is identical: find every copy of the subject, then either export it or erase it.

The router normalises the request.

• One request type, per-regime SLA. A request carries (subject_id, kind, jurisdiction, received_ts). The router stamps the deadline from the jurisdiction (30 / 45 / grievance-window days) and enqueues the work. Verification of the requester happens here — a "verifiable consumer request" under CCPA, identity confirmation under GDPR/DPDP.
• Deny-by-default on verification. An unverified request is never executed; responding to an impersonator is itself a breach.

Access: fan out over the data map, assemble a package.

• The data map is the index of where to look. Because every dataset holding PII is catalogued and keyed by subject_id, "find everything about this person" is a fan-out over the map, not tribal knowledge of which tables matter.
• Assemble, redact, deliver. Results are collected into an export bundle; other subjects' data that co-occurs (a shared thread, a joint transaction) is redacted. GDPR portability additionally requires a structured, machine-readable format — a shape DPDP does not mandate and CCPA satisfies with the "right to know".

Erasure: tombstone, then propagate.

• A tombstone is a durable erasure marker, not a DELETE. Writing {subject_id, erased_at, scope} to a tombstone table means the fact of erasure survives even as it propagates asynchronously to warehouses, lakes, backups, and search indices. A raw DELETE on the primary leaves stale copies downstream that no one tracks.
• Propagation to every copy. The tombstone is a change-data event every downstream consumer subscribes to; each applies the erasure to its own store and acknowledges. Erasure is complete only when every registered sink has acked — which is why the data map must also list sinks.
• Idempotent and re-runnable. Applying the same tombstone twice is a no-op, so a retried or replayed erasure is safe; this is the property that lets you re-drive a failed propagation without fear.

Worked example — an erasure tombstone with downstream propagation

Detailed explanation. The canonical erasure is not a delete statement; it is an event that fans out. You write one tombstone, and every downstream store consumes it and confirms. The pipeline is done when all sinks ack, and the tombstone table is the proof.

Question. A verified GDPR erasure arrives for s_88f1. Show how the tombstone drives deletion across the primary DB, the warehouse, and a backup, and how you know it is complete.

Input.

sink	keyed by subject_id	ack status before
primary_db	yes	—
warehouse	yes	—
backup	yes (restore-time apply)	—

Code.

def erase(subject_id, sinks, tombstones):
    # 1) Durable marker first, 2) fan out, 3) require all acks (idempotent).
    tombstones.append({"subject_id": subject_id,
                       "erased_at": now(), "scope": "all", "acks": set()})
    for sink in sinks:
        sink.apply_tombstone(subject_id)        # each is a no-op if already applied
        tombstones[-1]["acks"].add(sink.name)
    required = {s.name for s in sinks}
    complete = required.issubset(tombstones[-1]["acks"])
    return complete

Step-by-step trace.

step	action	sink	acks so far
1	write tombstone	—	{}
2	apply + ack	primary_db	{primary_db}
3	apply + ack	warehouse	{primary_db, warehouse}
4	apply + ack	backup	{primary_db, warehouse, backup}

1. The tombstone is written before any deletion, so if the process crashes mid-propagation the erasure is not lost — a restart re-drives it from the durable marker.
2. Each sink applies the erasure and acks; because apply_tombstone is idempotent, a re-driven step that already ran simply re-acks without error.
3. Completion is required.issubset(acks) — every registered sink must confirm, including the backup, which applies the tombstone at restore time so a recovered snapshot never resurrects the person.
4. The tombstone row, with its ack set and timestamp, is the audit artifact proving the erasure met the regime's deadline.

Output:

subject_id	sinks acked	complete
s_88f1	primary_db, warehouse, backup	yes

Rule of thumb. Erasure is finished when the last downstream copy acks the tombstone, not when the primary DELETE returns — model the sinks explicitly or you will leave the person in a lake or a backup.

Privacy-engineering interview question on cross-regime request routing

Question. Your inbox receives access and deletion requests for the same subject_id across GDPR, CCPA, and DPDP, each with its own deadline. Design a router that assigns the correct SLA, refuses unverified requests, and lets an erasure be retried safely if a downstream sink was offline.

Solution Using an idempotent router keyed on request id

Code.

SLA_DAYS = {"GDPR": 30, "CCPA": 45, "DPDP": 30}

def route(request, verified, done_ids):
    # Idempotent: a replayed request_id is skipped; unverified is refused.
    if request["request_id"] in done_ids:
        return {"status": "already_processed"}
    if not verified.get(request["request_id"]):
        return {"status": "refused", "reason": "unverified requester"}
    deadline = add_days(request["received_ts"], SLA_DAYS[request["jurisdiction"]])
    handler = access_pipeline if request["kind"] == "access" else erasure_pipeline
    result = handler(request["subject_id"], jurisdiction=request["jurisdiction"])
    done_ids.add(request["request_id"])         # mark handled -> safe to retry
    return {"status": "completed", "deadline": deadline, "detail": result}

Step-by-step trace.

request_id	jurisdiction	kind	verified	outcome
r1	GDPR	erasure	yes	completed, deadline +30d
r1	GDPR	erasure	yes	already_processed (replay)
r2	CCPA	access	no	refused (unverified)
r3	DPDP	erasure	yes	completed, deadline +30d

1. r1 is verified and new, so the router stamps a 30-day GDPR deadline and calls the erasure pipeline, then records r1 as done.
2. A replay of r1 (a retried queue message) short-circuits on the done_ids check, so the erasure is never applied twice — the router itself is idempotent even before the sinks are.
3. r2 is a CCPA access request but the requester failed verification, so it is refused; responding would have leaked data to an impersonator.
4. r3 is a DPDP erasure, verified, and gets the DPDP window — the same router serves three regimes by looking the SLA up from a table.

Output:

request_id	status	deadline
r1	completed / then already_processed	+30d
r2	refused	—
r3	completed	+30d

Why this works — concept by concept:

• Single router, table-driven SLA — the deadline is a lookup, so adding a fourth regime is a config row, not a new pipeline; the branch on kind chooses access vs erasure.
• Idempotent on request_id — a done_ids guard makes replays and retries no-ops, so an at-least-once queue never double-erases or double-exports.
• Verify-then-act — refusing unverified requests is the router's most important job; a fast, wrong response to an impersonator is worse than a slow, correct one.
• Deadline as a first-class field — stamping the SLA on completion produces the timeliness evidence each regulator expects.
• Cost — routing is O(1) per request; the real work is the O(copies) fan-out inside the handler, bounded by the data map's sink list.

Access control
Topic — access-control
Subject-request routing and verification problems

Practice →

Tokenization Topic — tokenization Tombstone-propagation and erasure problems

Practice →

---

5. Tokenization & pseudonymization

Keep raw PII in a vault, land tokens in the warehouse, and make erasure a single key delete

The technique that makes every previous section cheaper is de-identification: replace raw PII with tokens so the warehouse holds no direct identifiers, and use per-subject keys so erasure becomes deleting one key rather than chasing rows across every store. GDPR explicitly rewards pseudonymization (Article 4(5) and Article 32) as a security measure; CPRA and DPDP both treat de-identified data as lower-risk. Getting the vocabulary exact is itself an interview filter.

Three terms interviewers make you separate.

• Tokenization — reversible, vault-backed. The real value (a card number, an email) is stored in a secured vault and replaced everywhere else by an opaque token. Given the vault and authorisation, you can re-identify. The warehouse never holds the raw value.
• Pseudonymization — reversible only with separate info (GDPR Art 4(5)). The data can be attributed to a person only by using additional information kept separately and under controls. Tokenization is one implementation; the legal point is that the mapping is held apart and protected. Pseudonymised data is still personal data.
• Anonymization — irreversible. No key, no vault, no path back to the person. Truly anonymised data falls outside GDPR/DPDP entirely — but true anonymisation is hard, because quasi-identifiers can re-identify, so most "anonymised" data is really pseudonymised.

Crypto-shredding makes erasure O(1).

• A key per subject. Each subject's PII is encrypted with a subject-specific key; the ciphertext can live anywhere, even in backups. To erase the person you delete their key, and every ciphertext copy — warehouse, lake, backup, index — becomes permanently unreadable at once.
• Why this beats chasing rows. The propagation problem from Section 4 shrinks: you still tombstone the identity, but the content is neutralised the instant the key dies, so a missed backup copy is inert rather than a breach.

Deterministic vs randomized tokens — the analytics trade-off.

• Deterministic tokens join. The same input always maps to the same token, so you can GROUP BY token and join across tables without re-identifying. The cost is that deterministic tokens are vulnerable to frequency analysis, so they suit low-cardinality-safe joins, not secrets.
• Randomized tokens do not join. A fresh token per occurrence is safest but breaks joins, so it fits values you never aggregate on (a free-text note, a one-off id). Choosing per column is a data-modelling decision the tag from Section 3 can carry.

Worked example — tokenizing on ingest and re-identifying by purpose

Detailed explanation. The everyday pattern is: on ingest, swap each PII field for a deterministic token and stash the real value in the vault; downstream jobs work on tokens; a re-identify call, gated by purpose, is the only path back. This keeps the warehouse free of raw PII while preserving joins.

Question. Tokenize an incoming record's email, load only the token to the warehouse, and show what a fraud-purpose re-identify returns versus a marketing one when email is not marketing-allowed.

Input.

{"subject_id": "s_88f1", "email": "[e-mail]", "amount": 42.5}

Code.

import hmac, hashlib

VAULT = {}  # token -> raw value, secured & access-controlled in reality

def tokenize(value, key=b"subject-key-s_88f1"):
    tok = "tok_" + hmac.new(key, value.encode(), hashlib.sha256).hexdigest()[:12]
    VAULT[tok] = value                          # raw stays in the vault only
    return tok

def reidentify(token, purpose, allowed_purposes):
    if purpose not in allowed_purposes.get(token, set()):
        raise PermissionError("purpose not permitted for this field")
    return VAULT[token]

row = {"subject_id": "s_88f1", "email": tokenize("[e-mail]"), "amount": 42.5}

Step-by-step explanation. tokenize computes a deterministic HMAC token and files the raw email in the vault; the row written to the warehouse carries email = tok_..., never the address. A fraud job that is permitted to re-identify the email gets the raw value back; a marketing job, for which email is not an allowed purpose, hits the PermissionError and only ever sees the token. Erasure later deletes subject-key-s_88f1, after which the token can no longer be regenerated or matched.

Output.

caller purpose	re-identify email	result
fraud	allowed	[e-mail]
marketing	not allowed	PermissionError (token only)

Rule of thumb. If raw PII reaches the warehouse "just in case", you have inverted the model — tokenize at the edge and make re-identification the rare, purpose-gated exception.

Privacy-engineering interview question on erasure at scale

Question. You hold a subject's data across a warehouse, a data lake, and nightly backups. A GDPR erasure must guarantee the data is unrecoverable, but rewriting every backup is infeasible. How do you make erasure both complete and cheap?

Solution Using crypto-shredding with per-subject keys

Code.

KEYS = {}   # subject_id -> encryption key (the only thing that must truly die)

def store_pii(subject_id, value):
    key = KEYS.setdefault(subject_id, generate_key())
    return encrypt(value, key)                  # ciphertext may live anywhere

def crypto_shred(subject_id):
    # Erasure = destroy the key; all ciphertext copies become unreadable.
    KEYS.pop(subject_id, None)

def read_pii(subject_id, ciphertext):
    key = KEYS.get(subject_id)
    if key is None:
        return None                             # shredded -> permanently opaque
    return decrypt(ciphertext, key)

Step-by-step trace.

step	action	key present	read result
1	store_pii(s_88f1, email)	yes	ciphertext in warehouse, lake, backup
2	read_pii before shred	yes	[e-mail]
3	crypto_shred(s_88f1)	no	—
4	read_pii after shred (any copy)	no	None (unrecoverable)

1. Each subject's PII is encrypted with a per-subject key, so the ciphertext is safe to replicate to the lake and to backups.
2. Before erasure, an authorised read decrypts normally because the key exists.
3. crypto_shred deletes only the key — a single, cheap operation — instead of hunting and rewriting every ciphertext copy.
4. After shredding, every copy, including untouched backups, decrypts to None; the data is unrecoverable without ever rewriting a backup file.

Output:

after crypto_shred	warehouse	lake	backup
decryptable	no	no	no

Why this works — concept by concept:

• Key as the unit of erasure — destroying one per-subject key neutralises unbounded ciphertext copies at once, so erasure cost is O(1) rather than O(copies).
• Backups without a rewrite — because a stale backup holds only ciphertext, a shredded key makes that backup inert without touching it — the only practical way to honour erasure against immutable backups.
• Pseudonymization by construction — the warehouse holds ciphertext/tokens, so it is pseudonymised data under GDPR Art 4(5), reducing breach exposure even before any request.
• Tombstone still needed for identity — crypto-shred kills the content; you still write the Section 4 tombstone so the fact of erasure and the identity linkage are removed and audited.
• Cost — shredding is O(1) (one key delete); the only ongoing cost is per-read decryption, an acceptable trade for making erasure and replication cheap.

Tokenization
Topic — tokenization
Tokenization, vaulting and crypto-shred problems

Practice →

Data quality Topic — data-quality De-identification and re-identification-risk problems

Practice →

---

Cheat sheet — multi-jurisdiction privacy recipes

Regime field mapping.

REGIME = {
    "GDPR": {"subject": "data subject", "party": "controller",
             "trigger": "lawful basis (Art 6)", "sale_optout": False,
             "erasure": "right to be forgotten (Art 17)", "access_sla": 30},
    "CCPA": {"subject": "consumer", "party": "business",
             "trigger": "notice at collection", "sale_optout": True,
             "erasure": "right to delete (exceptions)", "access_sla": 45},
    "DPDP": {"subject": "data principal", "party": "data fiduciary",
             "trigger": "consent / legitimate use", "sale_optout": False,
             "erasure": "on withdrawal", "access_sla": 30},
}

Consent-check gate (deny-by-default for opt-in regimes).

def allowed(state, jurisdiction):
    if jurisdiction in ("GDPR", "DPDP"):
        return state == "consent_granted"
    if jurisdiction == "CCPA":
        return state != "opt_out"
    return False

Policy-tag column contract.

column = {"name": "email", "pii_class": "direct",
          "jurisdiction": ["GDPR", "CCPA", "DPDP"],
          "purpose_allowed": ["billing"], "retention_days": 730}

Erasure tombstone.

tombstone = {"subject_id": "s_88f1", "erased_at": now(),
             "scope": "all", "acks": set()}   # complete when all sinks ack

Crypto-shred (O(1) erasure).

def crypto_shred(subject_id, keys):
    keys.pop(subject_id, None)     # every ciphertext copy goes dark at once

Purpose-limited read.

visible = [c["name"] for c in schema if purpose in c["purpose_allowed"]]

Regime comparison at a glance.

dimension	GDPR (EU)	CCPA/CPRA (California)	DPDP (India)
individual	data subject	consumer	data principal
processing trigger	lawful basis	notice + opt-out	consent / legitimate use
access deadline	1 month	45 days	rules-set window
erasure	broad (Art 17)	delete with exceptions	on withdrawal

---

Frequently asked questions

What is a multi-jurisdiction privacy pipeline?

It is a data platform designed so the same personal data can be processed legally under several privacy laws at once — typically EU GDPR, California CCPA/CPRA, and India's DPDP Act 2023 — without forking the infrastructure per region. The differences between the laws are pushed into data: a consent ledger, per-column policy tags, jurisdiction fields, and subject-rights pipelines that read those structures at runtime. The goal is one physical platform whose behaviour is parameterised by jurisdiction rather than hard-coded to one statute.

How do GDPR, CCPA, and DPDP differ for a data engineer?

The biggest engineering difference is the legal trigger for processing. GDPR requires a lawful basis (often opt-in consent) before you process; CCPA requires no consent but a notice at collection plus an opt-out of sale or sharing; DPDP requires consent (or a listed legitimate use) plus an itemised notice. They also differ on vocabulary (data subject vs consumer vs data principal), response deadlines (1 month vs 45 days vs a rules-set window), and rights (GDPR has portability, DPDP does not; CCPA has no lawful-basis concept). A design that assumes opt-in everywhere is wrong for California, and one that assumes opt-out is wrong for the EU and India.

What is the difference between tokenization and pseudonymization?

Tokenization is a concrete technique: replace a raw value with an opaque token and keep the real value in a secured vault, so the token is reversible only through that vault. Pseudonymization is the legal concept (GDPR Article 4(5)): data that can be attributed to a person only with additional information kept separately and protected — tokenization is one way to achieve it. Both are reversible and both leave the data as personal data; only true anonymization, which removes any path back to the individual, takes data outside GDPR and DPDP.

How do you engineer the right to erasure across downstream copies?

Do not model erasure as a DELETE on one table. Write a durable tombstone marker for the subject, then propagate it as an event to every downstream sink — warehouse, lake, search index, backups — and treat the erasure as complete only when every registered sink acknowledges. To handle backups you cannot rewrite, combine the tombstone with crypto-shredding: encrypt each subject's data with a per-subject key and delete the key, which renders every ciphertext copy, including old backups, permanently unreadable in one cheap operation.

What is purpose limitation and how do you enforce it in a pipeline?

Purpose limitation (GDPR Article 5(1)(b), echoed by DPDP) means data collected for one purpose may not be silently reused for another. You enforce it by tagging each column with the closed set of purposes it may serve, and by having every job declare the purpose it runs for; the access layer intersects the two and strips or blocks columns whose purpose does not match. Repurposing then requires a new consent or basis record rather than a code change a reviewer might miss, which mirrors the legal reality.

Do I need separate warehouses per jurisdiction?

Usually no, and forking per region is the anti-pattern this whole design avoids. A single platform with a canonical subject_id, a consent ledger, jurisdiction tags, and rights pipelines handles all three regimes and stays maintainable. You may still need data-residency controls — DPDP can restrict transfers to blacklisted countries and some GDPR flows require EU storage — but residency is a placement policy on tagged data, not a reason to duplicate the entire stack and its logic three times.

Practice on PipeCode

Pipecode.ai is Leetcode for Data Engineering — every idea above, from the consent-ledger fold and the purpose-limited projection to the erasure tombstone and crypto-shred, maps to a hands-on practice room where you build the pipeline against real graded inputs. PipeCode pairs each reading with 450+ DE-focused problems and a real-time scoring engine, so your answer to "how would you make this erasure complete across every copy?" holds up under a senior interviewer's depth probes.

Practice access-control problems now →
Tokenization drills →