Your prod data, faked: one-way replication with deterministic PII masking

Every dev environment tells one of two lies. Either it runs on hand-made toy data — three users named "test", screens that look nothing like production, bugs that only reproduce for real people — or it runs on a copy of production, which is a privacy incident with a cron schedule.

This post is the third option: dev gets production's shape without production's people. Every write to prod mirrors into dev within seconds — same tables, same volumes, same weird edge cases — with every piece of personal data replaced on the way through, deterministically, so the fake people stay the same fake people forever.

TL;DR — Every prod table's DynamoDB stream feeds one replicator Lambda that masks and writes into the matching -dev table. The trick is determinism: the faker is seeded from the entity's id, so the same user gets the same fake name and email in every table, on every write, forever — referential integrity survives masking. The policy is default-deny per model (an unknown model copies nothing until someone decides), with one deliberate deviation: product vocabulary — item names, stores, receipt lines — is kept, because it's the analytics backbone and isn't personal; free text is faked; and push tokens are nulled, never copied — a masked token is still a loaded gun. Stream wiring lives in each table's stack, because the dependency arrow already points that way.

(Part 25 of Building CannyCart, a voice-first shopping app I'm building in public. Self-contained — no earlier context needed.)

The architecture in one diagram

prod  {Model}-prod ──stream──► replicator Lambda ──mask──► {Model}-dev
                                                               │
                npm run clone:dev-sandbox ─────────────────────┴──► {Model}-{sandbox}

One Lambda, subscribed to every production table's stream (the tables already carry env-scoped names — cannycart-{Model}-{env} — so source and destination derive mechanically from the stream ARN, and the handler refuses any source not ending in -prod). Inserts and updates become masked puts; deletes mirror as deletes. Strictly one-way: nothing in dev can touch prod, and the wiring only exists in the production deployment at all — every other environment carries an idle Lambda and no event source mappings.

Below dev sits a second, simpler hop: an on-demand script that copies dev tables into a local sandbox. It does no masking — dev is already fake, which is the quiet payoff of doing the masking at the top: every downstream environment inherits safety for free.

Determinism is the whole trick

Naive masking replaces "Jane" with a random name — a different random name on every write. Update your profile twice and you're two people; your shopping list belongs to a third. Cross-table joins dissolve, and dev data stops resembling reality in exactly the way that matters.

The replicator instead seeds its faker from the entity's id — a small hash (FNV-1a) of the user's stable id picks the persona. Same user, same fake identity, in every table, on every write, forever:

• A support question like "why did this user's receipt not match?" is followable in dev — one consistent fake person owns the profile, the lists, the receipts — without anyone knowing who they really are.
• An update to a real profile updates the same fake profile, so dev history stays coherent over months.
• Fields with no entity id nearby are seeded from the value itself — a list's name fakes identically wherever it appears, which matters because receipts denormalise the list name into their frozen snapshot. The live row and the snapshot agree in dev, just as they do in prod.

Masking that destroys identity relationships isn't anonymisation, it's corruption with extra steps. Determinism keeps the graph and drops only the people.

The policy: default-deny, one argued exception

The masking map is per-model, and the fallback for an unknown model is deny — a new table replicates nothing until someone consciously decides what's safe. New models appear in the mirror automatically; their fields don't. That default turned the scariest failure mode ("we added a model and silently mirrored its PII") into a visible chore.

One deviation from strictness, argued in writing: product vocabulary is kept. Item names, units, categories, barcodes, store names, receipt line items — none of it is faked. Two reasons: it isn't personal data (a thousand users buy "Milk" at "Tesco"), and it's the analytics backbone — the price-history and savings features key on product names, so faking them would leave dev with features that can never be exercised. Meanwhile the free-text fields where personal context actually lands — item notes, the raw voice-capture phrases, deletion-request messages — are all faked. The line isn't "strings vs numbers"; it's vocabulary vs voice.

And one field gets the harshest treatment: push tokens are nulled, never copied. A push token is a credential pointing at a physical phone; any test send in dev would light up a real customer's lock screen. Masking isn't enough — only absence is safe. (Part 23 made the same point from the client side; this is the server holding the same line.)

The ops details that make it boring (complimentary)

• Soft deletes need no special handling. This app's business deletes are flag updates (the convention pays again) — they arrive as ordinary MODIFY events and mirror like any edit.
• Hard deletes mirror as deletes — including TTL expiry: when prod's time-boxed rows age out, the stream's REMOVE events prune the dev copies too. Retention policy replicates itself.
• A torn-down dev doesn't wedge prod. If the dev tables are gone, the handler skips rather than retry-looping — the mirror is a convenience, and a convenience must never become a production liability.
• Partial batch failures report precisely, so one bad record doesn't poison a stream batch.
• The masking module was verified by a small test harness before any deploy: determinism, cross-table identity agreement, keep-list passthrough, token nulling. A masker is exactly the kind of code you want proven before it ever sees a real record.

Where the wiring lives (the recurring lesson)

The event source mappings sit in each table's stack, not the Lambda's. Not aesthetics: the data stack already depends on the function stack (other Lambdas serve the API), so a function-stack reference to a table's stream ARN would point the dependency arrow both ways — a circular dependency and a failed deploy. Same shape as Part 14's IAM policy placement: in a multi-stack backend, where a resource is declared is a dependency-direction decision, and the winning move is to declare it on the side the arrow already points to.

What's deliberately not built

No backfill. The mirror starts empty and fills as prod is used. If prod ever accumulates significant history before dev needs it, the documented variant is a one-off masked scan-copy — designed, written down, unbuilt until a trigger fires. (Parts 10, 11 and 18 readers know this move by now: ship the mechanism, file the escalation.)

What I took away

• Seed fakes from stable ids. Deterministic personas preserve the relationship graph — the difference between anonymised data and shredded data.
• Default-deny per model. New tables must earn replication field by field; silence should copy nothing.
• Distinguish vocabulary from voice. Shared product words power analytics and aren't PII; free text is where people live — fake it.
• Credentials don't get masked, they get nulled. Push tokens, and anything else that can reach a real person.
• Mirrors must fail politely. A dev convenience that can wedge production isn't a convenience.
• Declare stream wiring where the dependency arrow already points — the table's stack, not the function's.

Next up

Part 26 heads back to the app's front door: the Home dashboard — the screen that answers "what now?", built entirely from hooks the other tabs already own, and the two rules that stop a dashboard from becoming a dumping ground.

How does your team's dev environment get its data — toy fixtures, a prod copy you don't talk about, or something masked? And would your masking survive the "same user twice" test?