A system design story about GDPR, event sourcing, and a very long Tuesday.
It was a normal Tuesday. The coffee was fine. The build was green. Nothing was on fire.
Then Legal sent me a Slack message:
"Hey, quick one 🙂 A user sent a GDPR erasure request. We have 30 days to answer. It's been 20. Can you delete her data? Should be easy, right?"
I read it three times.
Twenty days. The request had been sitting in a shared inbox for twenty days. Nobody's fault. Everybody's fault. That's how shared inboxes work.
And then there was that last sentence: "Should be easy, right?"
In my experience, "should be easy" is the most dangerous phrase in software. It is right up there with "it works on my machine" and "I'll just push a tiny fix on Friday."
Quick background: what we built
Our product is a customer support platform. Agents answer tickets. Customers send messages. Everyone is slightly angry about something.
We built it with event sourcing. If you don't know the term, here is the simple version.
Think about your bank statement. The bank never goes back and erases a line. If you spend 10 euros by mistake, the bank adds a new line that says "refund +10". The history stays the same. The current balance is just the sum of all lines.
We did the same thing with tickets. Every change is an event:
TicketOpened
MessageSent
AgentReplied
TicketReassigned
TicketClosed
We write events. We never update them. We never delete them. The state of any ticket is what you get when you replay its events from the start.
This gave us a perfect audit trail, easy debugging, and the ability to rebuild any view of the data. We loved it. We gave talks about it. We had stickers.
We now had 420 million events.
And one person who wanted all of hers gone.
The first instinct (a.k.a. the part where I almost made it worse)
My brain said:
DELETE FROM events WHERE user_id = 'her-id';
One line. Done by lunch. Hero of the company.
Then my brain, which is smarter than my first brain, asked a few annoying questions.
1. Is her data only in the user_id column? No. People write things like "Hi, my name is Anna, my phone is 555-0123, and my router is on fire." That is personal data inside free text. No column name will save you.
2. Are her events alone? No. They live in the same streams as agent replies, other customers who were added to the thread, and internal notes. Delete a whole stream and you delete other people's data too. That is a different problem, and a worse one.
3. What happens to the order of events? They depend on each other. If you remove a MessageSent event, then a later event that says "agent replied to message #7" now points to nothing. Replays can crash. Totals stop adding up.
4. Where else did her data go? This was the scary one. Event sourcing means data is copied by design:
- Read models (the Postgres tables that power the UI)
- The search index
- The analytics warehouse
- The email provider that sent the notifications
- Application logs
- Backups
So even a perfect delete in the event store would leave her data in five or six other places. Like cleaning one room of a house that has six rooms, and all of them have glitter.
Pause. Your turn.
Before you read the rest, try this. Take one minute and write down the questions you would ask before touching any code.
I'll wait. I have nothing else to do. I have 10 days.
...
Done? Good. Here's what I wrote down.
The questions that mattered (none of them were about code)
- Is the request real? Did we verify that the person asking is the person in the data? A fake erasure request is a nice way to destroy someone else's account.
- What does "everything" mean? All tickets? Billing records? Just personal details?
- Do we have a legal reason to keep some of it? GDPR Article 17 has exceptions. If we must keep invoices for tax law, or the data is needed for a legal claim, we may keep that part. Not everything, but not nothing either. This is a question for Legal, not for me.
- Whose data is mixed in? Agents' names, other customers in the same thread.
- Can we ask for more time? GDPR lets you extend the deadline for complex requests, but you must tell the person within the first month and explain why. We had 10 days left to send that message at the latest. (Note to self: never look at this shared inbox on a Tuesday again.)
Notice something? Four of five questions are not technical. This is what the exercise really tests. The code is the easy half.
The options on the table
We had four ways to handle the event store itself.
| Option | What it means | The catch |
|---|---|---|
| Hard delete | Remove her events from the store | Breaks the history. Breaks replays. Breaks trust. |
| Redact in place | Replace the personal parts with [erased], keep the event shape |
You are editing an append-only log. Needs a careful, audited migration. |
| Crypto-shredding | Encrypt each person's data with their own key. To erase, destroy the key. The data is still there, but it is unreadable noise | Beautiful idea. But you must do it before writing the events. |
| Wait it out | Let backups expire by their normal schedule | Fine for backups. Not fine as the only plan. |
Crypto-shredding is the answer everyone mentions first. And honestly, it's the right design.
But we didn't do it.
Crypto-shredding is like a seatbelt. A great idea. It just needs to be fitted before the crash. Our 420 million events were already written in plain text. There was no key to destroy.
So for this request, we had to take the less glamorous path.
What we actually did
Step 1: Find her data (the hard part). We searched the event store, not just by user ID, but by her email, her phone number, and her name inside message text. We ran it as a dry run first: count only, change nothing. The number of hits was higher than we expected. It always is.
Step 2: Redact, don't delete. We wrote a controlled migration. It replaced the personal fields with a placeholder and kept the event types, order, and IDs. Every step was logged: who ran it, when, and on which events. The log itself did not contain her data. (Yes, we checked. Twice. That would have been a very sad joke.)
Step 3: Fix the copies. We rebuilt the read models from the redacted events. We removed her from the search index and the analytics warehouse. We sent a deletion request to the email provider. We checked the logs.
Step 4: Handle the backups honestly. We could not reach into old backups and edit them. So we did two things. We wrote down her erasure in a deletion ledger. And we made a rule: if anyone ever restores from a backup, the ledger is replayed before the system goes live again. The old backups age out on their normal schedule, and we told her that clearly.
Step 5: Answer the human. We wrote back. We said what we erased, what we kept (and the legal reason), and when the last backup copy will disappear. Plain words. No hiding behind legal language.
And for the future: new events now get per-person encryption keys. The seatbelt is installed. We checked that it clicks.
The real lesson
The word "deleted" means three different things:
- To an engineer: the row is gone.
- To a lawyer: not reasonably recoverable, and we can prove what we did.
- To the customer: nobody has my data anymore.
The job is to make those three meanings meet. And you cannot do that with one SQL line. You do it by asking boring, annoying, slightly scary questions first.
This is why "should be easy, right?" is dangerous. It treats a design decision like a task.
And one more thing. If you build any append-only system, ask this on day one, not on day 20:
"What will we do when someone asks us to forget?"
Because one day, someone will. And it will probably be a Tuesday.
I'm an engineer, not a lawyer, and this is not legal advice. GDPR details depend on your situation, so talk to someone qualified.
Try it yourself
This story is based on an exercise from The Senior Leap, an open-source repo of hands-on system design scenarios. Each one asks you to think like a senior engineer before you see the answer. It's called erasure-request in the system-design folder. Try it first, write your own analysis, and then compare.
How would you have handled it? Tell me in the comments. Bonus points if you found a better trick for the backups.
Top comments (0)