DEV Community: Sina Mohammadi

The Customer Who Said "Delete Me" (and the 420 Million Events That Said "No")

Sina Mohammadi — Mon, 19 Oct 2026 07:47:04 +0000

A system design story about GDPR, event sourcing, and a very long Tuesday.

It was a normal Tuesday. The coffee was fine. The build was green. Nothing was on fire.

Then Legal sent me a Slack message:

"Hey, quick one 🙂 A user sent a GDPR erasure request. We have 30 days to answer. It's been 20. Can you delete her data? Should be easy, right?"

I read it three times.

Twenty days. The request had been sitting in a shared inbox for twenty days. Nobody's fault. Everybody's fault. That's how shared inboxes work.

And then there was that last sentence: "Should be easy, right?"

In my experience, "should be easy" is the most dangerous phrase in software. It is right up there with "it works on my machine" and "I'll just push a tiny fix on Friday."

Quick background: what we built

Our product is a customer support platform. Agents answer tickets. Customers send messages. Everyone is slightly angry about something.

We built it with event sourcing. If you don't know the term, here is the simple version.

Think about your bank statement. The bank never goes back and erases a line. If you spend 10 euros by mistake, the bank adds a new line that says "refund +10". The history stays the same. The current balance is just the sum of all lines.

We did the same thing with tickets. Every change is an event:

TicketOpened
MessageSent
AgentReplied
TicketReassigned
TicketClosed

We write events. We never update them. We never delete them. The state of any ticket is what you get when you replay its events from the start.

This gave us a perfect audit trail, easy debugging, and the ability to rebuild any view of the data. We loved it. We gave talks about it. We had stickers.

We now had 420 million events.

And one person who wanted all of hers gone.

The first instinct (a.k.a. the part where I almost made it worse)

My brain said:

DELETE FROM events WHERE user_id = 'her-id';

One line. Done by lunch. Hero of the company.

Then my brain, which is smarter than my first brain, asked a few annoying questions.

1. Is her data only in the user_id column? No. People write things like "Hi, my name is Anna, my phone is 555-0123, and my router is on fire." That is personal data inside free text. No column name will save you.

2. Are her events alone? No. They live in the same streams as agent replies, other customers who were added to the thread, and internal notes. Delete a whole stream and you delete other people's data too. That is a different problem, and a worse one.

3. What happens to the order of events? They depend on each other. If you remove a MessageSent event, then a later event that says "agent replied to message #7" now points to nothing. Replays can crash. Totals stop adding up.

4. Where else did her data go? This was the scary one. Event sourcing means data is copied by design:

Read models (the Postgres tables that power the UI)
The search index
The analytics warehouse
The email provider that sent the notifications
Application logs
Backups

So even a perfect delete in the event store would leave her data in five or six other places. Like cleaning one room of a house that has six rooms, and all of them have glitter.

Pause. Your turn.

Before you read the rest, try this. Take one minute and write down the questions you would ask before touching any code.

I'll wait. I have nothing else to do. I have 10 days.

...

Done? Good. Here's what I wrote down.

The questions that mattered (none of them were about code)

Is the request real? Did we verify that the person asking is the person in the data? A fake erasure request is a nice way to destroy someone else's account.
What does "everything" mean? All tickets? Billing records? Just personal details?
Do we have a legal reason to keep some of it? GDPR Article 17 has exceptions. If we must keep invoices for tax law, or the data is needed for a legal claim, we may keep that part. Not everything, but not nothing either. This is a question for Legal, not for me.
Whose data is mixed in? Agents' names, other customers in the same thread.
Can we ask for more time? GDPR lets you extend the deadline for complex requests, but you must tell the person within the first month and explain why. We had 10 days left to send that message at the latest. (Note to self: never look at this shared inbox on a Tuesday again.)

Notice something? Four of five questions are not technical. This is what the exercise really tests. The code is the easy half.

The options on the table

We had four ways to handle the event store itself.

Option	What it means	The catch
Hard delete	Remove her events from the store	Breaks the history. Breaks replays. Breaks trust.
Redact in place	Replace the personal parts with `[erased]`, keep the event shape	You are editing an append-only log. Needs a careful, audited migration.
Crypto-shredding	Encrypt each person's data with their own key. To erase, destroy the key. The data is still there, but it is unreadable noise	Beautiful idea. But you must do it before writing the events.
Wait it out	Let backups expire by their normal schedule	Fine for backups. Not fine as the only plan.

Crypto-shredding is the answer everyone mentions first. And honestly, it's the right design.

But we didn't do it.

Crypto-shredding is like a seatbelt. A great idea. It just needs to be fitted before the crash. Our 420 million events were already written in plain text. There was no key to destroy.

So for this request, we had to take the less glamorous path.

What we actually did

Step 1: Find her data (the hard part). We searched the event store, not just by user ID, but by her email, her phone number, and her name inside message text. We ran it as a dry run first: count only, change nothing. The number of hits was higher than we expected. It always is.

Step 2: Redact, don't delete. We wrote a controlled migration. It replaced the personal fields with a placeholder and kept the event types, order, and IDs. Every step was logged: who ran it, when, and on which events. The log itself did not contain her data. (Yes, we checked. Twice. That would have been a very sad joke.)

Step 3: Fix the copies. We rebuilt the read models from the redacted events. We removed her from the search index and the analytics warehouse. We sent a deletion request to the email provider. We checked the logs.

Step 4: Handle the backups honestly. We could not reach into old backups and edit them. So we did two things. We wrote down her erasure in a deletion ledger. And we made a rule: if anyone ever restores from a backup, the ledger is replayed before the system goes live again. The old backups age out on their normal schedule, and we told her that clearly.

Step 5: Answer the human. We wrote back. We said what we erased, what we kept (and the legal reason), and when the last backup copy will disappear. Plain words. No hiding behind legal language.

And for the future: new events now get per-person encryption keys. The seatbelt is installed. We checked that it clicks.

The real lesson

The word "deleted" means three different things:

To an engineer: the row is gone.
To a lawyer: not reasonably recoverable, and we can prove what we did.
To the customer: nobody has my data anymore.

The job is to make those three meanings meet. And you cannot do that with one SQL line. You do it by asking boring, annoying, slightly scary questions first.

This is why "should be easy, right?" is dangerous. It treats a design decision like a task.

And one more thing. If you build any append-only system, ask this on day one, not on day 20:

"What will we do when someone asks us to forget?"

Because one day, someone will. And it will probably be a Tuesday.

I'm an engineer, not a lawyer, and this is not legal advice. GDPR details depend on your situation, so talk to someone qualified.

Try it yourself

This story is based on an exercise from The Senior Leap, an open-source repo of hands-on system design scenarios. Each one asks you to think like a senior engineer before you see the answer. It's called erasure-request in the system-design folder. Try it first, write your own analysis, and then compare.

How would you have handled it? Tell me in the comments. Bonus points if you found a better trick for the backups.

Check this article. I found it essential.

Sina Mohammadi — Wed, 30 Sep 2026 16:57:57 +0000

Includes strict Zod schema validation tips

Kamil Mysliwiec for NestJS

Sep 30

NestJS Production Checklist: 17 Checks Before You Deploy

#nestjs #node #security

17 min read

The Ladder to Senior Engineer Is Broken, But You Can Climb It Anyway.

Sina Mohammadi — Sat, 13 Jun 2026 10:36:44 +0000

We are deep in the AI era, and AI is stunning at writing code. It refactors functions in seconds, generates entire boilerplate scaffolds, and sails through technical interviews. But AI doesn’t make the hard calls. It won’t tell you whether to add a message queue or just bump the thread pool when traffic spikes. It can’t feel the weight of a shortcut that will hold for three months and then trigger a cascading outage at 3 AM.

Those decisions demand experience — the kind of instinct that only comes from watching real systems break in production, feeling the blast radius, and absorbing lessons that no textbook documents.

This is where the industry has painted itself into a corner. Companies now need senior engineers more desperately than ever to steer AI tools and make architectural calls. Yet those same companies have largely stopped hiring juniors and mid-levels — the very people you mold into tomorrow’s seniors. The ladder that used to carry you from writing your first feature to designing resilient systems has been kicked away. Mid-level developers are stuck prompting AI and reviewing generated code, starved of the deep, messy, high-stakes problem-solving that builds true senior-level judgment.

This gap between mid-level and senior is exactly why I built The Senior Leap.

Most learning resources hand you sanitized, textbook exercises: “Design Twitter.” “Implement a rate limiter.” They have clean boundaries and perfect answers. Real systems are nothing like that. The exercises in The Senior Leap are built entirely on real-world war stories — actual production incidents, ugly architectural trade-offs, and the weird gotchas that only reveal themselves when you’re paged after midnight.

Instead of asking you to write more code, the project drops you into a broken or incomplete system and demands an analysis. You lay out your reasoning, flag what you’re uncertain about, and commit to your read of the situation. Then you open the rubric. This isn’t a checklist of correct answers; it’s written from the perspective of a senior engineer who points out the exact failure mode you missed and the reframing question that would have changed your whole approach. It teaches you how to think, not just what to conclude.

Finally, an AI evaluator (running locally or via providers like Anthropic or OpenRouter) reads your analysis against that rubric and gives you direct, pointed feedback: what you caught, what you missed, and why it would have mattered in a real incident. It simulates the very feedback loop that usually takes years of on-the-job scars to accumulate.

If you’re a mid-level developer feeling stuck or worrying that AI is making you obsolete, hear this clearly: you are not obsolete. The industry still desperately needs people who can think critically, anticipate failures, and make tough calls when the playbook is empty. AI can write the code, but it needs a pilot. You can be that pilot — and this project is built to get you there.

The jump from mid-level to senior has never been about memorizing more syntax or chasing the newest framework. It’s about a shift in how you see systems. That shift is a skill, not a secret handshake, and you can build it deliberately. You just need the right scenarios to train on, the ones that feel uncomfortably close to reality.

The project is growing, and the scenarios are being built from real production stories. If you want to bridge the gap and start thinking like the senior engineer you’re meant to be, the door is open.

And if you’re a senior engineer carrying hard-learned lessons from incidents and architectural crossroads, I invite you to turn those experiences into exercises. Help mid-level devs learn how you think — and in doing so, help rebuild the ladder we all still need.