I Built a Notification System That Handles 10M Messages a Day, Here's What I Learned About System Design Interviews

Sophia Bennett — Fri, 19 Jun 2026 19:45:13 +0000

Last week I sat down to design a notification system from scratch. Push notifications, SMS, email, the whole stack, supporting 10 million daily alerts with sub 200ms latency and zero data loss tolerance. Sounds like a typical Tuesday at a big tech interview, right?

Except this time I wasn't whiteboarding alone in front of a panel hoping I remembered to mention idempotency. I was actually wiring up the architecture, watching costs tick up per hour, and getting real feedback on whether my design would survive contact with reality.

Here's the thing about system design interview prep that nobody tells you early enough: reading "Designing Data-Intensive Applications" and watching YouTube videos about CAP theorem will teach you concepts, but it won't teach you the actual skill being tested, which is decision making under constraints.

The Problem With Most System Design Practice

Most people prepare for system design interviews in one of two ways. They either memorize a handful of "famous" designs (design Twitter, design Uber, design a URL shortener) and hope the interviewer asks something close enough, or they pair up with a friend for a mock interview and spend half the session arguing about whether the friend's feedback is even correct.

Neither approach builds the muscle you actually need: thinking through tradeoffs in real time, justifying component choices, and reacting when requirements shift.

When I was prepping for interviews, the gap I kept hitting wasn't conceptual. I knew what a message queue was. I knew what a dead letter queue was for. What I didn't have was reps. Actual practice building these systems end to end, with constraints that forced real decisions instead of textbook answers.

What Actually Changed My Prep

I started treating system design practice the way I'd treat LeetCode grinding, except instead of solving the same two-pointer problem for the fortieth time, I was assembling architectures piece by piece and seeing immediate consequences.

For the notification system challenge, the requirements were specific and unforgiving:

Support push, SMS, and email delivery
Handle retries on failure
Process 10 million daily alerts
Keep latency under 200ms
Guarantee zero data loss
Everything has to be asynchronous
Stay under a fixed hourly budget

That budget constraint changes everything. Suddenly "just add more servers" isn't a free answer. You start asking yourself questions interviewers actually want to hear: do I need three separate notification services for each channel, or can one service handle routing logic and delegate to channel-specific workers? Where does the queue sit so a slow SMS provider doesn't block email delivery? What happens to a notification that fails three times, does it disappear or does it land somewhere you can inspect and replay?

I ended up landing on an API Gateway routing to dedicated notification services per channel, an MSK-backed queue decoupling ingestion from delivery, separate delivery workers per channel so failures don't cascade, and a dead letter queue catching anything that exhausts its retries. Monitoring and a logging layer sat underneath the whole thing because "zero data loss" is a claim you need to be able to prove, not just assert.

None of that is exotic. But getting there by actually building it, watching the cost meter, and getting nudged when something looked off was a completely different learning experience than reading a blog post about the same architecture.

Why Building Beats Watching

There's a specific moment in system design prep where things click, and it's not when you finish a course or finish a book. It's when you make a wrong call, see why it's wrong, and fix it yourself.

If you design a notification system and only use one shared queue for both push and SMS, and SMS providers are notoriously slow and flaky, you'll bottleneck your push notifications behind retry storms from SMS failures. You can read that warning in an article. Or you can build it, watch your design choke, and never forget the lesson again.

That's the difference between system design courses that hand you diagrams to memorize and a system design lab where you're the one placing components, connecting them, and dealing with the fallout of bad choices.

A Practical Way to Practice

If you're prepping for interviews right now, here's what I'd actually recommend instead of just rereading the same five blog posts about designing a chat app:

Treat constraints as the real test. Latency targets, budget caps, data guarantees, these aren't flavor text. They're the actual interview question. Anyone can draw boxes and arrows. The skill is justifying why this box and not that one, given the numbers you were handed.

Practice failure scenarios, not just happy paths. What happens when your queue backs up. What happens when one delivery channel goes down. What happens when you hit 10x traffic. If your practice sessions never break, you're not really practicing.

Get feedback that's specific to your design, not generic. A mock interview with a friend who's also prepping is fine, but it's not the same as something that flags "hey, your retry logic here introduces a single point of failure" while you're actually building.

Repeat with variation. Don't design the same notification system five times. Design a URL shortener, then a rate limiter, then a chat system, then come back to messaging systems with a different constraint set. The patterns start showing up, and so does your confidence.

I've been using Scaledojo for exactly this kind of structured practice, it's basically a sandbox where you assemble real architectures against specific design challenges, see live cost and latency tradeoffs as you build, and get nudged when a connection or component choice doesn't hold up. It scratches the itch of "I want to actually build this, not just talk about building it," which for me was the missing piece between knowing system design concepts and being able to perform under interview pressure.

The Real Takeaway

System design interviews aren't testing whether you've memorized the architecture for Twitter. They're testing whether you can take ambiguous requirements, apply real constraints, and make defensible engineering decisions out loud, in real time, while someone watches.

The only way to get good at that is to do it, over and over, with feedback loops tight enough that you actually learn from each rep instead of just feeling busy.

So if your prep so far has mostly been reading and watching, try flipping it around. Pick a system, set real constraints on yourself, build it, break it, and fix it. Your future interviewer will be able to tell the difference between someone who memorized a diagram and someone who's actually done the work.

What's the hardest system design question you've run into in an interview? I'd genuinely like to know, drop it in the comments.

THE LIE NOBODY TELLS YOU ABOUT SYSTEM DESIGN INTERVIEWS

Sophia Bennett — Mon, 15 Jun 2026 18:33:24 +0000

Every article about system design interviews tells you the same thing.

"Learn the components. Understand trade-offs. Practice on paper."

And so you do. You read ByteByteGo. You finish Grokking. You watch Alex Xu's videos. You feel ready.

Then you walk into the interview and freeze.

Not because you don't know the concepts. You know them cold. You freeze because knowing system design and actually building a system design are two completely different skills and nobody told you that before you sat down across from a Google engineer.

Reading about how a rate limiter works is like reading about how to swim. At some point you have to get in the water.

Here is the uncomfortable truth: most engineers preparing for system design interviews are doing the equivalent of reading swimming manuals and calling it practice. They can describe a CDN. They cannot design a content delivery architecture under time pressure with someone watching.

The gap is not knowledge. The gap is execution.

Why this matters more than you think

I spoke to a senior engineer who failed his Meta system design round three times. He had read every book. He had the components memorized. He failed because when the interviewer asked him to walk through a Twitter feed design, he had never actually done it. He had only read about it.

On his fourth attempt, after three months of building real architectures interactively, he passed. Not because he learned something new. Because he had trained the muscle, not just the brain.

That is the difference between knowing and doing.

What you should actually be practicing

Sit down with a blank canvas. No notes. Design a URL shortener from scratch. Not in your head. On paper, on a whiteboard, on whatever you have. Draw the components. Connect them. Decide where the database goes. Pick your caching strategy. Justify every single choice out loud as if someone is listening.

Do that once a week for two months and you will interview differently than 95% of the candidates in the room.

The engineers who clear FAANG system design rounds are not smarter than you. They have just built more architectures than you. That is a fixable problem.

Finally Joined DEV Community!

Sophia Bennett — Mon, 15 Jun 2026 05:43:12 +0000

Hey everyone,

I've been building software for about 14 years now, mostly working on enterprise applications, distributed systems, and cloud-based platforms. Over the years I've had the opportunity to work with some great teams and products, but I've mostly been a reader rather than someone who shares publicly.

I'm joining DEV because I think it's time to change that.

A lot of my career has been spent solving the kinds of problems that don't make it into tutorials legacy systems, scaling challenges, production incidents at odd hours, difficult migrations, and all the trade-offs that come with building software in the real world.

I'm hoping to share some of those experiences, along with lessons learned (sometimes the hard way), and connect with other engineers who enjoy talking about architecture, engineering culture, and building reliable systems.

Looking forward to learning from the community and contributing where I can.

What was the biggest lesson you learned after your first few years in the industry?

DEV Community: Sophia Bennett