Most system design content teaches you to draw the "correct" final architecture. Almost none of it makes you sit through the moment where that architecture breaks.
That gap is a problem, because "does your design survive failure" is exactly what senior and staff interviews probe for. So I want to walk through one specific scenario end to end-build it, break it, watch the metrics, fix it-the way it actually plays out when you're doing this hands-on instead of just describing it on a whiteboard.
The setup: a URL shortener, the "easy" level
Everyone starts system design prep with a URL shortener, so let's use it. Baseline design:
Client → Load Balancer → Web Server → Database
Web server handles hash generation and lookups
Single database, no replication yet
On paper this looks fine. It passes the "does it work" check. It does not pass the "does it survive Tuesday afternoon" check, and that's the part most prep material skips entirely.
Now break it
This is where I've been using ScaleDojo's Murphy's Lab, a chaos engineering simulator built directly into the HLD challenges. Instead of just submitting a diagram and getting a checkmark, you can inject real failure conditions into the running system and watch it respond live:
Crash a specific server
Spike incoming traffic
Sever a network connection between components
Watch latency, error rate, and throughput update in real time as the system reacts
Run that crash scenario against the baseline design above and here's roughly what happens: your single web server goes down, the load balancer has nowhere to reroute traffic, error rate climbs hard, and if your database was co-located or under-provisioned, you get cascading latency on top of it. Nothing about the diagram was "wrong" in a static sense-it just wasn't built to survive the failure it eventually met.
That distinction matters a lot in an interview. A candidate who can point at a diagram and say "this is correct" is worth less than a candidate who can say "this specific part fails first, and here's why."
What the fix actually looks like
Once you've watched the failure happen instead of just imagining it, the fixes stop being abstract vocabulary and start being specific responses to a specific symptom:
Redundant web servers behind the load balancer-so one crash doesn't take down the whole path
Database replication with a read replica-so read traffic doesn't die with the primary
Circuit breakers-so a failing dependency doesn't take the whole request chain down with it
Graceful degradation-return cached or partial results instead of a hard failure when a downstream service is unhealthy
Auto-recovery / health checks-so the load balancer stops routing to a dead instance without a human intervening
Rebuild the architecture with those in place, rerun the same crash scenario, and the difference in the live metrics is immediate-latency stabilizes, error rate recovers instead of climbing, throughput holds. You're not trusting that the fix works because a textbook said so. You watched it hold up against the exact failure that broke the first version.
Why this is the part interviews actually test
The "design a URL shortener" prompt is never really about URL shortening. It's a vehicle for the interviewer to ask "what happens when this breaks," usually multiple times, with the constraints shifting each time. If your prep has only ever involved reading about replication and circuit breakers-never actually watching a system fail without them and recover with them-you're rehearsing vocabulary, not the skill.
This is also why static diagram-based prep tends to produce candidates who can describe fault tolerance patterns but freeze slightly when asked to reason through this specific failure, in this specific system, right now. The knowledge is there; the rep under pressure isn't.
If you want to run through this exact scenario yourself, Murphy's Lab is free to try on the URL shortener level-no signup required to start.
Curious what other failure scenarios people think are underrated in interview prep-network partitions and thundering herd don't get nearly enough attention compared to "the database died." What's on your list?
Top comments (3)
❤️🙌
Great
Insightful, I really tried the platform. Daily puzzle of scaledojo.dev is exceptional!