30+ System Design Problems, Explained Through the Failures Behind Them

Three situations:

• Two users book the last seat at the same moment. Both get a confirmation.
• A payment request times out. The customer retries, and their card is charged twice.
• A viral product's cache entry expires, and thousands of requests hit the database at once.

All three are system design problems, and none of them is fixed by adding more servers.

The real skill is understanding what breaks, why it breaks, and which fix doesn't create a bigger problem. A URL shortener teaches caching and ID generation. A payment system teaches idempotency. A message queue teaches durability and failure recovery.

This post is a roadmap of 31 designs, ordered from foundational to advanced, plus the framework I use to reason through any of them.

The one question that matters

For every improvement you add, ask:

What new problem does this introduce?

• A cache cuts latency but introduces staleness.
• A queue decouples services but introduces retries, duplicates and ordering.
• Replication improves availability but complicates consistency.

A diagram shows the boxes. Understanding these exchanges is what makes it a design.

A six-step framework, applied once

1. Requirements: what must the system do, and what must never go wrong?
2. Scale: requests per second, storage, read/write ratio, spikes.
3. Simplest working design: one app server, one database. No Kafka without a reason.
4. First bottleneck: what saturates at 10x traffic?
5. One fix at a time: each tool answers a specific problem.
6. Break it: crash a node mid-write, time out after success, split two regions.

Here it is on ticket booking:

1. Requirement: a seat must never be sold twice.
2. Scale: a popular event means a huge spike on a small set of rows.
3. Simplest design: check if the seat is free, then mark it sold.
4. Bottleneck: two requests both see "free" before either writes. Checking and reserving are two separate operations, which is a race condition.
5. Fix: make the reservation atomic (a conditional update or a unique constraint), then add a temporary hold with expiry so payment has time to complete.
6. Break it: the user abandons payment, so the hold must expire. The payment succeeds but the response is lost, so you need idempotency.

Notice step 6 sends you straight into the payment problem. That's no accident, and I come back to it below.

---

Part 1: Foundations

Caching, queues, indexing, messaging and coordination.

1. URL Shortener: distributed ID generation, caching, viral links
2. Notification System: queues, retries, backoff, dead-letter queues
3. Search & Autocomplete: inverted indexes, tries, ranking
4. Dropbox: chunked uploads, dedup, sync conflicts
5. WhatsApp: WebSockets, ordering, offline delivery
6. Distributed Task Scheduler: leader election, leases, crash recovery

Part 2: Intermediate

Transactional correctness, real-time updates, cross-service workflows.

1. Payment System: idempotency keys, ambiguous outcomes, reconciliation
2. E-commerce: inventory reservation, flash sales, outbox pattern
3. Ticket Booking: race conditions, seat holds, admission control
4. Hotel Reservation: multi-night atomic allocation
5. YouTube: async transcoding, adaptive bitrate, CDN
6. Recommendation System: candidate generation, multi-stage ranking
7. Google Docs: operational transformation, presence
8. Gaming Leaderboard: sorted sets, sharded ranking
9. Splitwise: immutable ledgers, debt simplification
10. Feature Flag System: local evaluation, safe fallbacks

Part 3: Advanced distributed systems

Consistency, replication, partitioning, fault tolerance.

1. CDN: edge caching, invalidation, origin shielding
2. Instagram: feed fan-out, hot users
3. Twitter / X: timelines, hybrid fan-out
4. Uber: geospatial indexing, driver matching
5. Netflix: control vs data plane, regional resilience
6. Stock Exchange: matching engine, deterministic sequencing
7. Distributed Logging: buffering, retention tiers
8. Distributed Rate Limiter: token bucket, fail-open vs fail-closed
9. Message Queue & Pub/Sub: partitioned logs, offsets, idempotent consumers
10. Key-Value Store: WAL, LSM trees, quorum replication
11. Web Crawler: URL frontier, politeness, dedup
12. Distributed Cache: stampede protection, hot keys

Part 4: Modern AI systems

The model is one component. The rest is still APIs, caching, queues, authorization and cost control.

1. AI Assistant (ChatGPT-style): streaming, memory, tool calling
2. Enterprise RAG: hybrid retrieval, access control, citations
3. AI Gateway: model-aware routing, quotas, failover

---

Three questions, answered

If a popular cached item expires and 100,000 requests arrive at once, how do you stop them all hitting the database? Let one request rebuild the value while the others wait or get a slightly stale copy. Techniques include request coalescing, a short lock on the key, and refreshing before expiry. Adding jitter to TTLs stops many keys expiring together.

If a consumer processes a message but crashes before acknowledging it, what happens? The queue redelivers it, because from the broker's view it was never handled. So consumers must be idempotent: processing the same message twice has the same effect as once. You get at-least-once delivery from the queue and the "exactly once" effect from your consumer.

Should a feature flag outage take your application down? No. Evaluate flags locally from a cached copy of the configuration, and fall back to safe defaults if the control plane is unreachable. The flag service should be a dependency you can lose without losing requests.

Connections that make it click

Different systems fail for surprisingly similar reasons.

• Ticket booking and payments: both must not duplicate an operation. Atomic reservation protects inventory. Idempotency keys protect against retries and unknown outcomes. Same principle, different tools.
• Notifications and message queues: a queue lets you accept a request now and deliver later. It also hands you duplicates, retry exhaustion and backpressure. It makes failure manageable, not absent.
• URL shortener, CDN and distributed cache: all move hot data closer to the reader. The shared lesson is what happens when the cache is wrong, down, or suddenly empty.
• Assistant, RAG and gateway: three layers of one AI stack. The assistant orchestrates context and tools, RAG retrieves authorized knowledge, and the gateway governs routing, quotas and spend.

Where to start

Goal	Suggested order
Backend beginner	URL Shortener, Notifications, Search, WhatsApp, Task Scheduler
Interview prep	URL Shortener, Ticket Booking, Payments, Uber, YouTube, Distributed Cache
Distributed systems	Rate Limiter, Message Queue, KV Store, Cache, Logging, Stock Exchange
Real-time apps	WhatsApp, Leaderboard, Google Docs, Uber, Twitter/X
AI backend	AI Assistant, Enterprise RAG, AI Gateway

A quick challenge

Pick one system and answer these before opening any walkthrough:

1. What is the one correctness or reliability requirement that matters most?
2. What's the simplest architecture that satisfies it?
3. What becomes the first bottleneck at 10x traffic?
4. What happens if that component crashes mid-operation?
5. What trade-off does your next improvement introduce?

If you can't answer one, that's your next thing to learn.

Why I built SeeItFlow

A static diagram shows where components sit. It doesn't show how the system got there.

I wanted to see why each piece is introduced, so I built SeeItFlow, a free visual learning platform. Each of the 31 designs covers requirements and estimation, an evolving visual design journey, architecture deep dives, trade-offs, failure and scale scenarios, and an interview perspective. Many walkthroughs have synchronized narration and step controls. No account is needed, and signing in only saves your progress.

I'm also publishing longer write-ups in this DEV series.

Your turn

Which system design was hardest for you to understand at first? And which failure scenario would you most like to see visualized? I'm especially interested in the things architecture diagrams don't explain.