DEV Community: mseada

Designing data-intensive applications: replication, partitioning, quorums

mseada — Thu, 22 Oct 2026 06:13:33 +0000

Part 4 of 8 · week 4 of the learning path

When data volume, speed or complexity is the main challenge, the design comes down to four choices: how data is stored, copied, split and kept consistent.

Learning objectives

Choose a storage engine and data model for a workload.
Reason about replication, partitioning and the anomalies they create.
Pick transaction and consistency levels, and decide between batch and stream processing.

Storage

Choice	Strength	Used by
B-tree	Updates in place, fast reads, mature indexing	Most relational databases
LSM-tree	Fast writes through sequential appends and background compaction	Cassandra, RocksDB and similar
Row store	Whole-record reads and writes (OLTP)	Application databases
Column store	Scanning a few columns over many rows (OLAP)	Warehouses and analytics

Replication

Leader and followers: writes go to the leader and copy to followers, synchronously or asynchronously; simple and common.
Multi-leader: several leaders accept writes, useful across regions, but conflicts must be resolved.
Leaderless with quorums: with N replicas, a write needs W acknowledgements and a read asks R; when R plus W is greater than N, a read overlaps the latest write.
Replication lag causes anomalies: reading your own write from a lagging follower, or seeing data go backwards. Route a user's reads to the leader after a write, or track versions.

Partitioning

By key range: keeps related keys together and supports range scans, but sequential keys such as timestamps create hot spots.
By hash: spreads load evenly, but loses range scans.
Rebalancing: use many more partitions than nodes, or consistent hashing, so adding a node moves little data.
Identifiers: avoid a single auto-increment counter at scale; use IDs that combine a timestamp, a shard or machine part and a sequence so they are unique and roughly sortable.

Transactions and consistency

ACID transactions protect against partial failure; isolation levels trade safety for speed: read committed, snapshot isolation and serializable.
Know the anomalies: dirty reads, lost updates and write skew, where two transactions each read the same data and write different rows so that together they break a rule.
Distributed transactions with two-phase commit are slow and fragile; prefer sagas where the business allows.
Consistency ranges from linearizable (acts like one copy) to eventual. Choose the weakest level a feature can tolerate, and the strongest for money and inventory.

Batch and stream: batch jobs (Spark, MapReduce) process bounded data cheaply and accurately; stream processors (Flink, Kafka Streams) handle unbounded data with low latency and must deal with windows and late events. Design stream jobs to be idempotent so replays give the same result.

Worked numbers: 1 billion users at 1 KB each is about 1 TB. With three replicas that is 3 TB; at 500 GB per node you need about six nodes before growth and headroom. With N=3, W=2 and R=2 a read always overlaps the latest acknowledged write.

Real-world examples

Amazon Dynamo (2007): a published design for a highly available key-value store using leaderless replication and quorums; it inspired many later databases.
Google Spanner (2012): a globally distributed database that offers strong consistency by using tightly synchronized clocks, showing that the trade-off can be pushed with specialized hardware.
Instagram IDs: the team documented generating sortable IDs from a timestamp, a shard ID and a sequence, so each shard creates unique IDs with no central counter.

Design exercise: design the storage for a chat history store holding 100 billion messages. Choose the engine, the partition key, the replication setting and the consistency level, and explain how you would serve the latest 50 messages of a conversation.

Quiz 4 (5 questions, pass mark 4)

Why are LSM-trees good for write-heavy workloads?
- A) They give faster random reads than B-trees in every case
- B) Writes are sequential appends merged later by compaction
- C) They need no disk
- D) They give transactions across nodes
With N=3 replicas, which setting makes a read overlap the latest acknowledged write?
- A) W=1, R=1
- B) W=1, R=2
- C) W=2, R=2
- D) W=0, R=3
What causes the read-your-writes anomaly?
- A) Too many indexes
- B) Clock skew in browsers
- C) Encrypted disks
- D) Reading from a follower that has not yet received your recent write
Why hash-partition a table keyed by timestamp instead of using key ranges?
- A) All current writes hit one range partition, a hot spot, while hashing spreads them
- B) Timestamps are sensitive data
- C) Range queries become faster
- D) It removes the need for replicas
Two on-call doctors each see the other on call and both go off call. What is this and what prevents it?
- A) A dirty read, fixed by a cache
- B) Write skew, fixed by serializable isolation or explicit locking
- C) A network partition, fixed by retries
- D) A deadlock, fixed by more replicas

Quiz 4 answers

B
C with N=3, W=2 and R=2, R plus W is 4, which is greater than N, so a read overlaps the latest write.
D
A
B

Sources and further reading

Designing Data-Intensive Applications by Martin Kleppmann, the main reference for this module.
Dynamo: Amazon's highly available key-value store, the 2007 paper (hosted copy).
Spanner: Google's globally distributed database, the OSDI 2012 paper.
Sharding and IDs at Instagram, on time-sortable 64-bit IDs per shard.

Part 4 of 8 in the System Design Learning Path series. Next: part 5.

Event-driven architecture: events, outbox and idempotency

mseada — Thu, 22 Oct 2026 06:12:58 +0000

Part 3 of 8 · week 3 of the learning path

In an event-driven system a service records a fact that has happened and others react to it on their own time, which absorbs spikes and lets you add consumers without touching the producer.

Learning objectives

Tell events, commands and queries apart and name them correctly.
Choose between a queue and a log, and design for duplicates and ordering.
Apply the outbox, idempotent consumer, dead-letter and CQRS patterns.

Concepts

Concept	Meaning	Example name
Event	A fact that already happened, named in the past tense	OrderPlaced
Command	A request to a specific service to do something	ChargeCard
Query	A request for data	GetOrder
Queue	Each message goes to one consumer and is removed	Work distribution
Log	Ordered, retained stream that many consumers read at their own pace	Event streaming, replay
Consumer group	Instances that share the partitions of a topic	Scale out a consumer

Delivery and ordering

Most systems deliver at least once, so a message can arrive twice. Make handlers idempotent, for example by recording processed event IDs, so duplicates do no harm.
Ordering is only guaranteed within a partition. Use a key such as the order ID so all events for one order land in the same partition.
Move messages that keep failing to a dead-letter queue after limited retries, and alert on it.
Version event schemas, keep changes backward compatible and register them so consumers can evolve safely.

Patterns

Outbox: write the data change and the event in one database transaction, then a relay publishes the event, so you never update data without publishing or the reverse.
Saga: coordinate multi-step workflows through events, with compensating events on failure.
CQRS: keep the write model separate from one or more read-optimized models that are updated from events.
Event sourcing: store the sequence of events as the source of truth and rebuild state from it; powerful for audit, but adds complexity.

Worked example, order flow. The Order service saves the order and writes OrderPlaced to its outbox. Inventory reacts and emits StockReserved. Payment reacts and emits PaymentCaptured. Shipping starts only after that. Later the team adds an email service and an analytics pipeline as new consumers of the same events, with no change to Order.

Pitfalls: treating events as remote calls that expect a reply, event chains nobody can follow without tracing, breaking schema changes, assuming global ordering, and putting huge payloads in events instead of references.

Real-world examples

LinkedIn and Kafka: LinkedIn built Kafka around 2010 to move activity and operational data between systems at scale, and open sourced it; it became the standard log for event streaming.
Stripe webhooks: Stripe tells merchants about payments by sending events to their endpoint, retrying on failure, so merchants must handle duplicates and out-of-order delivery. It is a public example of at-least-once delivery.
Netflix: it streams very large volumes of operational and viewing events through a central pipeline for monitoring and analytics, decoupling the producers from the many teams consuming them.

Design exercise: design the events for a food delivery order (placed, accepted, picked up, delivered, refunded). Choose partition keys, say where you need an outbox, and describe what a consumer does on a duplicate and on an out-of-order event.

Quiz 3 (5 questions, pass mark 4)

Which of these is a command rather than an event?
- A) OrderPlaced
- B) PaymentCaptured
- C) ChargeCard
- D) UserRegistered
What does at-least-once delivery mean for a consumer?
- A) A message may arrive more than once, so handlers must be idempotent
- B) Duplicates never happen
- C) Messages may be lost
- D) Exactly one copy is always delivered
What does the outbox pattern guarantee?
- A) Messages are encrypted
- B) Ordering across all topics
- C) Faster consumers
- D) The data change and the event are committed together and the event is then published reliably
How do you keep all events for one order in order on a partitioned log?
- A) Use a random key
- B) Use the order ID as the key so they share a partition
- C) Create a topic per event
- D) Make messages bigger
What is CQRS?
- A) A type of queue
- B) A database index
- C) Separating the write model from one or more read-optimized models, often updated by events
- D) A caching protocol

Quiz 3 answers

C
A
D the outbox writes the data change and the event in one transaction, so neither can be lost without the other.
B
C

Sources and further reading

Apache Kafka documentation, starting with the introduction to event streaming.
The Log: what every software engineer should know about real-time data's unifying abstraction by Jay Kreps of LinkedIn, on why a log underlies messaging and replication.
Transactional outbox pattern on microservices.io.
Stripe webhooks documentation, a public example of at-least-once event delivery with retries and signatures.
Amazon Builders' Library: timeouts, retries, and backoff with jitter, for safe retries and idempotent APIs.

Part 3 of 8 in the System Design Learning Path series. Next: part 4.

Microservices: boundaries, sagas and failure

mseada — Thu, 22 Oct 2026 06:11:51 +0000

Part 2 of 8 · week 2 of the learning path

Microservices let many teams release and scale parts of a system independently, but they trade simple code for a distributed system, so use them when the organization needs the independence.

Learning objectives

Decide between a modular monolith and microservices for a given team and product.
Draw service boundaries, own data per service and choose how services communicate.
Keep multi-service workflows consistent and resilient.

Principles

Boundaries follow business capabilities, so a change usually touches one service.
Each service owns its data; others reach it only through its API or events.
Services deploy independently, and the team that builds a service runs it.
Start with a well-structured modular monolith and split out services when a real need appears.

Communication

Style	Use it for	Watch out for
Synchronous (REST, gRPC)	Queries and steps where the caller needs an answer now	Latency adds up, failures cascade
Asynchronous (events, queues)	Notifying others, long steps, absorbing spikes	Harder to follow, needs idempotency
API gateway	One entry point for clients: auth, limits, routing	Keep business logic out of it
Service discovery	Finding healthy instances	Needs health checks and fast failover

Keeping workflows consistent: a single database transaction cannot span services, so use a saga: a sequence of local steps where each step has a compensating action if a later step fails. A saga can be orchestrated by one coordinator or choreographed through events.

Worked example, checkout. Cart, Order, Inventory, Payment and Shipping are separate services. The order saga reserves stock, charges the card, then books shipping. If the charge fails, the compensating action releases the reserved stock and marks the order failed. Each step is idempotent so a retry does no harm.

Resilience patterns: timeouts on every call; retries with exponential backoff and jitter; circuit breakers that stop calling a failing service for a while; bulkheads that isolate resources per dependency. Be careful with retries: if three layers each try a call three times in total, one user action can become 27 calls to the bottom service.

Delivery: one repository or several, but each service has its own pipeline, containers, canary releases and rollback. Use contract tests so teams can change services without breaking each other, and pass a trace ID through every call.

Real-world examples

Amazon and Netflix: both moved from large applications to many small services owned by single teams, which let them release thousands of times a day, at the cost of heavy investment in tooling and operations.
Segment (2018): the company published that it moved a set of tightly coupled microservices back into one service because the operational cost outweighed the benefits at its team size.
Shopify: it runs a very large modular monolith with enforced internal boundaries, showing that good structure matters more than the number of deployables.

Design exercise: take the checkout example and write the saga as a table of steps, compensating actions and events. Then list two reasons you would keep it as a modular monolith for a team of eight.

Quiz 2 (5 questions, pass mark 4)

When are microservices a good choice?
- A) A team of three building a simple app
- B) You want fewer deployments
- C) Many teams need independent releases and parts of the system scale differently
- D) You want to share one database across services
What is a compensating action in a saga?
- A) Releasing reserved stock when the payment step fails
- B) Adding a cache
- C) Rolling back one database transaction across services
- D) Locking all tables
What does a circuit breaker do?
- A) Encrypts calls
- B) Caches responses
- C) Balances load
- D) Stops calling a failing dependency for a while so failure does not cascade
What do the Segment and Shopify examples show?
- A) Monoliths are obsolete
- B) Microservices bring operational cost, and a well-structured modular monolith can be the better choice
- C) You should always use microservices
- D) You should never split a system
Three layers each try a call three times in total. How many calls can reach the bottom service from one user action?
- A) 3
- B) 9
- C) 27
- D) 1

Quiz 2 answers

C
A
D
B
C three layers each making three attempts give 3 times 3 times 3, which is 27 calls.

Sources and further reading

Microservices by James Lewis and Martin Fowler, the article that defined the style and its characteristics.
Saga pattern on microservices.io, for orchestration versus choreography.
Timeouts, retries, and backoff with jitter from the Amazon Builders' Library, the source for the retry pitfalls above.
Why Segment returned to a monolith on InfoQ, summarizing Segment's own write-up.
Deconstructing the monolith from Shopify Engineering, on the modular monolith.
Prime Video: scaling up the monitoring service and reducing costs by 90%, with an InfoQ summary.

The Amazon and Netflix history in this module is from general knowledge and has no cited source here; add one if you want it referenced.

Part 2 of 8 in the System Design Learning Path series. Next: part 3.

System design fundamentals: estimate, scale and trade off

mseada — Thu, 22 Oct 2026 05:29:19 +0000

Part 1 of 8 · week 1 of the learning path

Every large design is the same few building blocks combined to meet numbers, so learn the blocks and learn to estimate before drawing any boxes.

Learning objectives

Turn a product request into functional and non-functional requirements.
Do back-of-envelope estimates for traffic, storage and bandwidth.
Explain what each core building block does and when to use it.

Requirements first

Functional: what the system does (send a message, upload a video, charge a card).
Non-functional: how well it does it: scale, latency, availability, durability, consistency, security and cost.
Ask about scope and priorities before designing; a design for 1,000 users and one for 100 million are different systems.

Estimation cheat sheet

Fact	Rule of thumb
Seconds in a day	About 86,400, so requests per day divided by 86,400 gives average QPS
Peak versus average	Plan for 2 to 3 times average, more for spiky products
Memory read	About 100 nanoseconds
SSD random read	About 100 microseconds
Round trip inside one datacenter	About 0.5 milliseconds
Round trip across continents	About 150 milliseconds
Storage	Records per day times record size times retention

Worked example: 10 million daily users each make 20 reads and 2 writes a day. Reads are 200 million a day, about 2,300 per second on average and about 7,000 at a 3x peak. Writes are 20 million a day, about 230 per second and about 700 at peak. At 1 KB per write, that is 20 GB a day, or about 7.3 TB a year before replication.

Building blocks

Block	Job	Key choices
Load balancer	Spreads requests across instances	Layer 4 or layer 7, health checks, algorithm
Cache	Serves hot data from memory	Cache-aside versus write-through, eviction, expiry, stampede protection
CDN	Serves static and cacheable content near users	What to cache, expiry, invalidation
Relational database	Structured data with transactions	Indexes, replicas, sharding
NoSQL store	Scale and flexible models	Key-value, document, wide-column, graph
Queue or log	Decouples producers and consumers, absorbs spikes	Ordering, retention, delivery guarantees
Object storage	Cheap durable storage for large files	Lifecycle, access control
Search index	Full text and filtered queries	Refresh lag, relevance
Rate limiter	Protects services from abuse and overload	Token bucket, per-user or per-key limits

Scaling basics: make services stateless so any instance can serve any request; scale out behind a load balancer; cache reads; replicate data for reads and safety; shard data when one node cannot hold or write it all. When the network splits, a distributed store must choose consistency or availability, so decide per feature which matters more.

Design exercise: a URL shortener gets 100 million new links a month with ten reads per write. Estimate write and read QPS, storage over five years and cache size for the hottest 20% of links, then list the blocks you would use.

Quiz 1 (5 questions, pass mark 4)

About what is the average QPS for 86.4 million requests a day?
- A) 100
- B) 1,000
- C) 10,000
- D) 86,400
Why is a stateless service easy to scale?
- A) It uses sticky sessions
- B) It keeps sessions in local memory
- C) Any instance can serve any request, so you add instances behind a load balancer
- D) It needs a single large server
What is cache-aside?
- A) The cache is the only store
- B) Reads bypass the cache
- C) The database pushes every write into the cache
- D) The app checks the cache first and, on a miss, reads the database and fills the cache
What does a CDN mainly do?
- A) Serves static and cacheable content from locations near users, cutting latency and origin load
- B) Stores financial transactions
- C) Runs SQL queries
- D) Encrypts the database
How should you size for traffic peaks?
- A) Use the average only
- B) Ignore peaks
- C) Plan for a multiple of average, commonly 2 to 3 times or more, plus headroom
- D) Assume every second is a peak

Quiz 1 answers

B 86.4 million divided by 86,400 seconds is 1,000 requests per second.
C
D
A
C

Sources and further reading

Latency numbers every programmer should know, the Jeff Dean and Peter Norvig figures behind the estimation table; the numbers are rough and dated, so use them for orders of magnitude.
System Design Primer, an open study guide covering the building blocks in this module.
Designing Data-Intensive Applications by Martin Kleppmann, for caching, replication and consistency in depth.

Part 1 of 8 in the System Design Learning Path series. Next: part 2.

System design learning path: from fundamentals to WhatsApp, YouTube and Stripe

mseada — Thu, 22 Oct 2026 05:05:32 +0000

An eight-week path that teaches software engineers to design large systems: core building blocks, three architecture styles, then full designs of WhatsApp, YouTube and Stripe, with a quiz after every module.

Goal: every engineer can take a vague product idea, ask the right questions, estimate the load, sketch an architecture and defend its trade-offs, using one repeatable method.

Audience: backend, full-stack and mobile engineers, tech leads and architects. Basic knowledge of web APIs and databases is assumed.

Format: 8 modules, about 3 hours per week. Each module has a short talk, self-study reading, one design exercise and a 5-question quiz (pass mark 4 of 5). The three case studies each include an architecture diagram and worked numbers.

The design method used in every module

Clarify requirements, both functional and non-functional (scale, latency, availability, consistency).
Estimate load and storage with back-of-envelope math.
Define the API and the core data model.
Draw the high-level architecture.
Deep dive on the hardest two or three parts.
Name the trade-offs, failure modes and what you would monitor.

Levels

Level	Requirements
Foundation	Quizzes 1 to 4 passed
Practitioner	Foundation, Quizzes 5 to 7 passed, and the final assessment at 9 of 12 or more
Architect	Practitioner, final assessment at 11 of 12 or more, and the capstone at 70 points or more

Outcomes by the end

Size a system from users and usage, and spot the bottleneck before building.
Choose between monolith, microservices and event-driven styles, and say why.
Reason about replication, partitioning, transactions and consistency.
Design a chat system, a video platform and a payment system, including failure handling.

The series

Part	Topic
1	System design fundamentals: estimate, scale and trade off
2	Microservices: boundaries, sagas and failure
3	Event-driven architecture: events, outbox and idempotency
4	Designing data-intensive applications: replication, partitioning, quorums
5	System design case study: WhatsApp
6	System design case study: YouTube
7	System design case study: Stripe and payment systems
8	Capstone and final assessment

Each post ends with a quiz. Answers are hidden in a collapsible block, so try the questions first.