Ask most engineers how DynamoDB scales and you get one answer. It partitions the data. That is true, and it is also the least interesting sentence you can say about it, because sharding a dataset across machines is a decades-old idea and most implementations of it are painful. What makes DynamoDB feel like it does not care how large your table is, returning a point read in single-digit milliseconds whether it holds a megabyte or a petabyte, is a stack of deliberate design decisions layered on top of that partitioning, each one a trade you can learn from and reuse in your own systems.
This piece is the long version, working from why a distributed key-value store is hard at all through consistent hashing, quorums, conflict resolution, the storage engine, secondary indexes, hot partitions, and multi-region global tables. Along the way I will separate the 2007 Dynamo paper, the conceptual ancestor, from production DynamoDB, which made deliberate changes to that design. By the end you should be able to defend both why the system is fast and exactly what you gave up to get that speed. One note on numbers: any specific latency or throughput figure here is an industry-typical range or an illustration, not a measured AWS-internal metric.
Why a distributed key-value store is genuinely hard
Start with the problem, because the whole design is a chain of answers to it. You have more data and traffic than one machine can hold or serve, so you must spread both across many machines. The moment you do, four hard questions appear at once, each with a wrong answer that looks fine until production. Which machine owns which key, in a way that stays balanced and does not reshuffle the whole dataset every time you add or remove a node? What keeps the data available when a machine dies, since at the scale of thousands of nodes something always is? When two copies of the same key disagree, which one is right? And how do you keep a single-key operation fast while the table grows without bound and the system is actively moving data around underneath you?
The partition key decides everything downstream, and the data is always in motion as partitions split and move.
The unifying decision that falls out of these questions is that the partition key is the most consequential thing in the system. It is hashed to decide which partition, and therefore which physical nodes, own an item. Get it right and traffic spreads evenly across the fleet and throughput grows almost linearly as you add hardware. Get it wrong and you concentrate load onto one partition, one node, one CPU, and no amount of provisioned capacity saves you. Everything that follows is built so a well-chosen key gives flat latency at any size, and so no single request ever touches more than a handful of nodes.
The request path stays thin on purpose
A client addresses an item by its primary key and talks to the service through a stateless request layer whose first job is routing. It hashes the partition key and looks up which partition owns that hash range in the partition metadata. That lookup tells it which storage nodes hold the replicas, and it forwards the operation. For a write it targets the partition leader; for a cheap read it can target any replica.
A request router hashes the key, consults the ring map, and forwards to the replica group on the owning storage nodes.
The important property of this tier is that it holds no durable state. Routing is a pure function of the key plus a metadata map, so you can run as many routers as you need and any of them can serve any request. This is the same instinct you see in every system that scales cleanly: keep the fast path simple and local, and push the slow global coordination somewhere it cannot get in the way. The router authenticates the caller, checks throughput, finds the partition, and forwards. That is all.
Underneath the routers sit the storage nodes, grouped into replica sets, one per partition. Underneath all of it sits a control plane the data path never touches directly. It tracks node membership and health, decides when a partition should split or move, drives the data movement while the partition stays online, and updates the metadata map the routers consult. The critical-path get or put never waits on it. In the Dynamo paper this membership spread through gossip; production DynamoDB backs it with a more centralized, consensus-backed metadata service. The principle is identical: the fast path is simple and local, and the expensive global agreement is kept off to the side.
Consistent hashing and why virtual nodes exist
Now to the first hard question, ownership. The naive answer is to hash the key and take it modulo the number of nodes. That works until the node count changes, then fails completely, because changing N remaps almost every key at once, moving almost all of your data. At scale that is not a rebalance, it is an outage.
Keys and nodes both hash onto a ring, and each physical machine appears many times as virtual nodes to smooth the distribution.
Consistent hashing solves this. Place both keys and nodes on a ring, a circular hash space, and let a key be owned by the first node clockwise from its hash. Now adding or removing a node only reassigns the keys in the arc immediately next to it, so roughly one over N of the data moves instead of nearly all of it. That single change is what makes online scaling possible at all.
The basic scheme has two flaws, and the fix for both is the same. First, if each node sits at one random point on the ring, chance alone gives some nodes much larger arcs than others, so load is uneven. Second, when a node fails, its entire arc dumps onto its single clockwise successor, which can then tip over from the doubled load. The Dynamo paper fixes both with virtual nodes: each physical machine is represented by many points on the ring rather than one. This smooths the distribution because the law of large numbers evens out the arc sizes, spreads a failed machine's load across many peers instead of one, and lets a more powerful machine carry more virtual nodes and therefore more data. DynamoDB's partitioning is the managed evolution of this idea, where partitions are the owned arcs and the service moves them for you rather than exposing the ring.
Replication and quorums, and the two flavors of read
Owning a key on one node is not enough, because that node will die, so each partition is replicated, typically three copies across three availability zones. Three copies in three failure domains is the standard durability-versus-cost point: it survives losing any one copy, or one entire zone, while still keeping a majority of two available to make progress.
With N copies, a write waits for W acknowledgements and a read consults R replicas; W plus R greater than N guarantees a read sees the latest write.
The Dynamo paper frames consistency as a quorum with three tunable numbers. N is the number of replicas, W is how many must acknowledge a write, and R is how many must be consulted for a read. The key inequality is that if W plus R is greater than N, any read set and any write set overlap on at least one replica, which guarantees a read sees the most recent write. Tuning these trades latency against consistency: W equal to 1 is fast but risky, W equal to N is durable but slow because the slowest replica gates every write, and a common balance is N equal to 3, W equal to 2, R equal to 2. Dynamo also used sloppy quorums with hinted handoff: if a target replica was unreachable, a healthy stand-in accepted the write and held it as a hint to forward later, keeping writes available during a partition at the cost of temporary inconsistency.
Production DynamoDB simplifies the developer's view by designating one replica per partition as the leader. Writes go to the leader, which appends to a write-ahead log and replicates to the others, acknowledging once a durable quorum has accepted. This gives reads two distinct flavors, one of the most common interview discriminators. An eventually consistent read, the default and cheaper option, can be served by any replica and might return data a few milliseconds stale, because the replica you hit may not have applied the latest write yet. A strongly consistent read is routed to the leader, which holds the latest committed value, so it reflects everything acknowledged before it. Strong reads cost more latency, consume roughly double the read capacity, and are slightly less available during a leader failover. Eventual is the default on purpose, because forcing every read through the leader would cost latency and availability that most workloads, a profile fetch or a product lookup, do not need.
When two writes collide
In a leaderless system, two clients can write the same key at nearly the same instant to different replicas, and those replicas end up holding divergent versions. Something has to reconcile them, and the two answers sit at opposite ends of a spectrum from correctness to simplicity.
Last-writer-wins keeps the higher timestamp and silently drops the other; vector clocks preserve causal history and surface true conflicts to the application.
The original Dynamo attached a vector clock to each version, a set of node-and-counter pairs recording the causal history of the update. When one version's clock strictly descends from another's, the newer wins automatically. When two versions are concurrent, neither descending from the other, Dynamo cannot decide and returns both to the application to merge. The canonical example is the Amazon shopping cart, where merging two concurrent carts by union is safe and losing either one silently is not. Vector clocks preserve correctness but push complexity onto the developer and can grow large.
The simpler alternative is last-writer-wins: attach a timestamp to each write and keep the higher value, discarding the other. It is trivial and keeps the API clean, but it can silently lose a write when two updates race, and it depends entirely on the quality of the clocks. Production DynamoDB keeps this problem away from developers on the single-region path by funneling writes through the per-partition leader, which orders them, so single-region writes never produce sibling versions. It resurfaces only in cross-region global tables, which resolve conflicts with last-writer-wins on a timestamp. That is precisely why multi-region writes carry a real last-writer-silently-wins caveat you have to design around.
The storage engine underneath a partition
Everything above decides which nodes hold an item. On each, the item lives in a local storage engine facing a specific workload: a high write rate, point and short-range reads, and a dataset far larger than RAM. That is what a log-structured merge tree was built for.
Instead of updating data in place on disk, which turns every write into slow random I/O, an LSM tree buffers incoming writes in an in-memory table and appends them to a durable log for crash recovery. When the in-memory table fills, it is flushed to disk as a single immutable, sorted file. Reads check the in-memory table first, then the on-disk files, using per-file indexes and Bloom filters, compact probabilistic structures that let a read skip a file that definitely does not hold the key. Over time the accumulating small files are merged into larger sorted ones by a background process called compaction, which is also where deleted and overwritten records are finally discarded. The trade is clear: writes become cheap and sequential, while a read may consult several files and compaction consumes background I/O. B-trees are the alternative, favoring read-optimized in-place updates at the cost of more expensive random writes. A store that must ingest writes at scale generally leans LSM, as Cassandra and the other Dynamo descendants do.
This explains the write path. The durability boundary is the log append, not the flush: once the mutation is in the durable log and a quorum holds it, the write is safe even though the sorted data file is not written yet.
Secondary indexes are a separate system that lags
A pure key-value store can only find an item by its primary key, too limiting for real applications, so secondary indexes let you query by other attributes. There are two kinds, they behave very differently, and confusing them is a frequent stumble.
A local secondary index shares the base table's partition key but offers an alternate sort key. Because it lives on the same partition as the items it indexes, it can be updated inside the same write and kept strongly consistent, but it inherits that partition's size limits and must be declared when the table is created. A global secondary index uses a completely different partition key, so its entries can live on entirely different partitions and nodes than the base items. Updating it synchronously would require a distributed transaction across the fleet on every write, destroying write latency. So instead the base table's ordered change stream is consumed asynchronously and applied to the index. The consequence is that it is eventually consistent: right after you write an item, a query against that index might not see it yet, usually for a very short window. It also has its own throughput budget, and here is the trap, if that budget is under-provisioned, writes to the base table itself can be throttled because the index cannot keep up.
The hot partition, the limit you cannot buy your way out of
This catches teams by surprise, and it follows from the physics of partitioning. Every partition has a hard throughput ceiling, illustratively a few thousand reads and about a thousand writes per second, because it is backed by a bounded slice of one node's CPU, memory, and disk.
A skewed partition key concentrates traffic on one partition; adaptive capacity lets it borrow idle throughput and can split it out, but cannot exceed one machine.
The subtlety that trips people up is that provisioned throughput is divided across partitions, not held in a global pool. A table provisioned for 10000 writes per second over ten partitions gives roughly 1000 per partition. If your key concentrates traffic, a status flag with three values, today's date, one celebrity account, a disproportionate share of requests lands on one partition, and it throttles at its 1000 write ceiling even though nine-tenths of the table's paid-for capacity sits idle. That is a hot partition, the direct punishment for a low-cardinality or skewed key.
There are two classes of mitigation. On your side, choose a naturally high-cardinality key, or use write sharding, appending a suffix to spread a single hot key across several synthetic keys and therefore several partitions, then reading back across all of them. On the service side, DynamoDB added adaptive capacity, which lets a hot partition temporarily borrow unused throughput from siblings and can isolate a persistently hot item by splitting the partition around it. That softens uneven load and short bursts. What it cannot do is break physics. A single key hotter than one partition can serve is still bounded by one partition, one leader, one machine, because one item cannot be split across machines. This is the ceiling sharding hits everywhere: it distributes keys, not the load on any single key, and the only real fix lives in how the application models that key.
Global tables and the price of multi-region writes
A single-region table survives node and zone failures but not the loss of an entire region, and it cannot give a user on another continent a local low-latency write. Global tables solve both by replicating across multiple regions with active-active, multi-primary writes. Every region accepts local reads and writes and asynchronously ships its changes to the others, typically within a second, though that lag widens with distance and load.
Every region accepts local writes and replicates asynchronously to the others, resolving concurrent updates by last-writer-wins on timestamp.
The benefit is real: local latency everywhere and tolerance of a whole region going dark. The cost is equally real, and it is consistency. Because writes are accepted independently in each region and replicated after the fact, two regions can update the same item concurrently, and the system resolves that conflict with last-writer-wins on a timestamp: the later write survives and the other is silently discarded. There is no cross-region strong consistency and no read-your-write guarantee across regions. An application on global tables must tolerate a brief window where regions disagree and accept that a losing concurrent write vanishes. Where that is unacceptable, a financial ledger, inventory that cannot oversell, you either pin writes for an item to a single region or add application-level reconciliation. This is the same last-writer-wins trade from the conflict section, surfacing at the one layer where DynamoDB could not engineer it away.
The trade-offs, stated plainly
Every design choice above bought something and cost something, and naming both sides separates an answer that recites features from one that understands the system. The per-partition leader traded the last increment of write availability for developer simplicity, since a leader sequences writes so single-region conflicts never arise and strong reads are trivial. Eventually consistent reads as the default traded freshness for cost, latency, and availability, the right call because most reads tolerate a few milliseconds of staleness and you opt into strong consistency only where correctness demands it. Single-table denormalized design traded query flexibility for single-round-trip speed, baking access patterns into the key schema so a new query pattern can require a new index or migration. And the bounded item size with no unrestricted cross-partition transactions is the price of predictable latency and near-linear scale. In every one of these, predictability was chosen over generality.
Where this design is the right tool, and where it is not
DynamoDB fits one class of workload extremely well: high-volume access by a known key, where you need flat latency as the data grows and your access patterns are stable enough to design keys around. Session stores, shopping carts, user profiles, feature stores, metadata services, and event records all sit comfortably here. The store is built for the one operation these share, a small addressable read or write over an item you can name in advance.
It is the wrong tool when your queries are rich and unpredictable and need joins and ad hoc analytics across the dataset, because there is no planner or join engine and paying per partition access punishes that pattern hard. It is wrong when you need strong global consistency with no chance of a silently dropped write, because the multi-region model gives up exactly that. And it is awkward when access patterns are still in flux, because the key schema is expensive to change after the fact. The correct mental model is a precise, predictable instrument for keyed access at scale, used alongside a relational database or an analytics store rather than instead of them.
Where to go from here
If this was useful, the thing worth keeping is the method, not the trivia. A system that scales cleanly usually does so because it refuses to let any single request touch more than a few nodes, and because it pushes the expensive global coordination off the fast path. DynamoDB is one of the clearest examples in the toolbox. Consistent hashing decides ownership without reshuffling everything, quorums buy availability without waiting on the slowest replica, the leader removes single-region conflicts, the LSM engine makes writes sequential, and a well-chosen partition key spreads the load so linearly that the table's size stops mattering. Understand which of those levers your own bottleneck sits on, and the fix stops being a guess.
I teach distributed systems and the rest of system design this way, from first principles with real diagrams and the trade-offs that only surface in production, as an interactive course at systemdesign.academy. The foundation lessons are free and need no signup.
Read the free lessons: https://systemdesign.academy
Top comments (0)