📺 Prefer to watch? 90-second YouTube Short · 💬 Telegram
Originally published on software-engineer-blog.com.
It is two in the morning. You run a ticket site, a big concert has just gone on sale, and checkout is timing out. Users see spinning wheels. Orders fail.
You open the cache dashboard first, because the cache sits in front of everything. You have ten machines. Memory is fine. Average CPU across the cluster: 10.5%. The cluster is 89.5% idle.
Everything looks healthy, and the site is on fire.
The reason is one of the most useful ideas in system design, and almost nobody meets it until it happens to them: sharding distributes keys, it does not distribute traffic.
How a key gets assigned to a machine
Here is the entire rule, and it is smaller than most people expect:
# cache_client.py — the whole of "which machine holds this key"
import zlib
NODES = ["cache-0", "cache-1", "cache-2", "cache-3", "cache-4",
"cache-5", "cache-6", "cache-7", "cache-8", "cache-9"]
def node_for(key: str) -> str:
h = zlib.crc32(key.encode()) # one number, from the key text alone
return NODES[h % len(NODES)] # that number picks the machine
# no lookup. no directory service. no coordinator anywhere.
node_for("evt:9001") # -> "cache-3"
Two lines. Hash the key text into a number, take that number modulo the node count, and that picks the machine. There is no directory, no coordinator, no lookup table — every client computes the same answer independently, which is exactly why this design scales.
The important property: the same key always lands on the same machine. crc32("evt:9001") is 778247973, and 778247973 % 10 is 3. It will be 3 today, tomorrow, and from every app server in your fleet.
Think of a post office with ten counters and a rule painted on the wall: the first letter of your surname decides your counter. On an ordinary morning it works beautifully. Surnames are mixed, so people spread out by themselves, and every counter has two or three people waiting.
Why that is normally fine
It really is fine, and the spread is better than you would guess. Hash a million keys across ten nodes and measure:
| Measure | Value |
|---|---|
| Keys | 1,000,000 |
| Mean per node | 100,000 |
| Busiest node | 100,360 (+0.36%) |
| Emptiest node | 99,645 (−0.355%) |
| Standard deviation | 248.8 (0.249% of mean) |
A third of one percent off the mean. Key placement is a solved problem. Every machine holds a tenth of the keys and uses a tenth of the memory.
But look at the assumption hiding inside that result. It assumes the reads are spread out the same way the keys are. They are not.
Your million keys really are spread evenly — that part is true. But nobody reads a million keys at random. People read what is linked to. One concert goes on the homepage. One post gets shared. One product goes on sale at nine in the morning. Popularity is not uniform; it never has been. And the link between the two is deterministic: one key, one hash, one machine, always.
What a hot key actually does
Same ten machines. Same 21,000 reads per second in total. The only change: 19,000 of those reads are for one key — the event page for that concert.
That key hashes to machine three. Every time.
| Node | Reads/sec | CPU |
|---|---|---|
| node 3 (hot) | 19,199 | 96.00% |
| node 0 | 200 | 1.00% |
| node 8 | 199 | 1.00% |
| every other node | ~200 | ~1.00% |
| cluster average | 2,100 | 10.50% |
Three numbers, all true at the same instant:
- 96% — the CPU load on machine three.
- 10.5% — the cluster average, which is the number your dashboard shows you by default.
- 800 reads/sec — all the headroom machine three has left before it starts refusing work.
The hot node is running at 96× the CPU of a typical node and carrying 9.1× the mean node's request rate, while the cluster sits 89.5% idle. At 19,000 reads a second, one more share of that concert link and you are over the line.
Back at the post office: a celebrity with the surname Davies announces a signing. Two hundred people queue at counter two. The other nine counters are empty, and the rule on the wall will not let anyone help.
Why adding machines does not work
This is the obvious move, and it is measurable. Same workload, more nodes:
| Nodes | Hot node CPU | Cluster average | Cluster idle |
|---|---|---|---|
| 10 | 96.00% | 10.50% | 89.50% |
| 11 | 95.91% | 9.55% | 90.45% |
| 15 | 95.66% | 7.00% | 93.00% |
| 20 | 95.50% | 5.25% | 94.75% |
You doubled the cluster. You doubled the bill. The hot machine came down by half of one percentage point — it shed 99.9 reads per second out of 19,199. Meanwhile the cluster average halved, so your dashboard looks twice as healthy as before while the actual problem is untouched.
The hot key is on exactly one node at every N. At 10 nodes it is node 3; at 15, node 3; at 20, node 13. It moves, but it is always alone.
That is the sentence worth keeping:
Sharding distributes keys. It does not distribute traffic.
Sharding genuinely balances how many keys each machine holds, how much memory each uses, and which machine owns a key after one dies. Adding machines divides all three. It never touches reads per second on a single key.
Why consistent hashing with virtual nodes does not work either
This is the sharpest distinction in the topic, and it is the answer people reach for immediately. Let us test it rather than assert it.
# ring.py — consistent hashing, the thing people reach for next
def build(nodes, V): # V = ring points per machine
ring = {}
for n in nodes:
for v in range(V): # each machine is placed V times
ring[md5(f"{n}#{v}")] = n
return sorted(ring.items())
def node_for(ring, key):
i = bisect_left(ring, (md5(key),)) # first point clockwise from the key
return ring[i % len(ring)][1] # one key -> one point -> ONE machine
Measured, at 160 virtual nodes per machine:
| Scheme | Key-count spread | Hot node CPU |
|---|---|---|
Plain crc32 % N
|
±0.36% | 96.00% |
| Ring, V=1 | +239.50% | 98.40% |
| Ring, V=160 | +17.62% | 95.97% |
Two things fall out of that table, and the second one surprises most people.
First: virtual nodes are repairing the ring's own damage. A ring with one point per machine is terrible at key balance — the busiest node holds 239% more than the mean, because the arcs between random points are wildly uneven. Going to 160 points per machine cuts the spread by 13×, down to ±17.6%. That is a real and necessary fix. But ±17.6% is still about 46× worse at key balance than plain modulo's ±0.36%. Virtual nodes do not beat modulo on balance; they claw the ring back toward it.
You use consistent hashing for a different, genuinely valuable reason: when a node joins or leaves, only 1/N of the keys move instead of nearly all of them. That is worth a lot. It is just not this problem.
Second, and the actual point: 160× more ring points moved the hot node from 98.40% to 95.97%. A 2.43 point improvement. The hot key still lands on exactly 1 of 10 physical nodes, because a key still has one hash and one hash still has one destination. Every placement scheme in this family has that property. Changing the hash function does not help either — the new function still sends one key to one place.
Fix 1: split the key
Store the same value under several names, so one logical key becomes several hashes and lands on several machines.
# hot.py — fix 1: store the same value under several names
FANOUT = 8
def read_hot(key):
s = random.randrange(FANOUT) # pick one copy at random
return cache.get(f"{key}#{s}") # 8 names -> 8 hashes -> 8 machines
def write_hot(key, value, ttl):
for s in range(FANOUT): # THE COST IS ON THIS LINE:
cache.set(f"{key}#{s}", value, ttl) # one update becomes eight writes,
# and the value is stored eight times
It works, but not as cleanly as the arithmetic suggests — because the copies collide:
| Fan-out | Distinct nodes | Busiest node CPU | Cluster average | Writes per update |
|---|---|---|---|---|
| 1 | 1 of 1 | 96.00% | 10.50% | 1 |
| 2 | 2 of 2 | 48.50% | 10.50% | 2 |
| 4 | 3 of 4 | 48.50% | 10.50% | 4 |
| 8 | 6 of 8 | 24.75% | 10.50% | 8 |
| 16 | 8 of 16 | 18.81% | 10.50% | 16 |
Look at the row for 4. Fan-out to 4 copies buys exactly zero over 2 — both sit at 48.50%. Two of the four copies hashed onto the same machine, so the busiest node still carries two replicas' worth of traffic. Copies are placed by the same blind hash as everything else; they do not politely spread out. At 16, you are paying for 16 copies and only reaching 8 distinct nodes.
And note the column that never moves: the cluster average is 10.50% at every fan-out. Splitting a key moves work around. It never removes any. You also pay in write amplification (one update becomes 8 writes), in memory (a 40 KB value becomes 320 KB across the cluster), and in consistency — eight copies are eight chances to disagree.
Fix 2: a small cache inside the app process
This one removes work instead of moving it. Put a tiny TTL cache in front of the cache tier, inside each application process.
# local.py — fix 2: a tiny cache inside the app process itself
from cachetools import TTLCache
local = TTLCache(maxsize=100, ttl=1.0) # 100 keys, one second, PER PROCESS
def get(key):
if key in local:
return local[key] # microseconds. no network at all.
v = cache.get(key) # only a miss crosses to the cache tier
local[key] = v
return v
With 40 app instances and a one-second TTL, the hot key can only be fetched once per instance per second — 40 refills per second, no matter how many reads arrive.
| Metric | Before | After |
|---|---|---|
| Hot key reads reaching the tier | 19,000 rps | 40 rps (475× less) |
| Hot node request rate | 19,199 rps | 239 rps |
| Hot node CPU | 96.00% | 1.20% |
| Reads served locally | — | 90.3% of all reads |
The cost is staleness, and you should size it deliberately: worst case a served value is 1.0s old (the TTL), average 0.5s, and 40 independent copies exist at different TTL phases — so two users can be served two different versions in the same instant. Bounded by L, and for a concert page that is completely acceptable. For an account balance it is not.
Two traps worth knowing. K=100 is worth essentially nothing over K=1 here — caching the one genuinely hot key captures the entire saving; the other 99 keys are far too cold to hit twice within a one-second window. And if the local cache refreshes ahead (re-fetching every key every TTL whether or not anyone asked), those 99 lukewarm keys cost 3,960 rps of refills to serve 0.198 rps of real traffic — 20,000× more load than having no cache at all. Fill lazily, on miss.
Fix 3: coalesce the rebuild
One specific moment needs its own fix: when the hot key expires. The value disappears, and every request arriving during the rebuild goes straight to the database.
# singleflight.py — fix 3: many misses on one key become ONE rebuild
_inflight: dict[str, Future] = {}
def get_or_load(key, loader):
with _lock:
fut = _inflight.get(key)
if fut is None: # only the FIRST caller
fut = _inflight[key] = pool.submit(loader, key)
fut.add_done_callback(lambda _: _inflight.pop(key, None))
return fut.result() # everybody else waits on that same one future
At 19,000 reads per second, the size of the stampede is just the rebuild time multiplied by the request rate:
| Rebuild takes | DB queries without coalescing | With single-flight |
|---|---|---|
| 50 ms | 950 | 1 |
| 200 ms | 3,800 | 1 |
| 800 ms | 15,200 | 1 |
At 800ms, that is 15,200 database queries for one value, 15,199 of which compute a result that is already being computed. Instantaneous database load during the window is the full 19,000 QPS — roughly 38× a comfortable single-primary budget.
The elegant part: with single-flight the answer is 1 for every rebuild duration. Coalescing does not shrink the stampede, it removes the dependency on rebuild time entirely.
What does not work
These are the things teams reach for first, and none of them touch the problem:
| Attempt | Why it fails |
|---|---|
| More machines | 96.00% → 95.50% for double the bill |
| More virtual nodes | Fixes the ring's own imbalance, not request rate |
| A bigger machine everywhere | You pay ten times over to fix one |
| More memory per node | The problem was never memory |
| A different hash function | Still sends one key to one place |
| Another cache replica | Helps you survive a failure, not a stampede |
The same shape in LLM serving
If you run inference infrastructure rather than a ticket site, you have already met this — under different names.
Prefix / KV-cache routing. Serving stacks route requests to the worker that already holds the matching prompt prefix in its KV cache, because a hit there is enormously cheaper than a recompute. That routing is a hash of the prefix. The moment one system prompt goes viral — one popular agent template, one shared RAG preamble — every request carrying it routes to the same worker. Perfect prefix-cache locality and a saturated GPU are the same decision.
Embedding and retrieval caches. A vector store sharded by document id spreads documents beautifully and spreads retrievals not at all. One trending document is one hot shard.
Mixture-of-experts routing. The router sends each token to its top-k experts. When the token distribution is skewed, some experts are chosen far more than others, and the GPU holding a popular expert becomes the step time for the whole batch. That is why production MoE training and serving carry explicit load-balancing losses and expert-capacity limits — a hot key that the system fixes by deliberately degrading placement quality.
The fixes rhyme too: replicating a hot prefix across workers is key splitting; an in-process response cache in the gateway is the local TTL cache; and de-duplicating identical in-flight completions is single-flight, exactly.
The verdict
One key, one machine, and no amount of hardware. Sharding balances storage, and you will keep believing it balances load until the night it does not.
| Situation | Reach for |
|---|---|
| Read-heavy hot key, staleness of ~1s is fine | Local in-process cache — 475× reduction, biggest win by far |
| Hot key that must stay fresh | Key splitting — accept 8× writes and 8× memory |
| Hot key with an expensive rebuild / a TTL | Single-flight — always, it is nearly free |
| Uneven key distribution, frequent membership changes | Consistent hashing + virtual nodes |
| A genuinely uniform workload that outgrew the cluster | More nodes — the one case where it is the right answer |
In practice you run the local cache and single-flight together: the local cache absorbs the steady-state read storm, and single-flight covers the moment the value expires.
And before any of that — stop charting the average. Chart the busiest machine. A maximum drawn next to a mean is the single most useful change you can make to a cache dashboard; it is the difference between seeing 10.5% and seeing 96%. Then use what you already have: Redis can sample which keys are read most often, and Memcached reports per-key statistics. You do not have to guess which key it is.
Watch the full 14-minute episode: Hot Keys in a Cache — Explained in Detail — every number in this post is executed, not estimated, and the episode walks the simulations that produced them.
Top comments (0)