DEV Community: Daniel Sam Pete Thiyagu

System Design, Plainly — Part 7: Consistent Hashing

Daniel Sam Pete Thiyagu — Sat, 14 Nov 2026 17:09:28 +0000

At 3 AM on a Thursday, an on-call engineer added one cache server to a fleet of eight — a routine scale-out, the kind that is supposed to be boring. By 3:05 AM, the database was on fire. Adding the ninth server had remapped 89% of the cache keys to new homes, so nearly every request missed the cache at once and fell through to the database. The scale-out meant to relieve the database had effectively deleted the cache.

Six weeks later, the same team had the opposite incident: one cache server crashed, and its neighbor melted. The dead server's keys all landed on the next server clockwise, which doubled its load in a second and crashed too — and the one after it. The cascade stopped only when the fleet was down to four servers holding the whole dataset.

Both incidents have the same root cause, and the same fix. The problem was never the servers. It was the map — hash(key) % N — and the fix is a circle.

This is Part 7 of System Design, Plainly: one concept per post, evergreen, beginner-friendly, with the concrete numbers and the failure modes. Today: consistent hashing — the ring that lets you add a server without moving 89% of your keys, and the virtual nodes that keep one dead server from taking its neighbor down with it.

The ELI5 version

Imagine a clock face with the numbers 0 to 100. You have eight friends standing around the clock, each at a random position. You also have 100,000 marbles, and each marble has a number. The rule is simple: a marble belongs to the first friend you meet going clockwise from the marble's number.

Now a ninth friend joins and stands somewhere on the clock. Which marbles change hands? Only the ones between the new friend and the previous friend clockwise from them — the marbles in that one small arc. Every other marble keeps its owner. That is the whole idea: adding a server only moves the keys in the new server's arc, about 1/N of the total, instead of nearly all of them.

The naive alternative is hash(key) % N: divide the hash by the number of servers and take the remainder. It looks fine until N changes — then the remainder of almost every key changes, and 89% of your data moves. Our simulation makes it exact: with 100,000 keys, growing a cluster from 8 to 9 servers remaps 89.0% of keys under modulo, and 9.8% on a consistent hash ring. Growing 10 to 11: 90.9% vs 10.1%.

There is one catch, and it is the catch that causes the second incident. With one point per server on the clock, the arcs are random — and random arcs are wildly unequal. In our simulation, one of eight servers landed on an arc holding 50.7% of all keys while another held 2.8%. The ring solves the membership problem and creates a load problem in the same stroke. The fix for the fix is virtual nodes, and it is the single most important detail in this post.

How it works: the ring, the vnodes, and the preference list

The ring. Hash the keys and the servers into the same space — say, 0 to 2^64 — and arrange them on a circle. A key belongs to the first server clockwise from it. Lookups are a binary search over the sorted server points: O(log N). Adding a server inserts its point and steals exactly the keys in its arc; removing one hands its arc to the next server clockwise. Nothing else moves. This is Karger et al., 1997 — the paper consistent hashing was invented in, built for web caches, and the idea Akamai was founded on.

Virtual nodes. Instead of placing each physical server once, place it at 100–200 points around the ring (server-3#0 … server-3#149). Now each server's responsibility is scattered into many small arcs, so the load evens out — and when a server dies, its arcs are scattered across many survivors instead of landing on one. The numbers from our simulation, 100,000 keys over 8 servers (ideal share: 12.5%):

vnodes per server	busiest server	quietest server
1	50.7%	2.8%
10	16.6%	5.5%
50	15.6%	9.2%
150	15.3%	11.4%

One vnode per server is a lottery ticket: one machine holds half your data. At 150 vnodes the busiest server is within 3 points of ideal. More vnodes means a smoother distribution, at the cost of a bigger ring map and more gossip on membership change — 100–200 per node is the usual range. Cassandra defaulted to 256 vnodes per node for years, then dropped it to 16 in 4.x once the team decided the metadata wasn't worth it.

Replication: the preference list. In a real store, a key doesn't live on one server — it lives on the first N servers clockwise (Dynamo's "preference list", 2007: usually N=3). Reads go to the first, writes to all three, and a dead server's keys are already sitting on its successors. This is where consistent hashing stops being a routing trick and becomes a storage design: the ring decides which servers hold a key, and the preference list decides how many.

Weighted nodes. Give a bigger server more vnodes — twice the capacity, twice the points — and it takes twice the keys. This is how heterogeneous fleets are handled without special cases: vnode counts are the weights.

The state of the art: five algorithms, one tradeoff

The ring is the classic, but it is not the only consistent hashing algorithm, and the differences matter. They all guarantee minimal movement — only ~1/N keys move on a membership change — but they differ in lookup cost, memory, and what kinds of cluster changes they survive:

Algorithm	Year	Lookup	Memory	The catch
Rendezvous / HRW	1996	O(N) per key	None	Scans every node per lookup
Hash ring (Karger)	1997	O(log N)	O(N × vnodes)	Needs vnodes for balance
Jump consistent hash	2014	O(log N)	None	Only sequential add/remove — can't drop an arbitrary node
Maglev	2016	O(1)	O(M) lookup table	Table rebuild on membership change
AnchorHash	2020	O(1) avg	O(A) anchor array	Fixed capacity — pre-allocate the max cluster size
MementoHash	2023	O(log N) worst	Minimal	Newest; handles random failures, no fixed capacity

The lookup-cost column is not academic. We measured all three classic approaches on this machine: at 5,000 nodes, a ring lookup costs 3.4 µs, Jump 6.0 µs, and rendezvous 7,302 µs — over 2,000× slower than the ring, because rendezvous hashes the key against every node to find the winner. Rendezvous is beautifully simple (no state at all, and every node computes the same answer independently, which is why it survives in some load balancers), but O(N) per lookup is a tax you pay on every single request.

The newer algorithms exist because each classic has a hole. Jump (Lamping & Veach, Google, 2014) needs no state and no ring — just a hash and a loop — but it can only add or remove buckets in order, so a random node failure breaks its invariant. Maglev (Google, NSDI 2016) precomputes a permutation table of size M (a large prime, typically 65,537 entries) and looks up in O(1); it powers Google's network load balancer, and the table rebuild is the price of a membership change. AnchorHash (2020) re-hashes into a fixed anchor set until it lands on a live bucket — O(1) on average, minimal disruption — but the anchor set is pre-allocated, so the cluster has a ceiling. MementoHash (2023, arXiv:2306.09783) keeps a small "memento" of removed buckets, handles random failures, needs no fixed capacity, and its authors report it beats AnchorHash and DxHash on average-case lookup while matching them worst-case. If you are choosing in 2026: the ring with vnodes is the default, Jump is the right call when you have no place to keep state and membership only grows, and Maglev is the right call when lookups must be O(1) at very large scale.

Why it breaks: the four failure modes

1. The cascade. This is the second incident from the opening. A server dies; its keys go to its clockwise successor; the successor, already hot, tips over and dies; its keys go to its successor. In our simulation with 1 vnode per server, node 3 dies holding 16.2% of keys — and node 6, which already held 50.7%, absorbs all of them and jumps to 66.8%. With 150 vnodes, node 3 holds 11.4% and its keys scatter across all seven survivors, each gaining at most 2.3 points. Virtual nodes are not an optimization. They are the difference between a node failure and a fleet failure.

2. The hot key. Consistent hashing balances keys, not load. If one key gets a million requests a second — a celebrity profile, a viral post, a config key every service reads on startup — it lands on one server, and no number of vnodes helps, because all the traffic is one key on one arc. The ring is the wrong tool for skewed popularity. The fixes live outside the ring: replicate hot keys to several servers, split a hot key into shards (user:123:0 … user:123:9), or cache it at the edge. If an interviewer asks "when does consistent hashing fail?", the hot key is the answer they want first.

3. The rebalancing storm. Adding a server moves ~1/N of keys — but 1/N of a large dataset is still a lot of data, and it all moves at once, from the servers that used to own it to the new one. A naive join can saturate the new server's disk and network before it serves a single request. Production systems throttle the handoff (Cassandra streams ranges in the background; Dynamo's preference lists let the new node catch up before it takes reads) and add servers one at a time, never in a batch. The ring guarantees which keys move, not how fast they should.

4. The hash you didn't test. Every property above assumes the hash function spreads keys uniformly. A bad hash — a weak function, or a good function fed structured keys like sequential user IDs with a common prefix — clusters keys on part of the ring and quietly reintroduces the imbalance vnodes were supposed to fix. This is also why the ring needs a real hash (SHA-256, xxHash, Murmur) and not hash() in a language where it is randomized per process: if two servers disagree about where a key lives, the key is effectively lost. Same hash, same ring, on every node, or nothing works.

One more, for the interview: consistent hashing is not consensus. The ring tells every node where a key should live, but it does not tell them when membership changed. If two servers disagree about the membership list, they route the same key to different homes. Real systems pair the ring with gossip (Cassandra), a coordinator (DynamoDB's control plane), or a strongly consistent membership view — the ring is the map, not the agreement about the map.

The interview angle

"Design a distributed cache" is the question this post is really about, and the grading rubric is almost entirely the failure modes above. The answer that passes:

Start with the problem, not the algorithm. hash(key) % N remaps ~89% of keys when N changes — say the number, don't gesture at it. The requirement is minimal movement on membership change, which is what the ring buys.
Draw the ring, then immediately add virtual nodes. The candidate who draws one point per server and stops has drawn a system where one server holds half the data. Name the vnode count (100–200) and the reason (load balance and failure spread).
Name the lookup cost and pick the algorithm by it. Ring O(log N) with a bisect; rendezvous O(N) and why that rules it out at scale; Jump when there is no state to keep; Maglev when lookups must be O(1). The 2,000× number at 5,000 nodes is the kind of concrete figure that ends the discussion.
Name the failure modes unprompted: the cascade (dead node's successor doubles), the hot key (hashing balances keys, not load), the rebalancing storm (throttle the handoff), and the membership-agreement problem (the ring is not consensus).
Close with replication: the preference list, N=3, and what happens on read when one replica is down — which is the bridge to Part 8.

Takeaways

Modulo is the bug. hash(key) % N remaps ~89% of keys when the cluster grows 8 → 9. The ring moves ~10% — the keys that must move, and no others.
Virtual nodes are not optional. One point per server leaves one machine holding 50.7% of the keys; 150 vnodes brings the busiest to 15.3% against a 12.5% ideal — and turns a node failure from a cascade into a rounding error.
Lookup cost is the algorithm choice. Rendezvous is stateless and elegant and 2,000× slower than the ring at 5,000 nodes. Jump is stateless and fast but can't handle a random failure. Maglev is O(1) but rebuilds a table on every change.
The ring balances keys, not load. A hot key still melts one server. Replicate it, split it, or cache it at the edge — the ring can't help you.
The ring is a map, not an agreement. Every node must share the same membership view, or the same key lives in two places at once. Pair the ring with gossip or a control plane.

Follow for the next part: Part 8 is **Replication* — the preference list, quorums, and what "three copies" actually guarantees when a server lies. If this series is useful, the follow button on the dev.to profile is where the next part lands first.*

Honesty note: the 89.0% / 9.8% remap figures, the vnode balance table, the cascade numbers, and the lookup benchmarks are simulations run on this machine (companion notebook) — real measurements of a toy ring, not production telemetry. The Karger 1997, Dynamo 2007, Jump 2014, Maglev 2016, AnchorHash 2020, and MementoHash 2023 (arXiv:2306.09783) facts are from the papers; MementoHash's "beats AnchorHash/DxHash on average case" is the authors' own reported result. Cassandra's 256 → 16 vnode change is from the project's own history. No Meta content, per the standing exclusion.

Companion notebook: the runnable tutorial for this post — download it here (open in Colab/Jupyter).

Prefill vs Decode: Why Splitting Them Wins

Daniel Sam Pete Thiyagu — Fri, 13 Nov 2026 17:06:20 +0000

A user pastes an 8,000-token document into a chat box and hits send. On the server, that prompt takes about 320 milliseconds of pure GPU math to process. For those 320 milliseconds, every other user already chatting on that same GPU gets nothing — their text freezes mid-sentence. One new request, thirty-two frozen streams. The average latency barely moves. The tail latency explodes.

That freeze is the whole reason LLM serving split in two. Your request is not one job. It is two jobs with opposite appetites, sharing one GPU, and the fix — running them on separate pools of GPUs and shipping the KV cache between them — is now a first-class feature of vLLM 0.33. This is GDE post #3 in the vLLM serving series: what prefill and decode actually cost, why they fight, and when splitting them pays — with the numbers, the launch commands, and the ways it breaks.

The ELI5 version

Imagine a restaurant kitchen with one stove and two chefs who need it for completely different things.

The prep chef gets a huge order all at once — chop 2,000 carrots. It is one big, parallelizable burst of work: the more burners she can use at once, the faster it goes. She is done in minutes, and then she has nothing to do until the next order arrives. That is prefill: the model reads your entire prompt in one compute-bound pass and builds the KV cache — the running memory of everything it has read.

The line chef plates one dish at a time, all evening. Each plate takes seconds, and most of the time is spent walking to the pantry and back — fetching ingredients — not cooking. He is busy for hours, but each individual step is small. That is decode: the model generates one token per step, and each step is dominated by reading the model's 16 GB of weights (plus every live user's KV cache) off memory. It is memory-bandwidth-bound, not compute-bound.

One stove, two chefs, opposite rhythms. The prep chef's big burst blocks the line chef's steady flow — that is the 320 ms freeze. The restaurant's answer is obvious: give them separate stoves. Prefill gets its own GPUs, decode gets its own GPUs, and the only thing that crosses between them is the prepped ingredients — the KV cache. That is prefill/decode disaggregation.

Two numbers track the two chefs. TTFT (time to first token) is dominated by prefill: how long until the user sees anything. TPOT (time per output token), and its per-gap cousin ITL (inter-token latency), are set by decode: how fluid the stream feels once it starts. A serving system is judged on both, at high percentiles — and on goodput, the request rate it sustains while 90% of requests still meet both SLOs.

How it works: two phases, one transfer

The arithmetic, for an 8B-class model on one H100. Prefill costs 2 × parameters FLOPs per token — about 16 microseconds per token at peak, ~40 µs at a realistic 40% MFU. A 2,000-token prompt costs ~81 ms; an 8,000-token prompt costs ~323 ms. Decode costs one full read of the weights per step: 16 GB at 3.35 TB/s is ~4.9 ms before any KV cache is read, ~7.3 ms with 32 users at 2,000 tokens of context each. Prefill time grows with prompt length; decode-step time grows with batch × context. Different knobs, different scaling — one schedule cannot serve both.

The interference, measured. Our simulation runs 400 decode steps with an 8,000-token prompt arriving at step 120. Colocated, whole-prompt prefill: the worst gap hits 330 ms — a visible freeze — while the average gap stays 7.3 ms. That is the trap: averages hide it, users feel it. Chunked prefill (Sarathi-Serve's fix: slice the prompt into 128-token chunks and interleave them with decode steps) caps the worst gap at 12.5 ms. Disaggregation removes it entirely: the decode GPU never prefills, so the worst gap is the baseline 7.3 ms. Chunked prefill is the cheap fix and the right default at small scale; disaggregation is the fix when you need the two phases tuned and scaled independently.

What crosses the wire. The KV cache for an 8B model is 128 KB per token (32 layers × K and V × 8 KV heads × 128 dimensions × 2 bytes). A 2,000-token prompt produces 256 MB of KV cache that must move from the prefill GPU to the decode GPU. The cost depends entirely on the link: 0.29 µs per token on NVLink (0.6 ms for the whole prompt), 5.2 µs on InfiniBand 200 Gb/s, 10.5 µs on 100 GbE (21 ms for 2,000 tokens), 41.9 µs on 25 GbE (84 ms). Against ~16 µs of prefill math per token, NVLink's transfer is 2% overhead and 25 GbE's is 260% — the transfer costs more than the compute it replaces. The interconnect is the decision, not the idea.

Running it in vLLM 0.33. Disaggregated prefill is experimental in 0.33 (docs at v0.33.0rc3) and the shape is simple: a prefill pool, a decode pool, and a proxy that choreographs them. Launch a producer and a consumer with the NIXL connector:

# Prefill pool (GPU 0, port 8100)
CUDA_VISIBLE_DEVICES=0 UCX_NET_DEVICES=all VLLM_NIXL_SIDE_CHANNEL_PORT=5600 \
  vllm serve Qwen/Qwen3-0.6B --port 8100 --enforce-eager \
  --kv-transfer-config '{"kv_connector":"NixlConnector","kv_role":"kv_producer","kv_load_failure_policy":"fail"}'

# Decode pool (GPU 1, port 8200)
CUDA_VISIBLE_DEVICES=1 UCX_NET_DEVICES=all VLLM_NIXL_SIDE_CHANNEL_PORT=5601 \
  vllm serve Qwen/Qwen3-0.6B --port 8200 --enforce-eager \
  --kv-transfer-config '{"kv_connector":"NixlConnector","kv_role":"kv_consumer","kv_load_failure_policy":"fail"}'

The proxy — vLLM ships a toy one for reference — sends each request to a prefill instance with max_tokens=1, stream=False, and kv_transfer_params={"do_remote_decode": True, ...}, reads the transfer parameters back from the prefill response, attaches them to the decode request, and streams the decoder's output to the client. The client sees one ordinary streamed response. NIXL (NVIDIA Inference Transfer Library, open-sourced March 2026) is the default transfer path; Mooncake and LMCache connectors plug into the same --kv-transfer-config slot — MooncakeConnector for point-to-point handoff, MooncakeStoreConnector for a pooled KV store, LMCacheConnector for prefix reuse. Three details that bite in practice: the side-channel port defaults to 5600 and must be unique per worker on a host (base + data-parallel rank); KV blocks are leased for 30 seconds by default and decode-side blocks live 480 seconds; and kv_role="kv_both" is deprecated for the NIXL connector — use producer/consumer.

Sizing the pools. The two pools want different GPU counts, and Little's law tells you the ratio: GPUs = arrival rate × service time / target utilization. At 20 requests per second, 2,000-token prompts, 200 output tokens: prefill needs ~2 GPUs, decode needs ~36 at 80% utilization — a 1:18 ratio. That ratio is traffic-dependent (short prompts, long outputs → even more decode-heavy), which is exactly why the pools must scale independently. Get the ratio wrong and the split just moves the queue from one pool to the other.

Beyond two pools. vLLM 0.32 added EPD disaggregation — a third pool for multimodal encoders — and the November 11 update extended it with metadata-only forwarding (the encoder sends a reference, not the pixels) and cross-encoder caching. Measured on Qwen3-Omni: +17.6% throughput, −13.0% mean latency. On DeepSeek V4.1 Flash over GB200: 44–47% lower P90 TTFT, 6.6–9.5% higher per-GPU throughput at concurrency 64–128. The pattern generalizes: any stage with a different resource profile earns its own pool.

The state of the art: what splitting actually buys

The idea is three years old and the measurements have converged. Splitwise (Microsoft/UW, ISCA 2024) ran prefill on H100s and decode on power-capped A100s (700 W → 350 W barely slowed decode, while prefill slowed ~2×) and reported up to 2.35× throughput at the same power and cost budget, or 1.4× throughput at 20% lower cost. DistServe framed the metric — goodput — and measured 1.6 requests/s colocated vs 10 requests/s on 2 prefill + 1 decode GPUs, 2.1× goodput per GPU, or equivalently 7.4× more requests at the same latency objective. Mooncake (Moonshot AI's KV-cache-centric design, behind Kimi) reported +525% throughput in simulation and +115% more requests absorbed on A800 hardware in production; vLLM's May 2026 engineering-blog benchmark on 12 GB200 GPUs measured 3.8× throughput, 46× lower TTFT, and 8.6× lower end-to-end latency on agentic traces.

Note what those wins have in common: they are latency and goodput wins, not raw-throughput wins. The GPUs do the same math as before, plus a copy. vLLM's docs say it in capitals: "Disaggregated prefill DOES NOT improve throughput." What it buys is separately tunable TTFT and tail ITL, and the freedom to run decode on cheaper, power-capped, or older GPUs. Anyone selling disaggregation as free throughput is selling the copy as if it were free compute.

Why it breaks

The pool ratio is the whole game. Prefill and decode scale differently with traffic mix. A burst of long prompts starves the prefill pool while decode GPUs idle; a burst of long generations does the reverse. Static ratios waste GPUs; the production answer is autoscaling per pool against its own SLO (TTFT for prefill, TPOT for decode), which is operational work the colocated setup never needed.

Slow networks invert the win. Our notebook's decision rule: the split wins when the transfer tax is smaller than the stall it removes. On 100 GbE the tax is ~26% of the stall at every prompt length — a clear win. On 25 GbE the tax exceeds the stall (41.9 µs/token of transfer vs ~40 µs/token of prefill math), so splitting makes TTFT worse. If your prefill and decode pools are in different racks on old Ethernet, chunked prefill on one pool beats disaggregation across the wire.

The failure policy matters. If a KV transfer fails, vLLM's default is fail — the request errors rather than silently degrading. The alternative, recompute, re-runs prefill on the decode instance, which is exactly the interference you split to avoid, and the docs warn it hurts tail latency. Fail loudly; alert on it; do not recompute quietly.

Stateful proxies and reasoning models. Bidirectional KV transfer needs a stateful proxy (multi-turn conversations cache transfer parameters per conversation), and there is a live sharp edge: reasoning models can produce wrong results if thinking traces are stripped between turns (vLLM issue #43094). The feature is experimental in 0.33 for a reason — pin the version, read the release notes before upgrading, and test your exact model and proxy, not the toy example.

The interview angle

"When would you disaggregate prefill and decode?" The answer they want: when TTFT and TPOT have different SLO pressure — long prompts freezing decodes, or decode needing cheaper hardware — and when the KV transfer is cheap relative to the stall (fast interconnect, long prompts). Then the follow-up: "When wouldn't you?" — short prompts on slow Ethernet, low traffic where chunked prefill already caps the tail, or anywhere you cannot autoscale the two pools independently. And the number to have ready: 128 KB of KV cache per token for an 8B model, which turns a 32K-token prompt into 4 GB that has to cross the wire before the first token appears. In the real world, that number is why Mooncake pools KV cache across DRAM and NVMe instead of moving it fresh every time, and why prefix caching (last post's subject) and disaggregation are usually deployed together: cache what repeats, split what doesn't.

Takeaways

Prefill is compute-bound and sets TTFT; decode is memory-bandwidth-bound and sets TPOT. One schedule cannot serve both — a whole 8K-token prefill freezes decoding users for ~330 ms while the average gap stays 7 ms.
Chunked prefill (128-token chunks) caps the freeze at ~12 ms and is the right default at small scale. Disaggregation removes it and lets each phase scale on its own hardware — at the cost of a KV transfer and a proxy.
The transfer is 128 KB/token for an 8B model: 0.3 µs/token on NVLink, 10.5 µs on 100 GbE, 41.9 µs on 25 GbE. On slow Ethernet the tax exceeds the stall, and splitting loses.
vLLM 0.33 ships it as experimental: --kv-transfer-config with NixlConnector, kv_producer / kv_consumer roles, a proxy that rewrites the prefill leg to max_tokens=1, and EPD extending the same idea to encoders.
It does not improve throughput. It buys separately tunable TTFT and tail ITL, and the freedom to run decode on cheaper GPUs — Splitwise's 2.35× and Mooncake's 3.8× are goodput and latency wins, not free compute.

This is post #3 of the GDE vLLM serving series — next in the lane is the scheduler: continuous batching, chunked prefill, and who waits. Follow along on dev.to/danielsamfdo for the next part.

Honesty note: Splitwise/DistServe/Mooncake figures are the papers' and vendors' own reported results (vLLM's Mooncake numbers are from its May 2026 engineering blog, measured on 12 GB200 GPUs on agentic traces — not independently replicated). The vLLM 0.33 config above was verified against the live docs (v0.33.0rc3, "Disaggregated Prefilling (experimental)") on Nov 13, 2026; the toy proxy script is linked from main, not pinned to the RC. The EPD figures are from vLLM's Nov 11, 2026 update. All simulation numbers are computed in the companion notebook for an 8B model at 40% MFU. No Meta content, per standing exclusion.

Companion notebook: the runnable tutorial for this post — download it here (open in Colab/Jupyter).

System Design, Plainly — Part 6: Rate Limiting

Daniel Sam Pete Thiyagu — Thu, 12 Nov 2026 16:59:57 +0000

At 2:17 AM on a Tuesday, an e-commerce platform's API went down. The incident report later named the root cause: not a traffic spike, not a bad deploy — a rate limiter that had allowed a burst of requests through at exactly the boundary between two time windows, overwhelming the downstream services it was supposed to protect. The limiter was doing its job by every metric it kept. It was counting correctly. It was just counting the wrong way.

That is the whole subject of this post. A rate limiter is one of the simplest components in a distributed system — allow N requests per period, reject the rest with a 429 Too Many Requests — and it is also one of the easiest to get subtly wrong, because the algorithm you pick decides what "N per period" actually means. The wrong one lets twice your limit through in two seconds. The right one costs two integers per user and is wrong 0.003% of the time. The difference between them is not theory: it is a 2 AM page.

This is Part 6 of System Design, Plainly: one concept per post, evergreen, beginner-friendly, with the concrete numbers and the failure modes. Today: rate limiting — the five algorithms that count, where each one breaks, and how to pick one by what you are actually protecting.

The ELI5 version

Imagine a nightclub with a bouncer. The club holds 100 people, and the rule is "100 people per hour." There are two ways the bouncer can enforce that, and they behave very differently at midnight.

The clipboard bouncer counts entries per clock hour. At 11:59 PM he has let in 100 people; at 12:00 AM he tears up the page and starts a fresh count. So 100 people walk in at 11:59:59, and 100 more walk in at 12:00:01 — 200 people in two seconds, in a club that holds 100. The rule was followed exactly. The club is still over capacity. That is the fixed window counter.

The stopwatch bouncer keeps a list of when everyone entered and counts how many came in during the last 60 minutes, no matter what the clock says. At 12:00:01 he looks back and sees 100 entries in the last minute — full — so the next person waits. That is the sliding window log: exact, and it costs a list entry per person.

The token bouncer holds a bucket of 100 tokens and drops one in every 36 seconds. Every entrant takes a token. If the club has been quiet, the bucket fills up, and a burst of 100 can walk in at once — then the door trickles at one person per 36 seconds until tokens refill. That is the token bucket: it allows bursts on purpose, then enforces a steady average.

The turnstile bouncer lets exactly one person through every 36 seconds, no exceptions, no queue-jumping, no burst — if you arrive early you wait in line outside. That is the leaky bucket (and its precise cousin, GCRA): perfectly smooth output, at the cost of making people wait.

Same club, same rule, four different doors. The rest of this post is about which door to build, and what each one costs you in memory, accuracy, and 2 AM pages.

How it works: five ways to count

Every rate limiter answers one question per request: has this caller used up their allowance? The algorithms differ in what state they keep per caller, and that state is the whole tradeoff.

Fixed window counter. One counter per caller per clock window — say, per calendar minute. Increment on arrival, reject past the limit, reset to zero at the next minute. State: one integer. This is the simplest limiter you can build, and it is the one Figma rejected for exactly the reason in the opening story: at a window boundary, a client can fire N requests at 11:59:59 and N more at 12:00:01, admitting 2N in about one second. Our simulation makes it exact: with a 100-per-minute limit, an adversary sending 100 requests in the last second of one minute and 100 in the first second of the next gets 200 admitted in 2 seconds — and the worst rolling-60-second count the limiter ever sees is 200, double the limit it claims to enforce.

Sliding window log. Store a timestamp for every accepted request — in Redis, a sorted set — and count how many fall inside the last 60 seconds. Exact: no boundary exists to exploit, so no rolling window ever exceeds the limit. The cost is memory proportional to the limit itself. Figma calculated that 500 requests per day per user, over 10,000 users, needs roughly 20 MB of Redis just for timestamps — fine for a login endpoint at 5 attempts per hour, ruinous for a 10,000-requests-per-hour API quota. Use the log where the limit is small and precision is non-negotiable: logins, password resets, one-time codes.

Sliding window counter. Cloudflare's pragmatic answer keeps just two counters per caller — this minute and the previous one — and estimates the rolling count by assuming last minute's requests were spread evenly: estimate = previous × (60 − seconds into current) / 60 + current. Two integers of state, and the boundary burst mostly disappears. How mostly? Cloudflare measured it across 400 million requests from 270,000 sources: 0.003% of requests were incorrectly allowed or rejected, with an average 6% gap between the true and observed rate and zero false positives. Our own simulation — Poisson traffic at 90/min with a 2× spike — lands in the same regime: mean absolute error 2.44 requests, max 8, and only 1 of 239 allow/reject decisions flipped at a 100-per-minute limit. The assumption (last minute's traffic was evenly spread) is usually wrong, and it turns out to be wrong in acceptable ways.

Token bucket. A bucket holds up to C tokens and refills at R tokens per second. Each request takes one token (or several, for an expensive call); an empty bucket means a 429. A full bucket allows a burst of C, and the long-run rate can never exceed R. State: a token count and a timestamp — two numbers. This is the default for public APIs, and for a reason: bursts are normal client behavior. AWS API Gateway works this way — its throttling rate is the refill, its burst is the capacity — and Stripe runs token buckets in Redis, setting capacity = 5 × the per-second rate, so a user with a 100 req/s limit can burst to 500. Our simulation shows the shape: a 500-request burst at t=0 is admitted in full, a second 500-request burst two seconds later gets only 205 admitted (200 tokens refilled in 2 seconds, plus 5 more during the 50 ms the burst takes to arrive), and steady 100/s traffic afterwards passes 1,000 out of 1,000. The burst is a feature, not a bug — but it is a feature you have to size: your downstream must survive C requests arriving at once, because the limiter will let them through by design.

Leaky bucket and GCRA. Flip the token bucket around: requests pour into a queue and leak out at a constant rate; when the queue is full, new requests overflow and are rejected. NGINX's limit_req works this way — rate=1r/s burst=5 allows one request per second on average with bursts of at most 5, and nodelay serves the burst at once instead of spacing it out. The output is perfectly smooth, which is exactly what you want in front of a fragile downstream (an SMS vendor with a hard per-second cap, a payment rail), and the price is latency: queued requests wait. GCRA (Generic Cell Rate Algorithm, from ATM traffic shaping) is the leaky bucket formalized as virtual scheduling, and it is the most economical limiter in production: state is a single scalar, the Theoretical Arrival Time. On arrival at time t: allow if t >= TAT − tau, then set TAT = max(t, TAT) + T, where T is the emission interval. Implemented as Redis's CL.THROTTLE command in redis-cell, it benchmarks at roughly 0.1 ms per command. Our simulation, at 100/s with a 100 ms tolerance: a 20-request burst gets 11 through (the tolerance buys exactly tau/T + 1 burst slots), 200/s traffic gets 49 of 98 admitted (the cap is 100/s, so half is rejected), and 50/s traffic gets 99 of 100 through. One timestamp, O(1) work, zero burst beyond the tolerance.

Algorithm	State per key	Bursts	Accuracy	Reach for it when…
Fixed window	1 counter	Up to 2× the limit at the boundary	Exact inside a window, loose across it	The limit is not load-bearing
Sliding window log	1 entry per request	None beyond the limit	Exact	Small limits, security endpoints
Sliding window counter	2 counters	Smoothed, small error	~0.003% wrong decisions (Cloudflare)	Edge scale, strict per-minute contracts
Token bucket	2 numbers	Up to capacity, then the refill rate	Good	Public APIs — the default
Leaky bucket / GCRA	1 queue / 1 timestamp	Smoothed, none beyond tolerance	Good / exact spacing	A fragile downstream needs a steady rate

The distributed problem: one limiter, many servers

Everything above assumes one limiter. Real systems have dozens of servers behind a load balancer, and a per-server limiter is not a limiter at all — it is N limiters, each enforcing the full limit. Ten servers with a 100/min limit each is a 1,000/min limit in practice. The fix is a shared store, and the shared store has one hard requirement: atomicity.

The read-modify-write cycle — read the count, add one, compare, write back — is a race condition with a hole exactly one request wide, and at 10,000 requests per second that hole is a highway. The standard answer is a Lua script inside Redis: Redis executes scripts single-threaded, so the read, increment, compare, and expiry collapse into one atomic round trip. Stripe's production token-bucket Lua is about 26 lines. That is the whole distributed limiter at moderate scale: one Redis, one Lua script, one round trip per request.

At global scale the answer changes shape. A single Redis becomes a latency and availability problem of its own, so the edge takes over: Cloudflare's sliding window counter runs at the edge, close to the client, because two counters per key replicate cheaply and a 0.003% error is cheaper than a cross-region round trip on every request. The pattern that works in production is three tiers: edge (IP-based, high limits, catches DDoS and scrapers), API gateway (API-key-based, per-plan limits — free 60/min, pro 600/min), and per-endpoint (tight limits on the expensive operations: search 10/min, export 5/min). Each tier answers a different question, and no tier trusts the one above it.

Two more distributed facts worth knowing before the interview. First, clocks skew: a token bucket that refills from wall-clock time drifts when servers disagree about the time by tens of milliseconds, and a sliding window counter keyed to calendar minutes has the same problem — keep your windows short and your tolerance generous, or use a monotonic source. Second, a global limit is not a per-key limit: capping each user at 100/min does nothing when one tenant spawns 10,000 API keys. Per-key limits need a per-tenant limit above them, or the attacker just buys more keys.

Why it breaks: the four failure modes

1. The retry storm. A limiter rejects a request with a 429, and the client — helpfully, automatically — retries. Immediately. Ten thousand clients retrying at once turns a 100/s overload into a 1,000/s overload, and the limiter that was protecting the backend is now amplifying the load on it. The fix is on both sides of the wire: the server must send a Retry-After header (and the IETF is standardizing richer RateLimit and RateLimit-Policy headers — draft version 11, May 2026, still not an RFC), and the client must back off exponentially with jitter, never retry immediately, and cap total attempts. A 429 without Retry-After is a limiter that only did half its job. Stripe's advice, worth stealing verbatim: dark-launch new limits in log-only mode first, so you learn which clients will retry before the limit starts rejecting them.

2. Limiting the wrong thing. A per-second request count protects a fast endpoint and does nothing for a slow one: 100 requests per second to an endpoint that takes 10 ms each is 1 concurrent request; to an endpoint that takes 10 seconds each, it is 1,000 concurrent requests and a dead database. Stripe runs a second limiter for exactly this — a concurrency limiter capping in-flight requests per user — and when even that is not enough, it switches tools entirely: a load shedder looks at the whole system and drops low-priority work when resources saturate, while a rate limiter shapes the rate of one well-behaved actor. Different questions, different tools, same request path. Netflix's adaptive concurrency limits go one step further and learn the safe rate instead of being configured with one, Vegas-style: probe upward while latency stays flat, back off the moment it rises.

3. The limiter becomes the outage. A Redis-backed limiter that goes down takes one of two paths, and both are bad in different ways: fail-open (allow everything, lose protection exactly when traffic spikes) or fail-closed (reject everything, turn a Redis outage into a full outage). There is no universally right answer — fail-open for a public API whose worst case is extra load, fail-closed for a login endpoint whose worst case is a credential-stuffing attack — but it has to be a deliberate choice, made before the incident, not discovered during it. The same logic applies to the limiter's own latency: a 5 ms Lua script on every request is a 5 ms tax on your p99, paid by every user including the ones nowhere near their limit.

4. The limit nobody can find. A limit that is not communicated is a limit that gets discovered by accident, in production, by your best customer, at their busiest hour. Publish the limits in your docs, return the remaining quota in headers on every response, and make the 429 body say what the limit is, when it resets, and which tier the caller is on. GitHub does this well — 5,000 requests per hour for authenticated callers, with the reset time in the response — and the reason it works is that a client who can see the budget can manage it.

The interview angle

"Design a rate limiter" is one of the most common system-design questions, and the grading rubric is almost entirely about the failure modes above. The answer that passes:

Start with the requirement, not the algorithm. Who is being limited — IP, API key, user, tenant? What is the limit protecting — the database, a paid downstream, abuse? A login endpoint and a search API want different limiters, and saying so is the point.
Pick token bucket as the default and justify it in one sentence: two numbers of state, bursts allowed, steady average enforced. Then name the exception: sliding window counter when the contract is a strict per-minute number, sliding window log when the limit is small and must be exact, GCRA when the downstream needs a smooth rate.
Draw the distributed version: shared Redis, Lua script for atomicity, TTL on keys so old counters clean themselves up. Then say what breaks at global scale (cross-region latency) and what replaces it (edge enforcement, approximate counters).
Name the failure modes unprompted: the boundary burst (why not fixed window), the retry storm (429 + Retry-After + exponential backoff with jitter), the per-server trap (N servers × limit = N × limit), and fail-open vs fail-closed.
Close with the tiers: edge, gateway, per-endpoint — and the distinction between rate limiting one actor and load shedding the whole system.

The candidate who says "token bucket in Redis with a Lua script" and stops has given the Wikipedia answer. The candidate who then says "but the burst size is a promise to the downstream, the retries are the real outage risk, and here is what happens when Redis dies" has given the production answer.

Takeaways

A rate limiter is a promise about the worst case, not the average. A fixed window counter promises 100/min and delivers 200 in two seconds. Check what your algorithm admits at the boundary, not in the middle of the window.
Token bucket is the default for a reason: two numbers of state, bursts up to capacity, steady rate after. Stripe sizes capacity at 5× the rate; AWS API Gateway exposes the same two knobs.
Two counters buy you almost-exact for almost-free: Cloudflare's sliding window counter is wrong on 0.003% of 400 million requests. When the contract is a strict per-minute number, that is the cheapest correct answer.
GCRA is the most state-efficient limiter in production: one timestamp, ~0.1 ms per decision in Redis, zero burst beyond the tolerance you choose.
The limiter is not the system. Per-key limits need per-tenant limits above them, concurrency limits beside them, and a load shedder for when the whole fleet is hot. And every 429 needs a Retry-After, or your clients will retry their way into the outage you were trying to prevent.

Follow for the next part: Part 7 is **Consistent Hashing* — the ring that lets you add a server without moving 80% of your keys. If this series is useful, the follow button on the dev.to profile is where the next part lands first.*

Honesty note: Cloudflare's 0.003% figure, Stripe's 5× capacity rule, Figma's 20 MB estimate, and the redis-cell 0.1 ms benchmark are the vendors' own published measurements, not independently replicated here; the boundary-burst, token-bucket, sliding-counter, and GCRA numbers are simulations in the companion notebook, run on this machine. The IETF RateLimit headers are draft version 11 (May 2026), not an RFC. No Meta content, per the standing exclusion.

Companion notebook: the runnable tutorial for this post — download it here (open in Colab/Jupyter).

System Design, Plainly — Part 5: CAP and PACELC

Daniel Sam Pete Thiyagu — Wed, 11 Nov 2026 17:00:44 +0000

Every system-design interview asks about CAP, and almost every answer is wrong. The candidate draws the triangle — Consistency, Availability, Partition Tolerance, pick two — and the interviewer nods, and both of them walk away having agreed on a statement that is, at best, a third of the truth. The triangle is not wrong the way a wrong number is wrong. It is wrong the way a map of a city with only the highways is wrong: it shows the rare disaster and hides the road you drive every day.

Here is the correction in one sentence: CAP only describes what your system does when the network is broken. PACELC describes what it does the other 99.9% of the time — and that is the tradeoff you actually live with. A partition is the crisis. Latency versus consistency is the rent. This post is about both, with the numbers: quorum math that makes stale reads exactly zero, a partition simulation that shows what availability really costs, and the latency ladder that prices every consistency level you could buy.

This is Part 5 of System Design, Plainly: one concept per post, evergreen, beginner-friendly, with the concrete numbers and the failure modes. Today: CAP and PACELC — what the theorem actually says, why "pick two of three" misleads you, and how to choose per operation instead of per system.

The ELI5 version

Imagine a bank with two branches and one account. You deposit $100 at branch A. Your friend walks into branch B five minutes later and asks for the balance. What should branch B say?

If the phone line between the branches is working, branch B calls branch A, confirms the $100, and answers correctly. That call costs time — the friend waits. If the phone line is cut — that is a network partition — branch B has two options, and there is no third one. It can refuse to answer ("I can't verify your balance right now, come back later") — that is choosing consistency over availability. Or it can answer from its own books ("your balance is $0, as far as I know") — that is choosing availability over consistency. It cannot do both, because the information it needs is on the other side of a cut wire.

Now the part the triangle hides: even when the phone line works fine, branch B still has a choice. It can call branch A on every balance question — always correct, always slow — or it can answer from its own books immediately and sync up with branch A later — always fast, sometimes stale. That everyday choice, made on a healthy network, is what PACELC adds. CAP is the crisis decision. PACELC is the daily one. And the daily one is the one your users feel on every single request.

How it works: CAP, precisely

The CAP theorem was Eric Brewer's conjecture in a 2000 keynote, proved by Seth Gilbert and Nancy Lynch in 2002. It says a distributed data store cannot guarantee all three of these at once during a network partition:

Consistency (C) — every read receives the most recent write, or an error. In CAP's sense this means linearizability: all nodes agree on one version of the truth, as if there were a single copy. This is not the "C" in ACID, which is about invariants inside one transaction.
Availability (A) — every request to a non-failing node gets a non-error response. Note what it does not promise: the response may be stale. Availability is about answering, not about being right.
Partition tolerance (P) — the system keeps operating when messages between nodes are dropped or delayed. A partition is not only a severed cable: a switch failure, a misconfigured firewall, a garbage-collection pause long enough to look like a dead node — all of these are partitions from the system's point of view.

The first correction: P is not a choice. Networks partition. Fiber gets cut, routers reboot, availability zones lose connectivity. Any system that runs on more than one machine must tolerate partitions, so the real CAP decision is always CP versus AP — what do you do while the partition lasts?

CP — refuse requests on the minority side rather than risk a wrong answer. A payment ledger, a seat reservation, a distributed lock: the system would rather say "no" than say something false. Our simulation: during a 60-second partition, the minority side rejects 55% of writes — every write that happened to land on the wrong side of the cut. Zero divergence, zero conflicts, but more than half the writes fail in front of the user.
AP — keep accepting writes on both sides and reconcile when the partition heals. A shopping cart, a like counter, a social feed: the system would rather take the write than lose it. Our simulation on the same 100 writes: both sides accept everything, but 13 of the 20 hot keys were written on both sides — 65% of keys now have two conflicting versions, and something has to decide which one wins.

Notice what decides the cost: not the number of writes, but the number of keys. The same 100 writes spread across 1,000 keys produce exactly 1 conflict (0.1%). The AP tax is a hot-key tax. A cart service with millions of distinct carts pays almost nothing for availability; a seat map with 20 contested seats pays for it on nearly every write. That is the first hint that the right answer is per operation, not per system — and it is the thread the rest of this post pulls.

Quorum math: the dial between stale and slow

Replication gives you a dial, and the dial is arithmetic. With N replicas, a write waits for W acknowledgements and a read waits for R responses. If W + R > N, the read set and the write set must overlap on at least one replica that has the latest write — so a read can never miss it. If W + R ≤ N, the sets can miss each other entirely, and the read can be stale.

Our Monte Carlo simulation (N = 3, random read/write sets, 20,000 trials per level) makes the cliff exact:

W=1, R=1 (write to one replica, read from one): 66.4% of reads are stale. This is the default for a system that acks locally and reads locally — fast, and wrong two times out of three.
W=1, R=2 or W=2, R=1: 33% stale. Reading from two replicas halves the miss rate, but a third of reads still see the old value.
W=2, R=2, W=3, R=1, W=1, R=3: 0.0% stale in all 20,000 trials. The moment W + R exceeds N, staleness drops off a cliff to exactly zero — not "rare", zero, because overlap is a mathematical guarantee, not a probability.

N reads zero stale — overlap is a guarantee, not a probability. Simulation in the companion notebook." width="800" height="409">

This is why Cassandra lets you set the consistency level per query: ONE for the like counter, QUORUM for the balance, ALL for the once-a-year reconciliation job. DynamoDB does the same with its strongly-consistent-read flag — eventual by default, strong when you ask and pay for it. Our simulation of that toggle: against a replica lagging on 5% of reads, eventual reads come back stale 5.0% of the time; strong reads, 0.0%. The price of strong is one quorum round trip per read. Which brings us to the bill.

PACELC: the tradeoff you pay every day

Daniel Abadi proposed PACELC in a 2010 blog post and formalized it in a 2012 paper, and its whole point is that CAP stops talking exactly where your system spends its life. The statement, in full:

If there is a Partition, choose Availability or Consistency. Else — normal operation, no partition — choose Latency or Consistency.

Read it as two decisions instead of one. The first half is CAP. The second half is new, and it is the one that shows up in your latency dashboard: to guarantee consistency even on a healthy network, replicas must coordinate before answering, and coordination is round trips. A write that acks locally costs about 1 ms. A write that waits for a same-AZ quorum costs about 2.5 ms. Cross-AZ, about 5 ms. A write that waits for a quorum across the continental US costs about 70 ms — 14× the same-AZ quorum — and a globally synchronous commit across three regions costs about 105 ms, 105× a local ack. Every consistency level you buy is a round trip you pay on every single write, healthy network or not.

That ladder is why the PA/EL systems and the PC/EC systems look so different in production, and why the classification below is more useful than any CAP label. A system's PACELC code is two letters for the partition branch and two for the normal branch:

PA/EL — DynamoDB (eventual mode), Cassandra (default), Riak. Available during a partition, fast normally. The cost is eventual consistency: reads can lag, and conflicts get resolved after the fact.
PC/EC — Spanner, CockroachDB, etcd, ZooKeeper, MongoDB (majority writes). Consistent during a partition, consistent normally — and slow in both branches, because every write waits for a quorum or a global commit. Spanner buys its global consistency with TrueTime, synchronized atomic clocks and GPS receivers, and still pays a commit wait measured in single-digit milliseconds per write (Google-reported); that is the tax for a ledger that is correct on every continent at once.
PC/EL — MongoDB (default reads from primary), Yahoo's PNUTS. Consistent when partitioned, but reads normally go to the local primary without waiting — the odd quadrant, and the proof that the two branches really are independent decisions.

The state of the art: tunable, per operation, and cheaper than it used to be

The 2026 picture is not "pick a quadrant and live in it." It is tunable consistency, set per operation, on systems that keep shrinking the latency tax:

1. Tunable reads and writes are the default now. DynamoDB offers strongly consistent reads as a per-request flag and replicates global tables typically in under a second (AWS-reported). Cassandra's per-query consistency levels — ONE, QUORUM, ALL, LOCAL_QUORUM — are the textbook example. The decision moved from the architecture document into the query: the same cluster serves the like counter at ONE and the balance at QUORUM, and the latency budget follows the operation, not the system.

2. PC/EC got cheaper. The old story was that global strong consistency costs a cross-region round trip on every write, full stop. Aurora DSQL (generally available December 2024) attacks exactly that: a disaggregated, multi-region PostgreSQL-compatible database that promises strong consistency with single-digit-millisecond reads, by separating the log from the compute and committing through a distributed transaction log instead of two-phase commit across regions (AWS-reported — independent benchmarks are still thin, treat the headline as a vendor claim). CockroachDB's 25.x releases keep pushing the same direction with follower reads: reads from the nearest replica for stale-tolerant queries, quorum reads only where freshness matters. The PC/EC quadrant is no longer a 100-ms sentence.

3. The consistency models got names, and the names matter. Linearizability (CAP's C) is the strongest: every operation looks instantaneous. Below it sit sequential consistency, causal consistency (your reply is never visible before the message it replies to), read-your-writes, monotonic reads, and eventual consistency. Most real systems sell you one of the middle rungs per operation: DynamoDB's read-your-writes on the same session, Spanner's external consistency, CockroachDB's serializable by default with follower reads as the escape hatch. In an interview, naming the rung you need — not just "strong" or "eventual" — is what separates a good answer from a recited one.

4. The edge cases moved to the center. The systems that win in 2026 are the ones that handle the cases CAP pretends do not exist: partial partitions (one node slow, not dead — a GC pause looks exactly like a partition to a timeout), clock skew (Spanner's TrueTime exists because wall clocks lie by milliseconds, and a commit that trusts a lying clock can order two writes backwards), and the read path (most systems are write-consistent and read-eventual by default, and the stale read is the bug users actually report).

Why it breaks

Every CAP/PACELC failure is one of these six:

The "CA system" that is really one node. A single-node database is consistent and available — and has no partition to tolerate, because there is nothing to partition. The moment you add a second node for failover, the triangle applies, and the marketing label does not survive the first network blip. Martin Kleppmann's 2015 critique, "Please stop calling databases CP or AP", is still the sharpest version of this: the labels describe one behavior under one failure, not the system.
Availability is not binary. CAP's A means "every non-failing node answers every request." Real systems degrade: a CP system that refuses writes on the minority side is still serving reads; an AP system that accepts writes is still dropping the ones that conflict. The useful question is never "is it available" but "what fraction of which operations still succeed, and what do the failures look like to the user?"
The stale read nobody tested. The write path is quorum-consistent, the read path is eventual by default, and the first user to refresh after an update sees the old value. This is the most common consistency bug in production, and it is a configuration bug, not a theorem: the read consistency level was never set, because nobody owned the read path.
The slow node that looks dead. A 30-second GC pause is indistinguishable from a partition to a 5-second timeout. The system declares the node dead, re-elects a leader, and the "dead" node comes back and tries to lead again — a flapping partition that no theorem covers, because CAP assumes partitions are clean cuts. Fix: failure detectors with suspicion levels, not binary timeouts, and fencing tokens so a resurrected leader cannot write.
The clock that lies. Two writes ordered by wall-clock time on two machines whose clocks differ by 40 ms can be applied in the wrong order — and no quorum fixes it, because both writes reached a quorum. Spanner pays for atomic clocks for exactly this reason; everyone else pays with logical clocks (Lamport timestamps, hybrid logical clocks) or accepts the risk. If your ordering depends on time.now(), you have a consistency bug with a timer on it.
The conflict resolution nobody designed. The AP system accepts writes on both sides of a partition, the partition heals, and 65% of the hot keys have two versions — and the resolution rule is "last write wins by timestamp", which silently discards one side's writes. Last-write-wins is a conflict deletion policy, not a resolution policy. CRDTs (for counters, sets, registers that merge by construction) or application-level merges (the shopping cart union both sides) are the honest answers; "last write wins" is the one you reach for when nobody thought about it.

The interview angle

"Design a payment system" and "design a like counter" are the same question with opposite answers, and the interviewer is checking whether you notice. The expected tour: define the operation's cost of a wrong answer first (a double charge is a lawsuit; a stale like count is invisible), then choose per operation — CP/EC for the charge and the seat reservation, AP/EL for the cart, the feed, and the counter — then name the mechanism (quorum levels, W + R > N, TrueTime or Raft for the consistent side, CRDTs or last-write-wins with eyes open for the available side).

The follow-ups that separate good answers from great ones: what consistency does a read need? (read-your-writes for the user who just posted, eventual for everyone else); how do you make a stale read impossible without paying the quorum on every read? (session stickiness plus read-your-writes, or a version token the client carries); what happens when a node is slow, not dead? (suspicion-based failure detection, fencing); when is strong consistency free? (single-region, same-AZ quorum — 2.5 ms — is the cheapest consistency you will ever buy, and the reason most systems are strongly consistent inside one region and eventual across regions); and what does your system do when the partition heals? (the merge policy is the design; the partition is just the trigger).

In the real world, the decision is usually not which database to buy — it is which consistency level each operation gets. The same cluster serves both answers. The theorem tells you the tradeoff exists; the per-operation choice is the engineering.

Takeaways

CAP only covers the crisis: during a partition you choose consistency or availability, and P is not optional — the real choice is always CP vs AP.
PACELC covers the other 99.9%: even on a healthy network, replication forces latency versus consistency — a global synchronous commit costs 105× a local ack, and you pay it on every write.
Quorum math is a cliff, not a slope: with N = 3, W=1/R=1 leaves 66.4% of reads stale and W=1/R=2 leaves 33.4%, but every level with W + R > N reads exactly 0% stale in 20,000 trials.
The AP tax is a hot-key tax: 100 writes during a partition conflict on 65% of 20 hot keys but only 0.1% of 1,000 keys — while CP rejects 55% of writes and keeps a single truth.
Classify by PACELC, not CAP: DynamoDB and Cassandra are PA/EL, Spanner and CockroachDB are PC/EC, MongoDB's majority writes are PC/EC — and the code is set per operation, not per system.
The most common consistency bug is the read path: a quorum-consistent write path with an eventual read path shows users the old value, because nobody set the read consistency level.
Try it yourself. The companion notebook replays all four simulations: the quorum cliff (66.4% → 0.0% stale), the partition divergence (65% → 0.1% conflicts as the keyspace widens), the latency ladder (1 ms → 105 ms), and the tunable-consistency toggle (5.0% stale eventual vs 0.0% strong).

This is Part 5 of System Design, Plainly. Part 6 — Rate Limiting — is next. Follow me on dev.to (@danielsamfdo) so you don't miss it.

Sources: Eric Brewer, "Towards Robust Distributed Systems" (PODC 2000 keynote); Seth Gilbert and Nancy Lynch, "Brewer's Conjecture and the Feasibility of Consistent, Available, Partition-Tolerant Web Services" (ACM SIGACT News, 2002); Daniel Abadi, "Consistency Tradeoffs in Modern Distributed Database System Design" (IEEE Computer, 2012) and the 2010 blog post that introduced PACELC; Martin Kleppmann, "Please stop calling databases CP or AP" (2015); Google, "Spanner: Google's Globally-Distributed Database" (OSDI 2012 — TrueTime commit wait, single-digit milliseconds, Google-reported); Amazon DynamoDB docs (strongly consistent reads per request; global-table replication typically under a second — AWS-reported); Amazon Aurora DSQL (GA December 2024, multi-region strong consistency, single-digit-millisecond reads — AWS-reported, independent benchmarks thin); CockroachDB docs (Raft quorum, follower reads); Cassandra docs (per-query consistency levels ONE/QUORUM/ALL). All simulation numbers are from the companion notebook and the diagrams in this post; latency-ladder figures are a stated model (1 RTT to quorum + 0.5 ms apply), not measurements. No content about excluded vendors appears in this post.

Companion notebook: the runnable tutorial for this post — download it here (open in Colab/Jupyter).

PagedAttention and the KV Cache, Plainly: Why Memory, Not FLOPs, Caps Your Concurrency

Daniel Sam Pete Thiyagu — Tue, 10 Nov 2026 17:03:15 +0000

One 128K-token conversation with a 70B model needs 40 GiB of GPU memory — not for the weights, for the memory of the conversation itself. An 8B model on the same GPU fits 56 such conversations at once; the 70B model in FP8 fits zero. Same GPU, same 8K-token chats, and the difference is not speed. It is memory. Every LLM serving decision you will ever make is downstream of this one fact: the KV cache, not the FLOPs, caps your concurrency.

The fix has a name you have heard a hundred times and a mechanism most people never actually work through: PagedAttention. It treats the KV cache like an operating system treats virtual memory — fixed-size pages, allocated on demand, freed the moment a request ends. The 2023 paper behind it reports 2–4x higher throughput at the same latency, with memory waste falling from 60–80% to under 4%.

And the ground just shifted under it. On November 5, 2026 — five days ago — vLLM merged PR #63231, changing the default block size from 16 tokens to 64. The number every tutorial teaches is now the old number. This post does the arithmetic the new default demands: what a token costs, what fragmentation actually wastes, what the 16→64 change costs you at scale, and how many conversations your GPU can really hold.

This is post #2 of a 25-post series on the Google AI stack — vLLM serving first, then JAX/Keras/Gemma, Google's Agent Development Kit, and Vertex AI — written for the Google Developer Expert AI track. One concept per post, concrete numbers, a runnable notebook. Yesterday: what vLLM 0.33 changed. Today: the memory system underneath all of it.

The ELI5 version

A language model reads your prompt one token at a time, and to generate the next token it needs to look back at everything it has already seen. Recomputing that from scratch every step would be absurdly slow, so the model takes notes: for every token, in every layer, it writes down two small vectors — a key and a value. Together they are the KV cache. The next token is produced by comparing the new token against all those notes.

Three things follow, and they explain everything in this post:

The notes grow with the conversation. Every token adds notes in every layer. A 70B model writes 320 KB of notes per token, so a 128,000-token conversation accumulates 40 GiB of notes. The model itself is 140 GiB in bf16 — the notes for one long conversation are nearly a third of the model.

The notes are the memory, not the model. Weights are fixed; the KV cache grows with your users. Double the context length or double the concurrent users and the cache doubles. The GPU's FLOPs sit idle while requests wait for a free page of cache memory. That is why serving is a memory problem wearing a compute costume.

The notes used to be stored wastefully. The naive approach reserves the maximum possible notes for every request up front — a full 8,192-token slab for a request that might end at 500 tokens — and in one contiguous block, so variable-length requests leave unusable gaps. PagedAttention borrows the operating system's oldest trick: split memory into fixed-size pages (blocks of 16 tokens, now 64), hand them out as tokens arrive, take them back when the request ends. No reservation, no gaps.

How it works: the four numbers that decide everything

1. What one token costs

The formula is one line:

bytes per token = 2 (K and V) × layers × KV heads × head dimension × bytes per value

Two design choices dominate it. Grouped-query attention (GQA) shares one key/value pair across a group of query heads — Llama 3.1 70B has 64 query heads but only 8 KV heads, so it stores 8x less than a naive design. And DeepSeek's Multi-head Latent Attention (MLA) goes further, compressing each layer's keys and values into a single 576-dimensional latent vector: 68.6 KB per token, against 320 KB for a 70B GQA model.

Model	Layers	KV heads	KB per token (bf16)	One 128K conversation
Llama 3.1 8B	32	8	128	16.0 GiB
Qwen3 32B	64	8	256	32.0 GiB
Llama 3.1 70B	80	8	320	40.0 GiB
DeepSeek V3 (MLA)	61	1 latent	68.6	8.6 GiB

Read the last column twice. A single 128K-token conversation with a 70B model needs 40 GiB of cache — more than an entire 8B model's weights. And MLA is not a small optimization: it is a 4.7x reduction in the memory wall, which is why DeepSeek's architecture choices keep showing up in every serving discussion. The notebook recomputes every number in this table from the public model configs.

2. Fragmentation: the 60–80% tax PagedAttention removes

Before PagedAttention, serving engines allocated the KV cache the way a bad landlord rents apartments: one contiguous slab per request, sized to the maximum length, held for the request's whole life. Two kinds of waste followed. Internal fragmentation: a request that ends at 500 tokens holds an 8,192-token reservation. External fragmentation: freed slabs of different sizes leave gaps no new request quite fits. The SOSP 2023 paper measured 60–80% of KV memory wasted this way.

PagedAttention removes both by decoupling the logical cache from its physical layout. Each request gets a block table — its logical blocks mapped to physical blocks anywhere in the pool, exactly like a process's page table. A request that uses 100 tokens holds 7 blocks of 16 (the last one half-full); the rest of the pool stays available. When the request ends, its blocks return to the free list immediately. The paper's result: waste under 4%, and 2–4x the throughput at the same latency on the same hardware.

The simulation in the notebook — 200 requests, lengths uniform between 500 and 8,000 tokens, max_model_len 8,192 — lands in the same band: naive allocation wastes 45.2% of the pool on average (the exact figure depends on the length mix; the paper's 60–80% is the worst-case workload), paging wastes 0.26%, reserving 1.82x less memory for the same 200 requests.

3. The block size just changed: 16 → 64 tokens

Here is the part that makes this post time-sensitive. vLLM's default block size was 16 tokens since the PagedAttention paper — "large enough to efficiently utilize the GPU and small enough to avoid significant internal fragmentation," in the authors' words. PR #63231, merged November 5, 2026, moves the default to 64 to cut per-block bookkeeping: fewer blocks means a smaller block table, fewer kernel launches per attention step, less metadata to manage.

The cost is the ragged tail. A request's last block is almost never full, so every request wastes, on average, half a block: 7.5 tokens at block size 16, 31.5 at 64. The extra waste is 24 tokens per request — and at concurrency 1,024 that is 24,576 tokens of KV capacity gone, which is the maintainer's own estimate in the PR discussion, derived in the notebook. At 320 KB per token on a 70B model, that is about 7.5 GiB of HBM.

Three operational consequences land with the change:

Prefix-cache hits now fall on 64-token boundaries. A shared prompt only produces reusable blocks at multiples of 64 tokens. A 40-token system prompt no longer yields a shareable block at all.
KV connectors must use multiples of 64. A block_size that isn't a multiple of 64 now fails validation at startup — configs that assumed 16 will error, not silently degrade.
ROCm keeps 16. AMD's kernel stays on the old default, so the same vLLM version now has different defaults per platform. If you run mixed fleets, pin the block size explicitly rather than relying on the default.

One honest caveat: the PR lists the 16-against-64 serving benchmark as still open. The 24K-token figure is a maintainer estimate, not a measured throughput delta. The direction of the trade is certain (less bookkeeping, more tail waste); the net throughput effect at your concurrency is not yet published. Measure it on your fleet before deciding the new default is a win for you.

4. Concurrency: the KV pool is what's left after the weights

vLLM pre-allocates a fraction of the GPU — gpu_memory_utilization, default 0.9 (0.92 in the latest docs, 1.0 with the new extensible KV cache, which sizes the pool from memory actually free after CUDA-graph capture instead of from a profiling estimate). Weights load first. Everything left is the KV block pool. Divide by the per-request cost at your context length and you have your concurrency ceiling — before a single request arrives.

Setup	KV pool	Per 8K request	Concurrent 8K requests
H100 80GB + Llama 3.1 8B (bf16)	56.0 GiB	1.00 GiB	56
H100 80GB + Llama 3.1 70B (FP8)	2.0 GiB	2.50 GiB	0
H200 141GB + Llama 3.1 8B (bf16)	110.9 GiB	1.00 GiB	110
H200 141GB + Llama 3.1 70B (FP8)	56.9 GiB	2.50 GiB	22
B200 192GB + Llama 3.1 70B (bf16)	32.8 GiB	2.50 GiB	13

The zero is the point. A 70B model in FP8 weighs 70 GB; on an 80 GB H100, 0.9 × 80 − 70 leaves 2 GB of pool — not enough for even one 8K-token conversation. It is not a bug, and no amount of batching fixes it: the model cannot serve a single 8K request on that card at that utilization. The remedies are a bigger GPU, a smaller model, a shorter context, or tensor parallelism across two cards. Note the B200 row too: even a 192 GB card holds only 13 concurrent 8K requests of a bf16 70B — the weights eat 140 GB before the pool even starts. Quantized weights are a concurrency decision, not just a cost decision: FP8 weights buy you 22 concurrent requests on the H200 where bf16 buys you a handful.

The state of the art: what else shares the cache

Prefix caching is PagedAttention's free second act. Because blocks are content-addressed, identical prefixes hash to the same physical blocks: a shared system prompt is stored once and referenced by every request. enable_prefix_caching defaults to True in current vLLM. The saving scales with (N−1) × prompt tokens — a 2,000-token system prompt shared across 100 concurrent requests on a 70B model saves 60.4 GiB of KV. For RAG systems, where every query carries the same retrieved documents, this is often the single biggest memory win available, and it costs nothing but keeping your prompt template byte-identical. One differing token at the start of the prompt kills the hit for the whole prefix — which is why prompt-template discipline is a serving concern, not a style concern.

The rival design is SGLang's RadixAttention. Instead of flat blocks, it organizes the cache as a radix tree indexed by token sequence, so prefixes share at token granularity and eviction follows an LRU policy over the tree. It is the better design for long shared prefixes and multi-turn chat; PagedAttention's flat blocks are simpler and remain the reference implementation every other engine copied. The choice matters less than knowing which one your engine uses — they fail differently when the pool runs dry.

When the pool runs dry anyway, vLLM preempts. The scheduler pauses the lowest-priority request and either drops its KV and recomputes it later (the default) or offloads its blocks to CPU RAM (--swap-space, default 4 GiB). Neither changes the answer; both explode that request's time-to-first-token. Preemption is the symptom. The arithmetic in section 4 is the cure.

What comes next in this lane: FP8 and NVFP4 KV-cache quantization halves the per-token cost again (GDE #6, Nov 24), and disaggregated prefill/decode splits the pool across machines (GDE #3, Friday Nov 13). The formula in section 1 is the thread through all of them — every later optimization is a way to make the same bytes/token number smaller, or the same pool hold more of them.

Why it breaks: the concurrency checklist

Most KV-cache outages are configuration, not capacity. Run this before you blame the GPU:

max_model_len is set to the model's theoretical maximum. A 128K-capable model with an 8K workload still sizes its pool for 128K unless you cap it. Set it to the longest context you actually serve — the pool, and therefore your concurrency, grows with the difference.
gpu_memory_utilization was raised to 1.0. The default 0.9 leaves headroom for activations and CUDA graphs. At 1.0 without the extensible KV cache, the first large activation allocation OOMs the server. The new extensible cache (0.33-era, latest docs) is the supported way to use the full card.
A KV connector still assumes block size 16. After PR #63231, connector block_size values that aren't multiples of 64 fail validation at startup. Grep your disaggregated-prefill and KV-transfer configs before upgrading past 0.33.
The prompt template drifts. Prefix caching needs byte-identical prefixes. A timestamp, a user ID, or a reordered field at the start of the prompt silently disables sharing for every request — and your pool usage doubles overnight with no code change.
Preemption is treated as normal. Occasional preemption under burst load is fine; steady preemption means the pool is undersized for the workload. The fix is the section-4 arithmetic — smaller model, quantized weights, shorter max_model_len, or more GPUs — not a bigger --swap-space.

Takeaways

The KV cache, not FLOPs, caps concurrency. A 70B model writes 320 KB of notes per token; one 128K conversation is 40 GiB. Size the pool before you size the fleet.
PagedAttention is OS paging for the KV cache. Fixed-size blocks, allocated on demand, freed on completion: waste falls from 60–80% to under 4%, throughput rises 2–4x at the same latency (SOSP 2023).
The default block size just changed, 16 → 64. PR #63231 (Nov 5, 2026) trades less bookkeeping for 24 more wasted tokens per request — ~24K tokens of capacity at concurrency 1,024 — and moves prefix-cache hits to 64-token boundaries. Pin your block size explicitly on mixed fleets; ROCm still defaults to 16.
Weights load first; the pool is what's left. On an H100, an 8B model serves 56 concurrent 8K chats and a 70B FP8 model serves zero. Quantization is a concurrency decision.
Prefix caching is the cheapest memory you will ever buy. A shared 2K-token system prompt across 100 requests saves 60.4 GiB — if the template stays byte-identical.

Next in this series, Friday Nov 13: Prefill vs decode: why splitting them wins — disaggregated prefill/decode, the TTFT-vs-TPOT trade-off, and the latency split on a small model. Follow the dev.to profile to get it — and the other 22 posts in this Google AI series — as they land.

Honesty note: per-token and concurrency figures are recomputed in the companion notebook from public model configs (Llama 3.1, Qwen3, DeepSeek V3) and vLLM's documented defaults; the fragmentation simulation is a labeled toy with a stated length mix, and the paper's 60–80% waste / 2–4x throughput figures are the SOSP 2023 authors' reported results. The 16→64 block-size change (PR #63231, Nov 5, 2026) is confirmed from the PR discussion; the 24K-token capacity figure at concurrency 1,024 is a maintainer estimate, derived in the notebook, and the PR's own 16-vs-64 serving benchmark is still open — treat the net throughput effect as unmeasured, not as a win. No Meta content, per standing policy.

Companion notebook: the runnable tutorial for this post — download it here (open in Colab/Jupyter).

vLLM 0.33, Plainly: CUDA 13, Model Runner V2, and What Breaks on Upgrade

Daniel Sam Pete Thiyagu — Mon, 09 Nov 2026 17:02:31 +0000

vLLM 0.33.0 shipped on November 2, 2026 — 927 commits from 389 contributors — and the headline is not a model. It is a platform break: the wheels and Docker images are CUDA 13.0 only, PyTorch moves to 2.14.0, and three things your deployment may depend on are deleted outright. Upgrade blind and your server does not start, your speculator config errors out, or your KV-transfer connector refuses to connect.

And yet the biggest free speedup in the release changed no kernel at all. A scheduler accounting fix took one speculative-decoding setup from 1,373 to 3,390 output tokens per second — a 2.47x jump — because a handful of leftover tokens was keeping every decode step off the fast path. That is the story of this release in one number: 0.33 is less about new speed than about removing the things that were quietly capping the speed you already had.

This is post #1 of a 25-post series on the Google AI stack — vLLM serving first, then JAX/Keras/Gemma, Google's Agent Development Kit, and Vertex AI — written for the Google Developer Expert AI track. One concept per post, concrete numbers, a runnable notebook. Today: what 0.33 actually changes, why Model Runner V2 is the engine underneath all of it, and the upgrade checklist that keeps your fleet serving.

The ELI5 version

vLLM is the kitchen that serves your language model to the world. Requests come in like orders, the GPU cooks tokens, and the engine's job is to keep the kitchen busy without letting any one order block the rest.

Three upgrades to that kitchen land in 0.33:

The power supply changes (CUDA 13). The kitchen now runs on a new electrical standard. The old plugs — CUDA 12.9 wheels and images — are simply not manufactured anymore. If your machine's driver cannot speak CUDA 13, the new kitchen does not turn on. This is the one breaking change that hits literally every CUDA user.

The head chef's system becomes the only system (Model Runner V2). Model Runner V2 is the redesigned execution engine that has been rolling out since 0.29 — batch-sharded sampling, CUDA-graph memory profiling that sizes the KV cache automatically, and speculative decoding built into the runner instead of bolted on. In 0.32 it became the default on ROCm too, removing the last platform exceptions. In 0.33 there is no old engine left to fall back to: every new feature in this release — the scheduler fix, the lossy verifier, the draft-model changes — is a Model Runner V2 feature. If you are upgrading, you are running V2, full stop.

The menu loses three dishes (the removals). Medusa and the MLP speculator are deleted, the legacy Ray compiled-graph executor is deleted, and the NIXL connector now demands NIXL 1.5 or newer. None of these is a deprecation warning you can ignore for one more release — they are gone, and configs referencing them fail.

How it works: the four changes that matter

1. The platform baseline: CUDA 13 only, PyTorch 2.14

The new baseline is PyTorch 2.14.0, torchvision 0.29.0, Triton 3.8.0, and CUDA 13.0 — for wheels, Docker images, and the default ROCm image moving to ROCm 10.0 (7.2 is still published under a -rocm72 tag). The CUDA 12.9 artifacts are discontinued, not deprecated. Practically: update your NVIDIA driver to one that supports CUDA 13 before upgrading, and install vLLM in its own environment — a shared environment with PyTorch 2.15 installed will get downgraded by the torch==2.14.0 pin, and that is a real way to break a second project while fixing the first.

2. The scheduler fix: 1,373 → 3,390 tokens a second, no kernel changed

This is the most instructive change in the release, and it is pure scheduling. In speculative decoding, a small draft model proposes tokens and the big model verifies them. On the main branch, 2 to 32 leftover draft tokens kept splitting each new prefill into small chunks, so most decode steps mixed prefill into the batch — and a mixed batch cannot take the full decode CUDA graph. The inter-token latency p50 sat at 169 ms.

The 0.33 scheduler change stops drafters from reserving token-budget slots they do not use, so the leftover tokens stop fragmenting the batch. Same model, same hardware (DeepSeek-V4.1-Flash with the DSpark drafter on GB300), same kernels: 1,373 → 3,390 output tokens per second, a 2.47x gain, and ITL p50 falls from 169 ms to 30 ms — 5.6x. The lesson generalizes beyond vLLM: when a serving stack is slow, the bottleneck is often not the kernel but the schedule that decides which work shares a step. Check the batch composition before you blame the math.

The same release also turns on lookahead block hashes for EAGLE and MTP prefix-cache hits by default, so a cached prefix keeps its last block, and adds an optional quantized draft lm_head that the PR reports as +18% decode throughput at concurrency 1.

3. FLy: a lossy verifier you have to opt into

FLy is the release's most interesting — and most dangerous — new feature. Standard speculative decoding is exact: the verifier accepts a draft token only if the target model would have produced it, so the output distribution is preserved. FLy, from the paper "Training-Free Loosely Speculative Decoding" (arXiv 2511.22972), deliberately relaxes that: a rejected position is forgiven when the draft token is semantically close enough, which the PR states "does not preserve the target distribution and may reduce output quality."

The payoff, across the PR's 48 model/platform/batch combinations (DeepSeek-V4-Flash with DSpark, Qwen3.8-27B with MTP, GLM-5.2-FP8 with MTP, Qwen3.5-122B-A10B-FP8 with MTP, on MI355X and B300, batch sizes 1–64): decode throughput 1.041x to 1.310x over standard speculative decoding, median 1.147x, with minimum accuracy retention of 96.3% against target-only decoding. The engine logs a one-time warning that the method is lossy, and standard verification stays the default.

The rule that follows is simple. FLy is a throughput-for-accuracy trade, and the accuracy side is unbounded in the worst case — the PR's tables are single-run accuracy figures, which cannot certify safety for your workload. Leave standard verification on wherever outputs feed an unreviewed downstream decision — code that ships, data that trains the next model, anything a user acts on without a human in the loop. Turn FLy on where speed is the product and a slightly different token is harmless: brainstorming drafts, free-form chat, internal summarization with a human reader. And measure the accuracy on your own prompts before you trust the 96.3% floor — that number is the minimum across the PR's four models, not a guarantee for yours.

4. The ABI-stable wheel: the end of the rebuild tax

The quietest big change is in the build. vLLM's C++ and CUDA code calls into LibTorch constantly, and before this work a change to LibTorch's binary interface could force a full vLLM rebuild even when the Python-visible API was identical. The 0.33 wheel is built against PyTorch's stable ABI — a stable C shim, C++ wrappers over it, and header-only utilities that never link libtorch.so — migrated across 175+ .cpp/.cu files, 120+ headers, and seven compiled dependencies (FlashMLA, DeepGEMM, FlashAttention-2/3, DeepSelect, FlashKDA, QuTLASS).

An independent audit of the published 336 MB wheel reports Torch ABI: STABLE, minimum torch 2.13.0, 13 extensions, zero unstable symbols. Why that matters: across the last four torch releases, vLLM took 33, 34, 44, and 60 days to ship the first release pinning each one — and torch 2.12 was never pinned at all. The stable ABI is supposed to end that wait: swap in a newer compatible torch and the same wheel keeps working.

One catch, and it is a good illustration of the difference between a binary and a package: the wheel can run on newer torch, but the PyPI metadata still declares torch==2.14.0, so pip will not install it that way yet. The capability is shipped; the packaging has not caught up. Expect that pin to loosen in a follow-up release — and do not hand-edit the metadata to force it in production.

The state of the art: what else shipped, and what it means

Eleven new model architectures, led by Reflection Beam-1.0 — the 501B-parameter MoE with 23B active that this rail covered on October 6 — now serving with its DFlash2 drafter, plus Muse Spark Open, Recursive Flint (40B MoE, 2.5B active), Yandex AliceAI-Foundation-80B-A3B, Step-5 Preview, NVIDIA Alpamayo 2 Super, Nemotron-Labs Dory1.0-4B, Nemotron Parse 3.5, MiniCPM-OCR 1.0, Domyn Edge, and the ColUltraVBERT visual document retriever. A new XPress refiner targets block-diffusion drafters. And an OpenAI-compatible /v1/decisions endpoint is now on by default — decision models, the class of model that outputs a choice instead of text, can be served through the same API surface as chat.

The vendor-reported performance numbers, with the caveat attached: opt-in latency-specialized decode for Kimi-K3 on GB300 at 1.5–2x (TP8/TP16), FlashInfer disaggregated-prefill cutting mean time-to-first-token 39–48% at 100K input on 8xB200, and MonoKernel decode at 2.32x P50 interactivity at concurrency 1 on MI300X/MI325X. These are the project's PR benchmarks on the project's hardware, not independent measurements — treat them as "worth testing on your fleet," not as a promise.

Serving operations get two small, genuinely useful changes: concurrent worker startup for tensor-parallel sizes above 1, so a TP8 server stops starting its workers one at a time, and a watchdog that dumps stack traces from stuck processes instead of leaving you to guess which worker hung.

Why it breaks: the upgrade checklist

This release breaks things in the specific, boring ways that take a fleet down at 2 a.m. Run this checklist in order:

Driver first. Update the NVIDIA driver to one that supports CUDA 13 before touching vLLM. The 0.33 artifacts will not install or run on a CUDA 12.9-era driver.
Fresh environment. Install 0.33 in its own venv or container. The torch==2.14.0 pin will downgrade a PyTorch 2.15 setup in a shared environment.
Grep for the removals. Search your configs and launch scripts for Medusa, the MLP speculator, and the legacy Ray compiled-graph executor. All three are deleted — replace Medusa/MLP drafters with EAGLE, MTP, or DSpark, and move Ray usage to the current executor.
NIXL ≥ 1.5. If you use the NIXL connector for KV transfer, upgrade NIXL first; the old connector version will not connect.
Handle the new 503. Requests arriving while generation is paused now receive HTTP 503 instead of queueing. If your clients retry on 503, fine; if they treat it as a hard failure, update the retry policy before you upgrade.
Smoke-test before you roll. Start the server on one node and hit it with the notebook's smoke test — a small model, a short generation, and the /v1/models and /v1/completions endpoints — before the fleet follows. The notebook in this post generates those exact commands, pinned to 0.33.0.

Takeaways

vLLM 0.33.0 is a platform release, not a model release. CUDA 13 only, PyTorch 2.14, and three deletions. The upgrade is a migration, not a version bump — budget an afternoon, not ten minutes.
The biggest speedup changed no kernel. The DSpark scheduler fix (1,373 → 3,390 tok/s, 2.47x) is a reminder to check batch composition before blaming the math.
FLy is a real trade, not a free lunch. Median 1.147x decode throughput for up to 3.7 points of accuracy, with no distribution guarantee. Opt in only where a different token is harmless, and measure on your own prompts.
The ABI-stable wheel ends the rebuild tax in principle. Minimum torch 2.13.0, zero unstable symbols — but PyPI still pins 2.14.0, so the packaging lags the binary by one release.
Model Runner V2 is now the only engine. Every 0.33 feature is a V2 feature. If you have been deferring the V2 migration, 0.33 defers it for you — by removing the alternative.

Next in this series, Tuesday Nov 10: PagedAttention and the KV cache, plainly — why KV memory, not FLOPs, caps your concurrency, with the memory arithmetic for three model sizes. Follow the dev.to profile to get it — and the other 23 posts in this Google AI series — as they land.

Honesty note: all performance figures in this post are the vLLM project's own PR benchmarks or release-notes claims on the project's hardware, not independently tested — including the 2.47x DSpark gain, the FLy 1.041–1.310x range, and the Kimi-K3 / FlashInfer / MonoKernel numbers. The 0.33.0 release is published under the tag v0.33.0rc3 with the prerelease flag off. No Meta content, per standing policy.

Companion notebook: the runnable tutorial for this post — download it here (open in Colab/Jupyter).

System Design, Plainly — Part 4: Message Queues

Daniel Sam Pete Thiyagu — Sun, 08 Nov 2026 16:57:40 +0000

A checkout calls five services in a row — payment, inventory, fraud, email, analytics. Each one takes 40 ms and each one fails 1% of the time. The user waits 200 ms, and 4.9% of checkouts fail in front of them, because a chain fails if any link fails: 0.99⁵ = 95.1% success. Add a sixth service and it gets worse. This is the tax on synchronous calls, and every system that grows past a handful of services pays it — until it stops calling services directly and starts leaving messages for them instead.

A message queue is the buffer between "I need this done" and "it is done." The producer drops a message on the queue and moves on in 2 ms; the consumer picks it up when it is ready, retries it if it fails, and the user never sees the failure. That one indirection — decoupling in time — is what lets a system survive a downstream outage, absorb a traffic spike without falling over, and scale its producers and consumers independently. It is also what introduces the three questions that define every queue design: what delivery guarantee do you get, what ordering do you get, and what happens when the consumer cannot keep up.

This is Part 4 of System Design, Plainly: one concept per post, evergreen, beginner-friendly, with the concrete numbers and the failure modes. Today: message queues — what they decouple, the three delivery guarantees and what each one costs, why ordering and parallelism are the same tradeoff, and the backpressure math that decides whether a spike is a blip or an outage.

The ELI5 version

Imagine a restaurant. The waiter takes your order and walks it to the kitchen — that is a synchronous call. If the kitchen is backed up, the waiter stands there waiting, the dining room fills with waiters standing around, and new customers cannot even get seated. Now put a ticket rail between them. The waiter clips the order to the rail and is back at your table in seconds. The kitchen pulls tickets in order, one cook per ticket, and if a dish burns, the ticket goes back on the rail to be remade — you never see it happen.

The ticket rail is the queue. The waiter is the producer, the kitchen is the consumer, and the rail holds the work while the two sides run at different speeds. Three things about the rail matter, and they are the whole subject of this post.

First, the rail absorbs bursts. Lunch rush hits and fifty orders arrive in a minute; the kitchen still cooks one at a time. The rail grows, nobody is turned away, and the wait grows in proportion to the queue — that is Little's law, and it is the only equation in this post you need to memorize: average wait = queue length ÷ processing rate.

Second, the rail decides what "done" means. A ticket can be thrown away after one attempt (fast, but a dropped ticket means a lost order), retried until it succeeds (nothing is lost, but a burnt dish can be served twice), or retried with a check that the dish was not already served (nothing lost, nothing doubled — but now the kitchen has to remember what it already cooked).

Third, the rail decides the order. One cook, one rail, strict first-in-first-out. Ten cooks, ten rails, ten times the throughput — but two tickets for the same table can now finish in either order, and "charge the card, then send the receipt" can arrive as "receipt, then charge" unless both tickets go down the same rail.

How it works: producer, broker, consumer — and the three guarantees

The moving parts

A producer writes messages to a topic (Kafka's name; RabbitMQ calls it a queue, NATS a stream, SQS a queue). The broker stores them durably — on disk, replicated across nodes — and a consumer reads them. In the log model (Kafka, Redpanda, NATS JetStream, Pulsar) the broker keeps messages after delivery, so a new consumer can replay history from an offset; retention is measured in days or size. In the classic queue model (RabbitMQ, SQS) a message is deleted once acknowledged; retention is measured in minutes to 14 days, and there is no replay.

Consumers scale in two dimensions. A consumer group splits a topic's partitions among its members — each partition is consumed by exactly one member at a time, so adding consumers adds throughput until you run out of partitions. And partitions split a topic by key: messages with the same key always land on the same partition, which is how you get per-key ordering without giving up parallelism. The arithmetic from our simulation: one partition, strict global order, 5,000 messages a second; 32 partitions, 160,000 a second — 32× the throughput, but ordering now holds only within a partition. The key you partition by is a sharding decision in disguise (Part 3), and it has the same whale problem: a key carrying 30% of traffic leaves the busiest partition at 32.2% even at 32 partitions, against an ideal 3.1%.

The three delivery guarantees, priced

Every queue offers one of three guarantees, and the price is paid in lost messages, duplicate messages, or bookkeeping:

At-most-once — acknowledge before processing. Fast, no duplicates, but a crash between the ack and the processing loses the message. Our simulation on 10,000 messages with a 1% crash probability per message: 95 lost (0.95%), zero duplicates. Use it for telemetry and metrics, where losing 1% of samples is invisible.

At-least-once — acknowledge after processing. Nothing is lost, but a crash between processing and the ack redelivers the message, so the consumer sees it twice. Same simulation: 0 lost, 95 duplicates (0.95%). This is the default for almost everything, and it means your consumer must tolerate duplicates — which is why "idempotent consumer" is the most repeated advice in queue design: processing the same message twice must have the same effect as processing it once. A payment handler that checks a dedup table before charging is idempotent; one that charges on every delivery is a double-charge machine.

Exactly-once — at-least-once delivery plus an idempotent consumer. Same simulation: 0 lost, 0 duplicates. Kafka's version is stronger still: idempotent producers (sequence numbers per producer, broker dedups retries) plus transactions (KIP-98) let a consumer read, process, and write atomically, so the "exactly once" holds end-to-end — but only inside the Kafka ecosystem, and only if the consumer is written to the contract. The honest framing: exactly-once is not a switch you flip on the broker. It is a property of the producer, the broker, and the consumer together, and any one of them can break it.

Backpressure: the queue is a shock absorber with a capacity

When producers outrun consumers, the queue grows — and the wait grows with it. Our simulation: producers burst at 12,000 messages a second for 60 seconds against a consumer holding 8,000. The backlog peaks at 240,000 messages, and the last message in line waits 30 seconds (240,000 ÷ 8,000). The burst ends, the producer drops to 4,000 a second, and the backlog drains in another 60 seconds. Add a 30-second consumer outage mid-burst and the peak doubles to 480,000 — the outage costs twice, once in the messages that pile up while the consumer is down and once in the drain time after it returns.

Three responses exist, and every production queue uses all three: slow the producer (backpressure — the broker rejects or throttles writes, or the client blocks), grow the consumer (add consumers up to the partition count; beyond that, add partitions, which is a resharding decision), and shed load (drop low-priority messages, or route them to a dead-letter queue). What you must never do is let the queue grow without a bound: an unbounded queue is a memory leak with a dashboard, and the disk-full broker is the most boring and most common queue outage there is.

Retries, dead letters, and the poison message

A message that fails processing goes back on the queue — but a message that fails every time is a poison message, and naive retry turns it into an infinite loop that starves every other message behind it. The standard fix is a retry policy with backoff (retry after 1 s, then 10 s, then 60 s) and a dead-letter queue (DLQ): after N attempts the message is moved aside for a human to inspect, and the consumer moves on. The DLQ is not a graveyard — it is a work queue for the on-call engineer, and a DLQ nobody reads is a silent data-loss channel. The same discipline applies to ordering: a single slow message at the head of a partition blocks every message behind it (head-of-line blocking), which is why per-message timeouts and DLQs exist — the queue must be able to give up on one message without giving up on the partition.

The state of the art: four systems, one split

The queue world splits into two camps — the log and the broker — and in 2026 the log is winning the workloads that matter.

1. Apache Kafka — the log, and the default. A distributed append-only log with topics, partitions, consumer groups, and retention measured in days to forever. Kafka 4.0 (March 18, 2025) removed ZooKeeper entirely — KRaft, its built-in Raft consensus, is now the only mode, so a 3-broker cluster is self-sufficient and controller failover runs under 5 seconds. The 4.0 release also made the rewritten consumer rebalance protocol generally available (KIP-848), and share groups — queue-style cooperative consumption that is not limited by partition count — went production-ready in 4.2 (February 2026); the current release, 4.4.0, shipped October 22, 2026. Scale is not theoretical: LinkedIn runs 7 trillion messages a day on Kafka, and more than 80% of the Fortune 100 use it. The cost is operational weight: JVM heap (typically 3–8 GB per broker), partition management, and rebalancing — which is exactly the gap the next two systems attack.

2. Redpanda — the log, rewritten in C++. Kafka-compatible on the wire, built on Seastar with a thread-per-core engine, no JVM, no ZooKeeper ever, single binary. Its pitch is 10× lower tail latency with 3× fewer nodes, plus tiered storage to S3/GCS that predates Kafka's equivalent. The honest benchmark picture is messier than either vendor's marketing: an April 2026 head-to-head on identical 4 vCPU / 8 GB hardware (Kafka 4.2.0 KRaft vs Redpanda 26.1.2, Kafka's own perf-test tools, 5M messages sustained) measured Kafka at 211,283 records/s sustained against Redpanda's 89,455 — 2.4× higher — with p99 latency 41 ms against 514 ms. Redpanda's own benchmarks report the opposite ranking on p99 (2–5 ms vs Kafka's 10–15 ms). Both cannot be right in general, which is the point: the ranking flips with message size, partition count, ack mode, and hardware. The same April test shows the shape — Kafka wins at 6 partitions and acks=all (479K vs 291K rec/s; 326K vs 39K rec/s), Redpanda wins on single-partition acks=1. Load-test your own workload; never buy a broker on a benchmark you did not run.

3. NATS JetStream — the log, made light. NATS has been the lightweight pub/sub system since 2011; JetStream (2021) layers persistence, delivery guarantees, and replay on top of core NATS. A NATS server is a single Go binary with a ~22 MB baseline footprint, sub-millisecond p99 pub/sub latency, and built-in request-reply — which makes it the default for service-to-service communication inside Kubernetes, where Kafka's operational weight is unjustifiable for under a million messages a day. On a 3-node, 8 vCPU test it sustained ~820,000 messages a second against Kafka's 1,200,000 batched. The tradeoff: retention is bounded by disk and replication, long-term replay across months is Kafka's job, and the ecosystem (Connect, Schema Registry, Streams) is thinner.

4. The managed queue — SQS, and the broker camp. Amazon SQS is the "no operations" answer: standard queues are nearly unlimited in throughput with best-effort ordering and at-least-once delivery; FIFO queues give exactly-once processing and per-group ordering at 300 transactions a second per action (up to 70,000 in high-throughput mode). Retention is 60 seconds to 14 days, there is no replay, and that is the point — SQS is a queue, not a log. RabbitMQ (AMQP, flexible routing, competing consumers, one CPU core per queue replica) and Apache Pulsar (BookKeeper storage, tiered storage, multi-tenancy) fill the same broker camp. Choose the log when you need replay, audit, and stream processing; choose the queue when you need task distribution and zero operations.

Why it breaks

Every queue outage is one of these seven:

The poison message. One malformed message fails every attempt, retries with no backoff, and pins a consumer — or, in a single-partition topic, blocks the whole partition behind it. Fix: exponential backoff, a max-attempts cap, and a dead-letter queue that a human actually reads.
The retry storm. A downstream service goes down, every consumer retries at once, and the retry traffic is a second outage on top of the first — aimed at the service that is already down. Fix: backoff with jitter, circuit breakers on the consumer, and a global retry budget.
The unbounded queue. Producers outrun consumers for hours, the backlog passes the disk, and the broker dies — taking the buffered work with it. Fix: retention limits, disk alarms at 70%, backpressure that throttles producers before the broker is full, and an autoscaling rule tied to consumer lag, not CPU.
The consumer-lag blindness. Nobody watches lag, so a slow consumer is discovered by users reporting stale data — 240,000 messages deep, 30 seconds stale, and the drain takes another minute after the burst ends. Fix: alert on lag (messages and age of oldest message), and know your drain time before the spike, not during it.
The exactly-once illusion. The broker is configured for exactly-once, the consumer charges the card on every delivery, and a crash between processing and ack double-charges 1% of orders. Fix: idempotency keys and dedup tables in the consumer — the guarantee lives in your code, not in the broker's config.
The ordering accident. Two messages for the same order land on different partitions (bad key choice — no key at all, or a key that does not match the entity), and "cancel" is processed before "charge". Fix: partition by the entity the order applies to, and test the ordering with a chaos run that shuffles delivery.
The rebalance storm. A consumer joins or leaves, the group rebalances, and during the rebalance nobody consumes — or worse, two consumers own the same partition briefly and both process the same messages. Kafka's KIP-848 rewrite and share groups (4.2) exist to shrink this window; the operational fix is the same as ever: stable consumer membership, generous session timeouts, and a rebalance budget in your latency SLO.

The interview angle

"Design a notification system" or "design Uber's ride-matching" is a message-queue question in disguise, and the expected answer is a tour of exactly this post: producers and consumers decoupled by a durable log; topics partitioned by the entity that needs ordering (user ID for notifications, ride ID for matching); consumer groups for parallel processing; at-least-once delivery with idempotent consumers as the default, exactly-once only where money moves; a dead-letter queue for poison messages; and backpressure plus lag alerting so a spike is a dashboard event, not an outage.

The follow-ups that separate good answers from great ones: how do you guarantee order for one user's notifications? (same key → same partition, one consumer per partition at a time); what happens when a consumer crashes mid-message? (offset not committed, message redelivered, idempotent consumer absorbs the duplicate); how do you handle a message that always fails? (backoff, then DLQ, then a human); when is a queue the wrong answer? (when the caller needs an immediate answer — that is an RPC — or when the data fits in a database and the "queue" is a table with a status column, which is a queue with transactions and no new infrastructure); and log or queue? (replay and audit → log; task distribution and zero ops → managed queue).

In the real world, the decision is usually not which guarantee to implement — it is which system to buy. Kafka if you need replay, ecosystem, and proven scale; Redpanda if you want Kafka's protocol with less operational weight; NATS JetStream if you are inside Kubernetes and under a million messages a day; SQS if you want zero operations and can live without replay. The guarantee still matters, because the broker does not make your consumer idempotent — and the consumer is where exactly-once actually lives.

Takeaways

A queue moves failure off the user's path: a 5-service chain fails 4.9% of requests and costs 200 ms; the same work through a queue answers in 2 ms and retries failures out of sight.
There are three guarantees and no free lunch: at-most-once loses ~1% at a 1% crash rate, at-least-once duplicates ~1%, exactly-once needs an idempotent consumer — the guarantee lives in your code, not the broker's config.
Backpressure is Little's law: a 60-second burst at 1.5× consumer speed leaves a 240,000-message backlog and a 30-second wait; a 30-second consumer outage doubles it. Alert on lag and know your drain time before the spike.
Ordering and parallelism are the same tradeoff: 1 partition gives strict order at 5,000 msg/s, 32 partitions give 160,000 msg/s with order only within a partition — and a 30% hot key floors the busiest partition at 32.2% no matter how many partitions you add.
Kafka 4.0 killed ZooKeeper (March 2025, KRaft only) and 4.2 made share groups production-ready; the benchmark ranking between Kafka and Redpanda flips with the workload, so load-test your own before choosing.
The poison message is the most common queue outage: backoff, a max-attempts cap, and a dead-letter queue a human actually reads — a DLQ nobody reads is silent data loss.
Try it yourself. The companion notebook replays all four simulations: the 4.9% sync-chain failure vs 0% async, the 95-lost / 95-duplicate delivery semantics, the 240,000-message backlog and 120-second drain, and the 5,000 → 160,000 msg/s ordering curve with the 32.2% hot-partition floor.

This is Part 4 of System Design, Plainly. Part 5 — CAP and PACELC — is next. Follow me on dev.to (@danielsamfdo) so you don't miss it.

Sources: Apache Kafka 4.0 release notes (March 18, 2025 — ZooKeeper removed, KRaft only, KIP-848 GA); Kafka 4.2 / 4.4.0 release notes (share groups GA, Oct 22, 2026); Apache Software Foundation ("more than 80% of Fortune 100 use Kafka"; LinkedIn 7 trillion messages/day — company-reported); ComputingForGeeks, "Kafka vs Redpanda: Real Benchmarks on Identical Hardware" (April 2026, Kafka 4.2.0 vs Redpanda 26.1.2, 4 vCPU / 8 GB, kafka-producer-perf-test — sustained 211,283 vs 89,455 rec/s, p99 41 vs 514 ms, avg 2.53 vs 315.42 ms; single-partition acks=1 and 6-partition figures from the same test); Redpanda vendor benchmarks (p99 2–5 ms vs Kafka 10–15 ms — vendor-reported, conflicts with the April 2026 head-to-head, flagged in post); NATS JetStream docs and 2026 comparisons (single Go binary, ~22 MB footprint, sub-millisecond p99, 820K msg/s on a 3-node 8 vCPU test vs Kafka 1.2M batched); AWS SQS docs (FIFO 300/s per action, up to 70K high-throughput; retention 60 s–14 days). All simulation numbers are from the companion notebook and the diagrams in this post; toy diagrams are labelled illustrations. No content about excluded vendors appears in this post.

Companion notebook: the runnable tutorial for this post — download it here (open in Colab/Jupyter).

System Design, Plainly — Part 3: Sharding

Daniel Sam Pete Thiyagu — Sat, 07 Nov 2026 16:58:04 +0000

One database server can only get so big. You can buy a bigger machine — vertical scaling — until the biggest machine you can buy is not big enough. You can add read replicas until reads are free — but replicas do not help writes, because every write still lands on one primary. At some point the dataset outgrows one server's disk, or the write rate outgrows one server's CPU, and there is exactly one move left: split the database across many machines, each owning a slice of the data. That is sharding, and it is the last scaling lever and the most expensive one. Get the shard key right and ten machines hold ten times the data and absorb ten times the writes. Get it wrong and you have ten machines doing the work of one, plus a routing layer, plus a resharding project, plus a class of bugs that did not exist before.

This is Part 3 of System Design, Plainly: one concept per post, evergreen, beginner-friendly, with the concrete numbers and the failure modes. Today: sharding — what it is, the four ways to split the data, why the shard key is the highest-consequence decision in the system, and the three costs (cross-shard queries, rebalancing, and the hot key) that decide whether sharding was worth it.

The ELI5 version

Imagine a library with one librarian. She can shelve only so many books, and she can only check out so many per hour. The library grows, the queue grows, and hiring a faster librarian — a bigger server — stops working when the fastest librarian in the world is still too slow.

So you split the library into ten branches. Each branch holds a slice of the collection and has its own librarian. Now ten checkouts can happen at once, and the collection can grow tenfold. But three new problems appear, and they are the whole subject of this post.

First, you need a rule for which branch holds which book. Alphabetical by title? By author? By genre? That rule is the shard key, and it decides everything: how evenly the books spread, how fast a lookup is, and whether one branch ends up holding all the popular books while the other nine sit idle.

Second, you need a directory. A reader walks in and asks for a book — someone has to know which branch holds it without asking all ten. That someone is the routing layer, and every request pays for it.

Third, some questions now span branches. "How many books did we check out today?" used to be one query against one catalogue. Now it is ten queries and a merge. And "move this book from branch 3 to branch 7" used to be one transaction; now it is a distributed transaction, and if the move fails halfway, the book exists in two places or in none.

Sharding is not a performance optimization. It is a trade: you buy write throughput and storage capacity, and you pay with query complexity, operational complexity, and a new family of failure modes. The rule of thumb from every team that has done it: shard last. Tune the queries, add indexes, split hot tables vertically, add read replicas — and only when writes or storage genuinely exceed one node do you shard.

How it works: the shard key, then the four ways to split

The shard key is the decision

Every row is assigned to a shard by its shard key — one column (or a combination) whose value determines the destination. The key must satisfy two conditions that pull in opposite directions: it must spread load evenly, and it must be present in the queries you actually run. A key that spreads perfectly but never appears in a query is useless — every query becomes a scatter-gather across all shards. A key that appears in every query but concentrates load is worse — you sharded and still have a hotspot.

The classic good keys are tenant IDs and user IDs: high cardinality, evenly distributed, and present in nearly every query a multi-tenant app runs ("show me this user's orders"). The classic bad keys are timestamps and auto-incrementing IDs: every new row lands at the newest end of the key space, so the newest shard absorbs all writes. And a key with a whale — one tenant, one celebrity, one channel that holds 12% of all rows — cannot be fixed by adding shards, because a single key's rows are indivisible: they all live on one shard. More on that below.

Four schemes, one tradeoff: even spread vs locality

Range sharding assigns contiguous key ranges to shards — IDs 0–1M on shard 1, 1M–2M on shard 2. Keys stay sorted inside a shard, so a range scan ("last week's orders") touches one or two shards and is cheap. The price is the hot tail: writes keyed by time or by auto-increment ID all land on the shard owning the newest range. In our simulation, 100% of today's writes land on the newest shard under range-by-time sharding, while hash sharding spreads them evenly at 12.5% per shard. Range is the right answer when range scans dominate and writes are spread across the key space — time-series data with a hashed timestamp, or data keyed by something that does not grow monotonically. CockroachDB, TiDB, HBase, and Bigtable all shard by range internally, splitting a range when it grows past a size threshold (CockroachDB at 512 MiB, TiKV at 96 MB, HBase at 10 GB).

Hash sharding hashes the key first — shard = hash(user_id) % N — and destroys key ordering to buy uniform distribution. Consecutive user IDs scatter across shards, so no shard is hotter than any other, and the hot-tail problem disappears. The price is the range scan: adjacent values live on different shards, so "last week's orders" fans out to all of them. Hash is the right answer when lookups are by exact key and range scans are rare. DynamoDB and Redis Cluster hash-partition; MongoDB's hashed shard key uses MD5.

Consistent hashing fixes hash sharding's worst property: what happens when you add a shard. With hash(key) % N, changing N remaps nearly every key — our simulation on 200,000 keys shows adding a 9th shard to an 8-shard cluster moves 88.9% of keys, and adding a 10th moves 80.0%. A consistent-hash ring maps keys and nodes onto the same circle, and each key belongs to the next node clockwise — so adding one node moves only the keys that fall between it and its predecessor, about 1/N of the data. Same simulation: 11.0% move when going 8 → 9, 20.5% when going 8 → 10 (two nodes added). Virtual nodes — each physical node owns 200 points on the ring instead of one — smooth out the distribution so no node inherits a fat slice by luck. Cassandra and Dynamo use consistent hashing; it is also the mechanism from Part 1's load-balancing ring, applied to data instead of requests.

Directory-based sharding keeps an explicit lookup table — key → shard — instead of computing the placement from the key. It costs one extra lookup hop on every request, and the directory is a critical-path dependency. What it buys is freedom: you can move a single tenant to its own shard without rehashing anything, which is exactly the escape hatch for the whale tenant that hash and range sharding cannot handle. Vitess calls this a lookup vindex; multi-tenant SaaS products that isolate their largest customers on dedicated shards all run some version of it.

The routing layer, and what a query costs without the key

Between the application and the shards sits a router — a proxy that parses the query, works out which shards hold the rows, sends the query there, and merges the results. In Vitess this is VTGate; in MongoDB it is the mongos router; in Citus it is the coordinator node. The application still speaks ordinary SQL, but the cost of a query now depends on whether it carries the shard key.

Our simulation on a 16-shard cluster, per-shard capacity 1,200 queries a second, makes the arithmetic concrete. A targeted query — shard key present — touches one shard: cluster capacity 19,200 queries a second, p99 latency 12.6 ms. A lookup-vindex query — one extra hop to resolve the key — touches two shards: 9,600 queries a second, p99 22.6 ms. A scatter-gather — no shard key, ask all sixteen and merge — touches every shard: 1,200 queries a second, p99 20.4 ms, bounded by the slowest shard plus the merge. The scatter query costs sixteen times the work of a targeted one, so a workload that goes from 5% scatter to 50% scatter does not get 45% slower — it saturates the cluster. A small independent experiment that rebuilt the Vitess pattern on five real MySQL servers measured the same shape in the wild: 1,194 targeted reads a second, 710 with a lookup vindex, 307 scatter-gather. The shard key decides the price of a query, and a query without it is the most expensive thing in the system.

Secondary indexes and cross-shard transactions

Two things that were free on one node become expensive or impossible across shards.

Secondary indexes only work locally. An index on email inside each shard answers "find the user with this email" only by asking every shard — unless you build a global secondary index, a separate distributed structure that maps email → shard, which is itself a sharded table that must be kept in sync on every write. Most teams either route by the indexed column (make it the shard key) or accept the fan-out for rare queries.

Cross-shard transactions lose atomicity. Moving 10 credits between two accounts on different shards is two commits, and if the process dies between them, the credits exist in both places or in neither. A five-server experiment that simulated exactly this failure lost the credits in 52 of 2,000 transfers — 2.6% of cross-shard transfers silently half-applied. The fixes are well known and all cost something: two-phase commit (correct, but slow and blocking — a crashed coordinator leaves shards holding locks), the saga pattern (a sequence of local transactions with compensating actions that undo the earlier steps, which means your application has to know how to un-do a payment), or avoiding the problem entirely by co-locating related rows — putting a user's orders on the same shard as the user, so the common transaction never crosses a shard boundary. Distributed SQL systems (CockroachDB, Spanner, TiDB) keep ACID across shards with Raft consensus and distributed commit protocols, and charge for it in write latency.

The state of the art: three ways to shard in 2026

1. The proxy: Vitess (MySQL). Built at YouTube in 2010 after replicas and application-level sharding stopped being enough. It sits between the application and a fleet of MySQL servers: VTGate parses SQL and routes by vindex, vttablet multiplexes connections, and resharding and schema changes run online — a schema change builds a ghost table, backfills it, tails the binlog, and swaps it in, instead of locking a large table for an ALTER. Slack moved 99% of its MySQL traffic onto Vitess, peaking at 2.3 million queries per second. GitHub split its main cluster by schema domain first, then moved its largest domains onto horizontally sharded Vitess clusters; by 2023 it ran 1,200+ MySQL hosts, 300+ TB, and 5.5 million queries per second. PlanetScale runs Vitess as a managed service. The constraints reach into your schema: no stored procedures, triggers or events, no foreign keys across shards, limited cross-shard joins and transactions — most teams design around them without noticing. Vitess is now the reference answer for "we have MySQL and we have outgrown one primary."

2. The extension: Citus (PostgreSQL). Citus turns Postgres into a distributed database by adding hash-distributed tables to ordinary Postgres: the application still writes standard SQL, the coordinator routes it. It is now part of the Azure ecosystem and is the default answer for multi-tenant Postgres and for analytical workloads that outgrow one node. Like Vitess it uses a fixed shard count — create far more shards than nodes (say 1,000 shards across 10 nodes), map keys with hash(key) % 1000, and move whole shards when you add a node, never splitting them. The limit is that you can never have more nodes than shards, and if the original estimate was wrong, resharding means rewriting everything.

3. Distributed SQL: CockroachDB, TiDB, Spanner. These are sharded from birth — the storage engine is natively distributed (Raft consensus plus LSM trees), ranges split and rebalance automatically, and the application connects to a single endpoint while the database manages placement underneath. Cross-shard transactions are ACID and serializable. The tradeoff is write latency: a write that needs consensus across regions is slower than a write to one Postgres primary, and you are now operating a distributed system with its own failure modes. MongoDB sits in the middle: sharding is built in, you pick a shard key, and the balancer moves chunks (128 MB by default) between shards automatically. DynamoDB hash-partitions behind the scenes — each partition holds about 10 GB and roughly 3,000 read units — and its adaptive capacity shifts throughput to hot partitions, which papers over a mild hot key but cannot save a truly indivisible one.

The honest 2026 summary: managed distributed databases have made sharding boring in the good sense — automatic rebalancing, online schema changes, and cross-shard transactions without DIY complexity. What they have not changed is the shard key. No system picks it for you, and a bad one still costs you the cluster.

Why it breaks

Every sharding outage is one of these six:

The hot key. One tenant, one celebrity, one flash-sale SKU — a single key that holds a large share of rows or receives a large share of writes lands wholly on one shard, and no shard count helps. Our simulation: a whale tenant holding 12.3% of all rows leaves the busiest shard at 12.8% of rows even at 128 shards, against an ideal 0.78%. An independent Vitess rebuild measured the same shape: the busiest channel held 12.3% of messages, and at 128 shards the busiest shard still held 14.6%. Fixes, in order of cheapness: split the hot key (per-region copies, a suffix on the key to spread one logical key across shards), replicate it, or move the whale to a dedicated shard via a directory — never just add shards.
The scatter-gather avalanche. A new feature ships a query without the shard key — a dashboard, an admin search, a report. It fans out to every shard on every request, consumes sixteen times the work, and the cluster saturates at one-sixteenth of its rated capacity. Guard it with a lint rule (GitHub's approach when it split by schema domain: linters that reject cross-domain joins) or a query governor that refuses fan-out above a threshold.
The resharding project. You outgrow your shard count and must redistribute live data. DIY resharding is a maintenance window, dual writes, a backfill, and a cutover — YouTube's first reshard took an entire night of primary downtime, and Slack's migration from one shard key to another ran from 2016 to 2020 with double writes and a parallel read-diffing system. Managed systems do it online (Vitess splits a shard with zero write errors in the five-server rebuild: 2,158 lookups, every message found exactly once, on the right shard), but it is still a project, not a config change. Design the rebalancing strategy before you need it.
The cross-shard transaction that half-applied. Two commits, one crash between them, and money or inventory exists twice or not at all. If a business operation spans shards, either co-locate the rows, use a saga with compensating actions, or accept two-phase commit's latency and blocking — do not assume two local transactions equal one distributed one.
The schema change multiplied. A 30-second ALTER TABLE becomes 30 seconds × 16 shards, and every one must succeed. Rolling migrations need tooling — Vitess's ghost-table flow, or a migration runner that applies the change shard by shard with a rollback plan. A migration that fails on shard 14 of 16 is a partial schema, which is worse than no migration.
The rebalance that moved the hot shard onto the small node. Rebalancing moves data, and moving data costs network and disk. At petabyte scale, even consistent hashing's "only K/N keys move" is terabytes of transfer — adding one node to a 10-node cluster still moves 10% of all data. Rebalance during low traffic, throttle it, and never let an automatic balancer migrate a hot range onto a node that is already saturated.

The interview angle

"Design Twitter / Instagram / WhatsApp" is a sharding question in disguise, and the expected answer is a tour of exactly this post: shard only after vertical scaling, replicas, and indexes are exhausted; pick the shard key by the queries you actually run (user ID for a social app, tenant ID for SaaS); hash for even spread, range when range scans dominate, consistent hashing so adding a node moves ~1/N of the data instead of ~90%; a router in front, targeted queries cheap, scatter-gather expensive; co-locate related rows so transactions stay single-shard; and name the failure modes — hot key, scatter-gather avalanche, resharding, half-applied cross-shard transactions.

The follow-ups that separate good answers from great ones: how do you choose the shard count? (fixed and generous — far more shards than nodes — so you move whole shards, or let the system split ranges automatically at a size threshold); what happens when a shard dies? (each shard is itself replicated — sharding is about scale, replication is about availability, and you need both); how do you run a query that needs all shards? (you fan out, but you budget for it — aggregations and reports go to a read replica, a warehouse, or a pre-computed rollup, not to the live shards); and when is sharding the wrong answer? (when the dataset fits on one node with room to grow, when the workload is read-heavy — replicas fix that — or when every query needs a cross-shard join, which means the data model was not designed for sharding in the first place).

In the real world, the decision is usually not which scheme to implement — it is which system to buy. Vitess if you are on MySQL and outgrowing one primary, Citus if you are on Postgres, CockroachDB or TiDB if you are starting greenfield and want the database to manage placement, MongoDB or DynamoDB if your data model fits a document or key-value shape. The scheme still matters, because the system does not pick your shard key — and the shard key is the one decision that follows you for the life of the system.

Takeaways

Sharding is the last scaling lever, not the first: vertical scaling, indexes, and read replicas come first — shard only when writes or storage genuinely exceed one node.
The shard key is the highest-consequence decision in the system: it must spread load evenly and appear in the queries you run — a key that does one without the other is a hotspot or a scatter-gather.
Adding a node must not move your data: hash(key) % N remaps 88.9% of keys when going 8 → 9 shards; a consistent-hash ring moves 11.0%. That is why the ring exists.
A query without the shard key costs 16× the work on a 16-shard cluster: targeted 19,200 qps, lookup vindex 9,600, scatter-gather 1,200 — and its p99 is bounded by the slowest shard.
More shards do not fix a hot key: a whale tenant holding 12.3% of rows leaves the busiest shard at 12.8% even at 128 shards, against an ideal 0.78%. Split the key, replicate it, or move the whale to its own shard.
Cross-shard transactions are not atomic by default: two commits with a crash between them half-applied 2.6% of transfers in one experiment. Co-locate related rows, use sagas, or pay for two-phase commit.
Try it yourself. The companion notebook replays all four simulations: the 88.9% vs 11.0% remap, the whale-tenant floor (12.8% at 128 shards), the scatter-gather throughput collapse (19,200 → 1,200 qps), and the range-vs-hash write split (100% vs 12.5% on the newest shard).

This is Part 3 of System Design, Plainly. Part 4 — Message Queues — is next. Follow me on dev.to (@danielsamfdo) so you don't miss it.

Sources: Vitess / PlanetScale docs and engineering history (YouTube 2010 origin; Slack 99% of traffic, 2.3M qps peak; GitHub 2023, 1,200+ hosts, 300+ TB, 5.5M qps — company-reported); "How YouTube, Slack and GitHub Shard MySQL" (devops-daily / dev.to, Oct 2026 — independent five-server Vitess-pattern rebuild: targeted 1,194 vs lookup 710 vs scatter 307 reads/s; 52/2,000 half-applied cross-shard transfers; 14.6% busiest shard at 128 shards; Slack re-shard 2016–2020; YouTube first reshard "an entire night of master downtime"); HLD Handbook, "Partitioning Schemes: Range, Hash, Consistent Hash, Directory" (CockroachDB 512 MiB / TiKV 96 MB / HBase 10 GB split thresholds; fixed-shard systems Citus, Riak, Elasticsearch, Couchbase); ByteLedger, "Database Sharding Explained 2026" (Citus 12, Vitess 18, CockroachDB 23.x); MongoDB sharding docs (chunk balancing; MD5 hashed shard key); DynamoDB partitioning (hash partitions, ~10 GB / 3,000 RCU per partition, adaptive capacity). All simulation numbers are from the companion notebook and the diagrams in this post; toy diagrams are labelled illustrations. No content about excluded vendors appears in this post.

Companion notebook: the runnable tutorial for this post — download it here (open in Colab/Jupyter).

System Design, Plainly — Part 2: Caching

Daniel Sam Pete Thiyagu — Fri, 06 Nov 2026 17:04:15 +0000

Every request your system serves is a race against a ladder. Your CPU's L1 cache answers in about 1 nanosecond. RAM takes ~100 nanoseconds. An NVMe SSD takes ~100 microseconds. A round trip across the local network takes ~500 microseconds. A database query takes 5 to 50 milliseconds. Each layer is roughly a thousand times slower than the one above it — and a cache is the act of answering from a higher rung than the question deserves. Get the hit ratio from 90% to 99% and the average request gets 5.5× faster, from 5.5 ms to 1.0 ms. That is the whole game: a cache is not a performance optimization, it is a decision about which rung of the ladder your users live on.

This is Part 2 of System Design, Plainly: one concept per post, evergreen, beginner-friendly, with the concrete numbers and the failure modes. Today: caching — what it is, the four layers that matter, why the eviction policy is worth real money, and the stampede that takes down systems that did everything else right.

The ELI5 version

Imagine a library. You need a book. The book you want is either on your desk (instant), on the shelf next to your desk (a few seconds), in the library's stacks (a few minutes), or in a warehouse across town (an hour). A cache is the desk and the shelf: you keep the books you actually use close, and you accept that the desk only holds ten books.

The librarian's job has three parts, and each one is a design decision. What goes on the desk — when you fetch a book from the warehouse, do you put a copy on the desk for next time? What leaves the desk when it is full and a new book arrives — the one you haven't touched in longest, or the one you touch least often? And what happens when the book changes — someone edits a chapter in the warehouse, and your desk copy is now wrong. The first question is the read pattern, the second is the eviction policy, the third is invalidation — and invalidation is the one with the famous warning: "There are only two hard things in computer science: cache invalidation and naming things." (Phil Karlton.)

The numbers make the stakes obvious. A cache hit from an in-memory store like Redis or Valkey costs about 0.5 ms. A database query costs about 50 ms. The average latency of a request is hit_ratio × 0.5 ms + (1 − hit_ratio) × 50.5 ms — a miss pays for the cache check and the database trip. At 90% hits the average is 5.5 ms. At 99% it is 1.0 ms. At 99.9% it is 0.55 ms. Nine points of hit ratio buys a 5.5× speedup, and the last 0.9 points buys another ~2×. That is why the eviction policy — the librarian's second decision — is not an implementation detail. On one simulated Zipfian trace, the wrong policy at the same cache size costs 13 points of hit ratio. At 10,000 requests a second, that is 1,300 extra database queries a second, for free, forever.

How it works: four layers, then the eviction ladder

A request crosses up to four caches before it reaches the database, and each layer only knows what it needs to.

The browser cache is the closest one to the user and the cheapest to run — it costs you nothing, because it is the user's disk. Cache-Control headers tell the browser how long to keep a static asset; a hashed filename (app.9f3c2a.js) makes the cache entry immutable, so a new deploy is a new filename and there is never a stale read. This layer absorbs the majority of repeat traffic for static content and you do not operate it at all.

The CDN edge cache is the same idea at planetary scale: the response is stored in a point of presence a few milliseconds from the user instead of in your origin data center. A cache hit at the edge never touches your infrastructure. This is the layer that absorbs product launches and breaking-news spikes — and the one whose miss rate you watch during an incident, because a cold edge cache is a traffic spike aimed at your origin.

The application cache is the one you operate: an in-memory store — Redis, its fork Valkey, Memcached, Dragonfly, or a local in-process cache like Caffeine — sitting between your application servers and the database. It holds the computed answers: a user profile, a product page, a feed, a session. A hit is ~0.5 ms; a miss is a database query plus the work of filling the cache. This is the layer the rest of this post is about.

The database's own cache is the last line: the buffer pool, the query-result cache, the operating system's page cache. You rarely tune it directly, but it is why the first query after a restart is slow and the second is fast, and why a database that fits in RAM behaves like a different machine from one that does not.

The read pattern: cache-aside is the default for a reason

On a read, the application checks the cache first. Hit: return it. Miss: read the database, store the answer in the cache, return it. That is cache-aside (lazy loading), and it is the default in most systems because it only caches what is actually requested — the cache fills itself with the working set and nothing else. The alternative, read-through, moves the same logic into the cache layer itself, so the application never sees a miss; same behavior, cleaner application code, and it is what managed caches offer as a feature.

The write pattern: freshness vs. speed vs. durability

Writes are where the tradeoffs live, and there are four patterns. Cache-aside (write to the database, then delete the key) is the default that survives production: the next read misses once and refills. Write-through (write to the cache and the database together, wait for both) means reads are never stale — but every write pays full database latency. Write-behind (write to the cache, flush to the database asynchronously) makes writes cache-fast, at the price that a crash before the flush loses data. Write-around (write to the database only; the cache fills on the next read) is right when most writes are never read again — a write-heavy log, say — and wasteful otherwise.

Our notebook runs the arithmetic on a read-heavy workload, 100 reads per write: cache-aside and write-through cost exactly the same in total, 100.0 ms per 100 reads plus one write either way. The latency difference is a rounding error. The real difference is the staleness window: cache-aside has one — the gap between the database write and the key delete, during which a concurrent reader gets the old value. Write-through has none. So the choice is not about speed. It is about whether a stale read is an annoyance or a bug. If it is a bug — a balance, a permission, a price — pay for write-through. If it is an annoyance — a display name, a feed ordering — take cache-aside and its simplicity.

The eviction ladder

Memory is finite. When the cache is full and a new item arrives, something must leave — and that choice can swing your hit ratio by double digits. The policies form a ladder, and each rung fixes the failure of the one below it.

FIFO — evict the oldest insertion. Nearly free, completely blind: it evicts the item that arrived first whether or not anyone still wants it. On our simulated trace (10,000 keys, Zipfian popularity, 200,000 requests, cache size 400) it manages 50.9%.
LRU — evict the item untouched for longest. The textbook default, and the right answer for maybe half of real workloads: it adapts to shifting popularity, and a hash map plus a doubly-linked list makes every operation O(1). Same trace, same size: 55.7%. But it has a known failure — scan resistance: one big sequential read (a backup, a full-table scan, a crawler) walks through the whole cache and flushes everything useful, because every scanned item looks "recently used" for exactly one moment.
LFU — evict the item with the fewest hits. It keeps long-term favourites and laughs at scans, since a scanned item has a count of 1. On our trace it wins outright: 63.7%. Its failure is the mirror image of LRU's: popularity that shifts — yesterday's celebrity is today's nobody — leaves LFU clinging to stale counts until they decay. Production LFU always adds aging or decay (Redis's LFU decays counts on a timer for exactly this reason).
S3-FIFO / SIEVE / TinyLFU — the modern rung, and the interesting one. The insight, from the S3-FIFO paper (Yang & Yue, SOSP 2023), is that a large fraction of objects in production cache traces are one-hit wonders: requested once and never reused while resident. LRU wastes its bookkeeping promoting those; S3-FIFO filters them first. Three FIFO queues: a small probationary queue (~10% of capacity) that admits everything and evicts the un-reused in one pass; a main queue (~90%) that protects anything that proves itself by being hit twice; and a ghost queue that remembers only the keys of evicted items, so a re-requested key re-enters the main queue directly. No linked-list touch on a hit, no per-item frequency counters — just three FIFO queues and a bit of metadata. Across 6,594 traces from 14 datasets it achieved a lower miss ratio than each of 12 state-of-the-art algorithms in the comparison, cutting LRU's miss ratio by up to 72% on some traces, and matched LRU's hit ratio with roughly 46% less memory. It is now in production at Google, VMware, and Redpanda. SIEVE (NSDI 2024) distills the same idea to one FIFO queue, one visited bit per object, and a moving hand — and cut ARC's miss ratio by up to 63.2% on skewed web workloads. TinyLFU (2017), the admission policy inside Caffeine and Ristretto, takes the other route: a small Count-Min Sketch estimates each key's frequency before it enters the cache, and rejects the one-hit wonders at the door.

Our own simulation tells the honest version of the same story. On a stable-popularity Zipfian trace at cache size 400, S3-FIFO scores 60.4% — 4.8 points above LRU's 55.7%, and still 3.3 points below LFU's 63.7%, because LFU is near-optimal when popularity never shifts. That is the correct reading: the modern policies do not beat LFU on a friendly trace. They beat LRU and LFU on hostile ones — scans, churn, one-hit wonders — which is what production traffic actually is. The 5–15 point gap between the right policy and LRU is the difference between a database that idles and one that melts, and it costs nothing but the choice.

The state of the art: what changed in 2025–2026

Three shifts, in order of importance.

1. The eviction research went mainstream. S3-FIFO and SIEVE were 2023–24 papers; by 2026 they are the default answer to "which eviction policy?" in the same way LRU was for the previous twenty years. The reason is not the hit-ratio charts — it is the throughput: no linked-list touch on a hit means no cache-line bouncing between cores, so the same cache serves more requests per second on many-core machines. And the lesson has crossed into LLM serving: a 2026 study of KV-cache eviction (WAKMan) found no single policy wins across traffic mixes — LRU beat S3-FIFO by 12% on agentic-heavy traffic and trailed it by up to 22% on other mixes — and a workload-aware selector that switches policies on live traffic improved hit rate by ~5 points over LRU on average. The policy is becoming adaptive, not fixed.

2. The in-memory store split in two, and the fork is winning on speed. Redis changed its license in 2024; the community fork Valkey is now the default in-memory store for new projects at most cloud providers. The numbers, from Percona's 2025 benchmarks and independent runs: Valkey 8.1 (April 2025) serves ~1.20M ops/s against Redis OSS 7.4's ~1.11M on the same AWS hardware — an 8% edge — with p99 latency 1.8 ms against 2.3 ms and ~20% less memory from a redesigned hash table. Valkey 9.1 (May 2026) is the current stable line, and Redis 8.x has answered with its own work — native vector sets, per-field hash TTL, AGPLv3 as a third license option — so the gap is narrow and both are healthy. The practical takeaway is not which one is 8% faster. It is that the choice of store is now a real decision again, and that the multi-threaded I/O path Valkey built (parallel command execution on non-overlapping keyspaces) is where the next throughput gains live.

3. The TTL got smarter. A fixed TTL is a cliff: every copy of a value expires at the same instant, and the traffic that was spread across a minute of cache hits lands on the database in one second. The defences are old but underused, and they stack: TTL jitter (a random ±10% on each expiry, so expirations spread out), stale-while-revalidate (RFC 5861: serve the expired value immediately and refresh it in the background, so no user ever waits for a recompute), request coalescing (Go's singleflight, Caffeine's LoadingCache: if a load for a key is already in flight, wait for it instead of starting another), and probabilistic early expiration — XFetch, from Vattani, Chierichetti & Lowenstein (VLDB 2015), which refreshes a value before it expires with a probability that rises as expiry approaches, scaled by how long the recompute takes. The paper proves the expected stampede size stays below a constant that does not depend on the request rate. Our simulation replays the paper's own example: a key read 10 times a second, 3 seconds to recompute, expiring at t = 10 s — with no defence, 30 simultaneous database queries; with single-flight, 1; with XFetch, the refresh lands before expiry and the stampede never happens. Over ten simulated minutes, XFetch performed 11 refreshes with at most 1 recompute in flight — against 300 database queries for a hard TTL with no defence.

Why it breaks

Every caching outage is one of these six:

The stampede. A hot key expires and every request that depended on it misses at once. The recompute is expensive, the database slows, the miss window widens, more requests pile in — a feedback loop. Fix it with single-flight first (cheapest), TTL jitter second (free), stale-while-revalidate third (best user experience), XFetch when the recompute is so expensive that even the coalescing overlap hurts.
Stale data served as truth. Cache-aside's delete-after-write has a race: a read that starts before the delete and finishes after it can refill the cache with the old value, which then lives for a full TTL. Versioned keys (user:42:v7) or write-through eliminate the race instead of narrowing it.
Cache penetration. Requests for keys that do not exist — a mistyped ID, a bot enumerating URLs — miss the cache and the database on every request, because there is nothing to cache. The fix is to cache the negative: store a short-TTL tombstone for "this key does not exist," or a Bloom filter in front of the cache.
The hot key. One celebrity, one flash sale, one config blob — a single key that gets 40% of reads lands on one cache node (and, behind it, one database row). Sharding does not help; a single key lives on a single shard. The fixes are replication of the hot key across nodes, splitting it (per-region copies), or serving it from a local in-process cache with a short TTL.
Cold start. A deploy, a failover, a cache flush — and every request misses at once, which is a stampede you scheduled yourself. Warm the cache before cutting traffic over: replay recent keys, or run the new fleet at a fraction of traffic until its hit ratio recovers.
The cache that became the database. The quietest failure: the team stops writing data anywhere durable because "it's in Redis." Then a restart, an eviction storm, or a maxmemory policy drops data that was never persisted. A cache is a cache: if losing it loses the business, it is not a cache, it is an unreplicated database with no backup.

The interview angle

"Design a cache" is a top-10 system-design question, and the expected answer is a tour of exactly this post: the four layers and what each one absorbs, cache-aside with a TTL as the default read pattern, the write-pattern tradeoff (staleness, not speed), the eviction ladder with the S3-FIFO insight about one-hit wonders, and the stampede defences in order of cheapness. The follow-ups that separate good answers from great ones: how do you size it? (measure the working set — the keys that get 95% of reads — and size to that, not to the dataset); how do you invalidate? (TTL for everything, explicit delete on write, versioned keys when the race matters); what happens when the cache dies? (the database must survive the full miss rate — load-shed, circuit-break, and warm before cutover); and when is a cache the wrong answer? (when the data changes faster than it is read, or when a stale read is a correctness bug — then the answer is a faster database, not a bigger cache).

In the real world, the decision is usually not which eviction policy to write — it is which layer to buy. Browser caching and CDN edge caching are free performance you configure with headers; the application cache is Redis or Valkey with cache-aside and a TTL; the stampede defences are three lines of code (single-flight) and one config value (jitter). You write a custom eviction policy only when your trace proves LRU is costing you real database capacity — which, as the S3-FIFO paper showed, is more often than anyone expected.

Takeaways

Caching is a latency ladder: L1 at ~1 ns, RAM at ~100 ns, SSD at ~100 µs, a cache hit at ~0.5 ms, a database query at ~50 ms. Hit ratio decides which rung your users live on — 90% hits averages 5.5 ms, 99% averages 1.0 ms.
The eviction policy is worth real money: on one simulated trace at cache size 400, FIFO hits 50.9%, LRU 55.7%, S3-FIFO 60.4%, LFU 63.7% — and S3-FIFO's paper reports up to 72% lower miss ratio than LRU on production traces, because a large fraction of web objects are one-hit wonders.
S3-FIFO and SIEVE are the modern default: three FIFO queues (or one queue plus a visited bit) beat LRU's linked list by filtering one-hit wonders before they pollute the main cache — and by doing no bookkeeping on a hit, so they scale on many-core machines.
Write patterns trade staleness, not speed: cache-aside and write-through cost exactly the same on a read-heavy workload (100.0 ms per 100 reads + 1 write). Choose by whether a stale read is an annoyance or a bug.
The stampede is the failure that takes down systems that did everything else right: 10 requests/s × 3 s recompute = 30 simultaneous database queries for one expired key. Single-flight makes it 1; XFetch makes it 0 by refreshing before expiry.
It breaks on freshness and overload, not on throughput: stampedes, stale-data races, cache penetration on nonexistent keys, hot keys, cold starts — and the cache that quietly became the only copy of the data.
Try it yourself. The companion notebook replays all four simulations: the 5.5× latency curve, the eviction ladder (FIFO 50.9% → LRU 55.7% → S3-FIFO 60.4% → LFU 63.7%), the 30-to-1 stampede, the XFetch rule (11 refreshes, max 1 in flight over 10 minutes), and the write-pattern cost tie.

This is Part 2 of System Design, Plainly. Part 3 — Sharding — is next. Follow me on dev.to (@danielsamfdo) so you don't miss it.

Sources: Yang & Yue, "S3-FIFO: FIFO Queues Are All You Need for Cache Eviction" (SOSP 2023; 6,594 traces, 14 datasets; adopted at Google, VMware, Redpanda); "SIEVE: Cache Eviction with One Queue and One Bit" (NSDI 2024); Einziger et al., "TinyLFU: A Highly Efficient Cache Admission Policy" (ACM ToS 2017); Vattani, Chierichetti & Lowenstein, "Optimal Probabilistic Cache Stampede Prevention" (VLDB 2015; XFetch); RFC 5861 (stale-while-revalidate); Percona / Momento Valkey 8.1 vs Redis benchmarks (2025; third-party numbers, hardware-specific); WAKMan KV-cache eviction study (2026, abstract-level results). All simulation numbers are from the companion notebook and the diagrams in this post; toy diagrams are labelled illustrations. The one-hit-wonder prevalence is the S3-FIFO authors' qualitative finding (a large fraction of objects in production traces are requested once and never reused); the 9.3% figure in the notebook is my own milder Zipfian trace. No content about excluded vendors appears in this post.

Companion notebook: the runnable tutorial for this post — download it here (open in Colab/Jupyter).

System Design, Plainly — Part 1: Load Balancing

Daniel Sam Pete Thiyagu — Thu, 05 Nov 2026 16:58:08 +0000

One DNS name. Fifty-seven million requests a second. That is the average load Cloudflare's network absorbs — 77 million at peak, across 330+ cities, with 405 Tbps of capacity behind it. No single server on earth handles that. No single data center does either. What handles it is the most invisible piece of infrastructure in system design: the load balancer, a machine whose entire job is to decide, in microseconds, which of a thousand identical servers gets your request — and to keep deciding correctly when servers die, traffic spikes 10x in a minute, and one unlucky key gets hot.

This is Part 1 of System Design, Plainly: one concept per post, evergreen, beginner-friendly, with the concrete numbers and the failure modes. Today: load balancing — what it is, the four algorithms that matter, why the naive version breaks, and why in 2026 the load balancer is suddenly part of the AI inference engine.

The ELI5 version

Imagine a food court with 100 identical burger counters and one entrance. A host stands at the door. Every customer who walks in, the host points at a counter.

The laziest host points at counters in order — 1, 2, 3, … 100, then back to 1. That is round robin, and it works fine until one customer orders a 40-burger catering tray and blocks counter 7 for twenty minutes while the other 99 counters stand idle. The host kept dealing fairly; the work was not fair.

A smarter host glances at each counter's line and sends you to the shortest one. That is least connections, and it is nearly perfect — but now the host has to look at all 100 lines for every single customer, which is itself slow when 57 million customers show up per second.

The clever host does something cheaper: pick two counters at random, glance at just those two lines, and send you to the shorter one. That is the power of two choices, and the math says it is almost as good as checking all 100. One extra random glance buys nearly everything perfect knowledge would.

Load balancing is that host, automated, at a scale where "glance" means a hash-table lookup in under 100 nanoseconds — and where the counters are servers that crash, get deployed, and come back with different speeds.

How it works: layers, then algorithms

A request crosses up to three load-balancing layers before it reaches your code, and each layer only sees what it needs to route.

DNS + Anycast is the first, coarsest layer. One IP address is announced from hundreds of data centers at once (BGP Anycast); internet routers send you to the nearest one automatically. If a whole data center goes dark, BGP reroutes you to the next-nearest in seconds — no config change, no DNS TTL wait. This is also how a 71-million-requests-per-second DDoS attack gets absorbed: the attack is spread across the whole footprint instead of landing on one target.

Layer 4 (transport layer) sees only IP addresses and ports. It never opens the packet's contents, never terminates TLS. It hashes the connection's 5-tuple (source IP, source port, destination IP, destination port, protocol) and looks up a table to pick a backend. Because it is so dumb, it is fast: AWS's Network Load Balancer handles millions of requests per second at sub-millisecond latency, and Google's Maglev — the software load balancer behind Google Cloud — processes roughly a gigabit per second per core with a precomputed lookup table of 65,537 entries.

Layer 7 (application layer) terminates TLS, reads the HTTP request, and routes on the URL, headers, and cookies: /api/* to one fleet, /images/* to another, this user's session to the server that already holds it. It is smarter and slower — it has to do real work per request — so the typical production stack is L4 in front of L7 in front of backends, in two or more availability zones.

The algorithm ladder

Four algorithms cover nearly every real system, and they form a ladder: each one fixes the failure of the one below it.

Round robin — deal requests in order. Free, stateless, and blind. It breaks the moment requests cost different amounts: one slow request queues behind nothing while another server idles. Use it only when every request costs about the same.
Least connections — send to the server with the fewest in-flight requests. Near-perfect balance, but it costs a full scan of the fleet per request, and the "fewest" number is stale the instant you read it at high concurrency.
Power of two choices — pick two servers at random, send to the emptier one. The theorem (Mitzenmacher, 1996/2001) is the prettiest result in the field: with n balls into n bins, blind random assignment leaves the busiest bin at ~log n / log log n balls, while two choices leaves it at log log n + O(1). At a million bins that is ~5.3 versus ~2.6. Our simulation (companion notebook) makes it concrete: 1,000 balls into 1,000 bins, random leaves the busiest bin with 5, two choices leaves it with 3, perfect knowledge gets 1 — and at 10,000 requests over 100 servers, two choices lands at 102 versus random's 122 and perfect's 100. Two random peeks buy 90% of omniscience. This is why NGINX, HAProxy, and Envoy all ship it.
Consistent hashing — for stateful work. Sometimes the request has to land on the same server as last time: a cache key, a user session, a shard. Hashing the key modulo N is the obvious answer and the wrong one: add a 5th server and 80% of your keys move, because every key's hash % 5 differs from its hash % 4. Consistent hashing (Karger et al., 1997; popularized by Amazon's Dynamo, 2007) puts servers and keys on the same ring; a key belongs to the next server clockwise. Add a server and only the keys in its arc move — about the new server's fair share, 20%. Our simulation: 200,000 keys, 4 → 5 servers, modulo moved 80.0%, the ring moved 20.2%.

One catch with rings: a single position per server is a lottery. The largest arc on a 10-node ring runs about 2.9× the fair share in expectation, so one server can inherit nearly triple the keys. Our simulation measured the busiest node at 2.26× the mean with one token per node — and 0.06× for the quietest, which is nearly idle. The fix is virtual nodes: give each server 200 positions on the ring instead of one, and the spread collapses to 1.11× busiest, 0.83× quietest. Cassandra 4.0 defaults to 16 tokens per node plus deliberate placement; the principle is the same either way.

The two tricks that make it fast

Direct Server Return (DSR). The load balancer only handles the incoming packet. The server replies directly to the client, skipping the balancer. That halves the balancer's work and removes it from the latency path of every response — it is why L4 balancers can run at line rate.

Health checks, at every layer. A balancer that routes to a dead server is worse than no balancer. Every layer probes its backends — a TCP connect, an HTTP 200 on /healthz — and a failed backend is dropped from the table in seconds. Maglev rebuilds its lookup table in milliseconds when a backend dies, so only the connections that belonged to the dead machine are re-hashed; the rest are untouched.

The state of the art: what changed in 2025–2026

Three shifts, in order of importance.

1. The cloud balancers keep getting smarter about the same old job. AWS's 2025–26 updates are all about control: NLB gained QUIC passthrough and weighted target groups in November 2025, ALB gained a Target Optimizer that caps concurrent requests per target to keep tail latency down. Pricing is still metered per capacity unit (about $0.0225/hour plus $0.008 per LCU-hour for ALB) — which is why the "just add more balancers" instinct has a bill attached.

2. The research frontier is still the ring. Consistent hashing is 29 years old and still getting papers. The August 2026 Local Rendezvous Hashing paper cuts the peak-to-average load ratio from 1.28 to 1.09 across 5,000 nodes and 50 million keys, with zero excess churn on failover — by running the election among a small window of ring-local neighbors instead of probing scattered memory. The lesson for interviews: consistent hashing is not a solved problem, it is a live tradeoff between balance, churn, and lookup cost.

3. The load balancer became part of the AI inference engine. This is the genuinely new part. An LLM request is stateful: the first turn of a chat builds a KV cache — thousands of tokens of prefill compute — and every follow-up turn reuses it only if it lands on the same worker. A round-robin balancer sends turn 6 to a random worker, the cache misses, and you pay the full prefill again. So the inference world built its own balancers: SGLang's cache-aware router scores each worker by prefix-match length and queue depth, claiming up to 3.8× the cache-hit rate of round-robin; the vLLM Router (Rust, December 2025) routes by session and understands prefill/decode disaggregation, reporting 25% higher throughput than llm-d and 2× over Kubernetes' native balancer in its own benchmark. Our toy simulation shows the same shape: 200 eight-turn sessions over 20 workers, round-robin hits the cache 5% of the time and burns 6.12M prefill tokens; session affinity hits 100% and burns 0.80M — 7.7× less recompute. And a June 2026 paper, RouteBalance, goes one step further and fuses model selection with instance selection in a single decision, Pareto-dominating 231 of 246 baseline configurations and cutting serving cost 26–37%. The load balancer is no longer just plumbing; for LLM serving it is a scheduling decision that costs real GPU money.

Why it breaks

Every load-balancing outage I have read about is one of these six:

The hot key. Consistent hashing is fair on average and useless against skew: if one celebrity's profile gets 40% of reads, one server gets 40% of reads. The fix is bounded loads (cap each node at (1+ε)× the average and spill the overflow clockwise) or splitting the hot key — never "add more servers," which does nothing for a single key.
Slow start. A fresh server joins the pool, the balancer floods it at full rate, and it melts before its caches warm. Good balancers ramp new backends up over tens of seconds. AWS's Target Optimizer exists partly for this.
Sticky sessions that stick too well. Session affinity is a cache optimization until a server dies and every one of its sessions re-homes at once — a thundering herd onto the survivors. Cap affinity lifetime; prefer consistent hashing over cookie pinning.
Health checks that lie. A backend that returns 200 on /healthz while its database connection pool is exhausted will receive traffic it cannot serve. Probe the thing that matters (a real query), and fail closed.
Retry storms. A client that retries a timed-out request 3× turns one outage into 4× load on the survivors — which then time out, which triggers more retries. This is the cascade that takes down whole fleets; the fixes are timeouts, backoff with jitter, and circuit breakers, all above the balancer.
Cross-zone traffic. Balancers that ignore zones will happily send a request across the continent to a slightly emptier server, paying latency and egress for a balance gain nobody measures. Keep traffic local unless a zone is actually failing.

The interview angle

"Design a load balancer" is a top-10 system-design question, and the expected answer is a tour of exactly this post: DNS/Anycast → L4 → L7, the algorithm ladder with the power-of-two-choices theorem, consistent hashing with virtual nodes for stateful routing, health checks, and DSR. The follow-ups that separate good answers from great ones: how do you handle a hot key? (bounded loads), what happens when a backend dies mid-request? (retry once, elsewhere, with an idempotency key), how do you add capacity without moving 80% of the keys? (the ring), and what does your balancer do that a plain DNS round robin cannot? (health checks and state).

In the real world, the decision is usually not which algorithm to write — it is which layer to buy. One static IP per AZ and millions of requests per second is a Network Load Balancer; path-based routing and TLS termination is an Application Load Balancer or Envoy; global failover and DDoS absorption is Anycast at the edge. You write a custom balancer only when the routing decision needs application state — which, as the LLM world just discovered, is exactly when the interesting work starts.

Takeaways

Load balancing is three layers, not one: DNS/Anycast picks the region, L4 picks a machine without reading the packet, L7 routes on the URL. Each layer only sees what it needs.
Power of two choices is the best deal in distributed systems: one extra random peek takes you from 122 to 102 requests on the busiest server (perfect: 100), and the theorem says the gap grows with scale.
Consistent hashing exists because modulo is a trap: adding a 5th server moves 80% of keys under modulo, ~20% on a ring — and virtual nodes (200 per server) tame the ring's own imbalance from 2.26× to 1.11×.
Direct Server Return and health checks are what make balancers fast and safe: the reply skips the balancer, and dead backends leave the table in seconds.
The 2026 twist: for LLM serving, the load balancer is a cache decision. Round-robin re-pays the prefill bill (6.12M tokens in our toy); session affinity pays it once (0.80M). SGLang and vLLM now ship their own routers.
It breaks on skew, not on averages: hot keys, slow starts, lying health checks, and retry storms — not raw throughput — are what take balancers down.
Try it yourself. The companion notebook replays all four simulations: the ladder (random 5 vs P2C 3 vs perfect 1 at m = n = 1,000), the 80%-vs-20% scale-out, the vnode collapse, and the 7.7× prefill saving.

This is Part 1 of System Design, Plainly. Part 2 — Caching — is next. Follow me on dev.to (@danielsamfdo) so you don't miss it.

Sources: Karger et al., "Consistent Hashing and Random Trees" (1997); Amazon Dynamo (2007); Eisenbud et al., "Maglev: A Fast and Reliable Software Network Load Balancer" (NSDI 2016); Mitzenmacher, "The Power of Two Choices in Randomized Load Balancing" (2001); AWS Elastic Load Balancing docs (NLB: millions of requests/second; 2025–26 feature updates); Cloudflare security architecture (57M HTTP req/s average, 77M peak, June 2024 figures; 71M req/s DDoS attack mitigated); SGLang / vLLM Router blogs (cache-aware routing claims are the vendors' own benchmarks, not independently replicated); RouteBalance (arXiv 2606.17949, June 2026); Local Rendezvous Hashing (arXiv 2512.23434, August 2026). All simulation numbers are from the companion notebook and the diagrams in this post; toy diagrams are labelled illustrations. No content about excluded vendors appears in this post.

Companion notebook: the runnable tutorial for this post — download it here (open in Colab/Jupyter).

109 Million Beats 72 Billion: LeCun's Theorem Says Every Model Sees the Same World

Daniel Sam Pete Thiyagu — Wed, 04 Nov 2026 16:56:04 +0000

On October 29, 2026, Randall Balestriero (Brown University) and Yann LeCun (New York University / AMI Labs) posted a paper that turns a two-year-old conjecture into a theorem — and then embarrasses the scale-everything playbook with a 109-million-parameter model. Their good-view text JEPA scored 0.184 on a cross-modal alignment benchmark, ahead of every language model they tested up to 72 billion parameters, including Qwen2.5-72B at 0.154 and CLIP's text encoder at 0.124. The small model is 660 times smaller than the 72B model it beat (72,000 / 109 = 660.6). It also reached the same alignment as next-token prediction with 42 times fewer training samples: 21.6 million versus 900 million.

The paper is "Proving and Improving the Platonic Representation Hypothesis with JEPAs" (arXiv 2610.39302). The Platonic Representation Hypothesis, proposed at ICML 2024, observed that image models and language models seem to converge on the same internal picture of the world as they scale — a photo of a cat and a sentence about the same cat point at the same thing. Until now that was an empirical curve with no proof and no recipe. This paper supplies both, and the recipe is the interesting part: it says next-token prediction — the training objective behind nearly every language model — is the wrong way to learn a picture of the world.

This post explains the theorem in plain language, what "good views" are, why the 109M result is real but narrow, and where the argument stops applying.

The ELI5 version

Imagine two friends who have never met, each describing the same room. One walked through it; the other only read a description of it. If both descriptions are faithful, you can line them up: "the thing by the window" in one account is "the chair" in the other. They used different words, different coordinate systems — but they describe one room, so a single rotation of one map onto the other makes them agree.

Now imagine a third friend who only ever memorized one description word for word, and practiced predicting the next word of that description. Ask them about the room and they can recite beautifully — but their map is a map of the description, not of the room. Change the wording and they get lost, because nothing in their training ever forced them to separate the room from the words about it.

That is the paper's distinction in one paragraph. A JEPA (joint-embedding predictive architecture) learns by pulling together embeddings of inputs it declares to be views of the same thing. If those pairs are two genuine observations of one scene — a photo and its caption, two camera angles — the model is forced to learn the scene, because the scene is the only thing the two views share. The authors call these good views. If the pairs are just corruptions of a single observation — the same sentence with words masked, the same image cropped — the model can cheat by memorizing the observation itself. Those are bad views. And next-token prediction, the paper shows formally, is a bad-view JEPA: its "pairs" are two parts of the same document, not two observations of one scene.

How it works: one theorem, one rotation

The Platonic Representation Theorem, as the authors state it, works in an idealized setting: a finite world, where each observation is sufficient to determine the world state it came from, and where training reaches the population optimum with a unique minimizer up to rotation. Under those conditions, JEPAs trained separately on any number of modalities — images, text, video, audio, actions — learn the same representation, up to one orthogonal rotation each.

"Up to a rotation" sounds like a technicality. It is the whole point. A rotation is just a change of coordinates: north-up versus east-up on two maps of the same city. If two models differ only by a rotation, they have learned the same geometry with different labels on the axes. Find the rotation — a standard Procrustes alignment, the same math used to align protein structures — and the two models coincide. The companion notebook checks this on a toy: two embeddings differing by a random rotation recover each other with relative error around 10^-15.

Three steps make the argument bite:

1. JEPAs learn the spectral embedding of their positive pairs. With a whitened second-order anti-collapse loss, what a JEPA ends up representing is determined by which inputs it declares to be "the same thing." The pairs are the specification; everything else is implementation detail.
2. Good pairs specify the world; bad pairs specify the modality. If positives are two observations of one scene, the only thing they share is the scene, so the learned embedding belongs to the world. If positives are corruptions of one observation, what they share includes all the modality's own quirks — wording, pixel statistics, sensor noise — so the embedding belongs to the modality. The toy in the notebook shows the mechanism: averaging two independent observations washes out modality noise (world share of variance 0.67), while averaging a view with its own corruption keeps the shared noise (0.50).
3. Next-token prediction is a bad-view JEPA. This is the paper's most provocative formal step. An autoregressive model's positive pairs are a prefix and its continuation — two parts of one document. Nothing in that pairing ever points at a scene outside the document. So the representation it learns is, by the theorem's logic, a representation of text, not of the world text describes. That reframes the last five years of scaling: we have been scaling a bad-view objective and watching world-like structure emerge despite it, not because of it.

The state of the art: the numbers, and what they do not say

The empirical case runs from 145,000 Flickr captions up to the texts of 940 million web images, each paired with its original alt text plus descriptions generated by a vision-language model. Four results carry the story:

42x fewer samples. On identical texts with an identical encoder, a good-view JEPA reached 0.109 alignment after 21.6 million samples; next-token prediction needed 900 million samples to reach 0.111. That is 900 / 21.6 = 41.7x, and the paper also reports 14x fewer text tokens and 435x fewer unique images for the same alignment. Sample efficiency, not parameter count, is where the good-view advantage is largest.
It scales with data. A 33M-parameter text JEPA scored 0.034 on Flickr captions alone, but 0.125 when trained on the texts of 50 million web images — level with much larger language models. The 109M model on 940 million images' texts reached 0.184. The ladder is smooth: more genuine pairs, better world representation.
It beats 72B on alignment. The 0.184 score sits above Qwen2.5-72B (0.154) and CLIP's text encoder (0.124) on the original hypothesis paper's benchmark against a frozen vision-encoder reference. The gap to the 72B model is +0.030 absolute, +19.5% relative.
It wins pair-free, too. On a newer benchmark that needs no paired image-text data, the 109M JEPA reached 64.4% top-5 accuracy on ImageNet-100, above all 21 vision-language pairs the benchmark reported (at most 56%). Rerun under identical conditions, the benchmark's strongest model reached 47.6% — a 16.8-point gap. The authors also note next-token models failed in half of their runs on the same texts.

Read the headline with its scope attached. The 0.184 is an alignment score, not a capability score. Nobody is claiming the 109M JEPA can chat, code, or reason better than a 72B language model — it was not evaluated on those tasks, and a representation that aligns with vision is not a chatbot. What the result does claim is narrower and, in a way, more threatening to the status quo: on the specific job of learning a world representation from text, the standard objective wastes roughly 40x the data, and a small model with the right pairs beats a giant with the wrong ones.

The catch: idealized theorems, paired data, and one metric

Four caveats, stated plainly:

The theorem is idealized. Finite world, sufficient observations, population optima, a unique minimizer up to rotation (proved under a spectral gap). Real training has none of these cleanly: finite samples, optimization error, non-unique minima. The theorem tells you what the objective wants; it does not guarantee what gradient descent gets. The gap between the two is where most of deep learning lives.
Good views need paired observations of the same scene — and text often has none. A photo and its caption are a natural pair. Two paragraphs about the same event are harder to certify as "the same scene," and most raw text has no second observation at all. The paper's large-scale corpus was built by pairing image texts with generated descriptions — a construction step that itself costs a vision-language model. If good views are expensive to manufacture, the 42x sample saving has to be netted against the cost of building the pairs.
The empirical case is one paper's metrics. The alignment benchmark, the pair-free benchmark, and the sample-efficiency comparison all come from the authors' own evaluation choices. Independent replication has not happened yet — the paper is a preprint from October 29, 2026, not peer-reviewed work. Treat 0.184 as a claim with a clear address (arXiv 2610.39302), not as a settled fact.
Alignment is not intelligence. A model whose text embeddings rotate-align with vision embeddings has learned a shared geometry. It has not thereby learned to plan, to reason, or to act. The paper's contribution is to representation learning; whether good-view representations transfer to downstream reasoning as well as next-token representations do is the open experiment this paper invites, not one it settles.

None of that cancels the result. It locates it: this is a theorem about what training objectives specify, with an experiment showing the specification matters at 40x data efficiency and 660x parameter scale on the metric the theorem governs. If the next generation of world models trains on good views instead of next tokens, the data bill — not just the parameter bill — is what collapses. That is the bet LeCun has been making for years, and this paper is the first version of it with a proof attached.

Takeaways

The Platonic Representation Hypothesis is now a theorem (for JEPAs): separately trained models learn the same representation up to one rotation, provided their positive pairs are two observations of the same scene — good views.
109M beats 72B on alignment: a good-view text JEPA scored 0.184 versus Qwen2.5-72B's 0.154 and CLIP text's 0.124 — a 660x parameter gap, +19.5% relative over the 72B model.
42x fewer samples: good views reached 0.109 alignment at 21.6M samples; next-token prediction needed 900M for 0.111. The headline efficiency win is data, not just parameters.
Next-token prediction is formally a bad-view JEPA: its pairs are parts of one document, so it learns a representation of text, not of the world text describes.
Scope is everything: alignment is not chat, coding, or reasoning; the theorem assumes idealized conditions; the paper is an October 29, 2026 preprint, not yet peer-reviewed or independently replicated.
Check it yourself. The companion notebook recomputes the headline ratios (660.6x, 41.7x, 16.8 points), verifies the rotation claim on a toy (error ~10^-15), and shows why good views wash out modality noise while bad views keep it.

Sources: Balestriero & LeCun, "Proving and Improving the Platonic Representation Hypothesis with JEPAs" (arXiv 2610.39302, October 29, 2026); the original Platonic Representation Hypothesis paper (ICML 2024). All benchmark figures are the authors' reported results, not independently replicated; toy diagrams and notebook demos are labelled illustrations. No content about excluded vendors appears in this post.

Companion notebook: the runnable tutorial for this post — download it here (open in Colab/Jupyter).

10 Hours to 3: How KAIST's CluSTER Picks the 31% of Data That Actually Teaches

Daniel Sam Pete Thiyagu — Tue, 03 Nov 2026 16:55:07 +0000

On November 2, 2026, KAIST announced a fine-tuning method called CluSTER that cut a 10-hour training run to about 3 hours — a 69.6% time saving — while keeping measured performance nearly unchanged. The trick was not a faster chip, a bigger cluster, or a smaller model. It was choosing better data: the models kept their scores while training on only about 31% of the full dataset.

CluSTER — Clustering-Based Balanced Sampling and Allocation with Data Parallelism for High-Performance Fine-Tuning — was presented in October 2026 at EMNLP 2026 (paper: arXiv 2609.12584, first author Hyunjin Kim, corresponding author Professor Jae-Gil Lee, KAIST School of Computing). The team tested it on models of about 7 billion and 13 billion parameters, on three datasets from coding and medicine, across eight benchmarks, mostly on four GPUs with an eight-GPU test as well.

This post explains what CluSTER actually does, why grouping by topic is the wrong grouping, and where the headline number stops applying.

The ELI5 version

Imagine studying for an exam with a 1,000-question practice bank. Half the questions are the same skill wearing different clothes: solve for x, but with different numbers, different names, different stories. Doing all 500 teaches you that skill once and then wastes your evening repeating it.

Now imagine a study partner who first sorts the bank by what each question teaches you, picks a few from every pile, and hands you a short set that still covers every skill. You finish in a third of the time and score about the same. That study partner is CluSTER. The "piles" are clusters of examples with similar effects on learning, the "short set" is a balanced sample across those clusters, and the "handing out" is distributing diverse batches evenly across GPUs so no chip wastes its time re-learning the same lesson.

The key insight is embarrassingly simple once stated: fine-tuning time is mostly spent on redundant examples. If your dataset teaches the same skill fifty ways, forty-nine of those ways are nearly free to drop — provided you keep the one that teaches it best, and provided you don't accidentally drop an entire skill.

How it works: influence, cluster, sample, allocate

CluSTER has four steps, and the order matters:

1. Score each example by its effect on learning. This is the step that separates CluSTER from ordinary data filtering. Instead of asking "what topic is this example about?" it asks "how does this example move the model?" Two examples can use different words, different domains, even different languages, and still push the model in the same direction. Those are duplicates in the sense that matters — learning effect — even though no deduplication tool based on text similarity would ever catch them.
2. Cluster examples with similar learning effects. Examples that teach the same thing land in the same cluster. A cluster is, in effect, one lesson the dataset is trying to teach, however many surface forms it comes in.
3. Sample balanced across clusters. Instead of taking the first 31% of the data (which would over-represent whatever skill dominates the dataset) or a random 31% (which can still miss rare skills), CluSTER picks representative examples from every cluster. The result keeps the full skill coverage of the dataset at a fraction of its size. In the reported medical question-answering experiment, that fraction was about 31%.
4. Allocate balanced across GPUs. Data-parallel training splits examples across GPUs, and random splitting can hand one GPU a batch of near-duplicates while another gets the variety. CluSTER distributes clusters evenly, so each GPU's batch is diverse. That matters because a GPU chewing through redundant examples is not just slow — it is computing gradients that mostly cancel or repeat, and the synchronization step at the end of each round waits for the slowest, most redundant worker.

The state of the art: why this lands now

Data selection for fine-tuning is not new — practitioners have long filtered by quality, deduplicated near-identical text, and upsampled rare categories. CluSTER's contribution is putting the learning effect at the center and treating allocation as part of the method, not an afterthought.

Three things make the timing right. First, fine-tuning is now the dominant way organizations adapt models: few teams train from scratch, but nearly everyone fine-tunes a base model on domain data — coding assistants, medical retrieval, internal support. A 69.6% cut in that recurring cost compounds across every adaptation run. Second, GPU time is the binding constraint for most of those teams, not data scarcity: they have more data than they can afford to train on, which is exactly the regime where dropping redundant examples pays. Third, the method improves existing infrastructure rather than demanding new hardware — the KAIST framing is that it makes the GPUs you already have more productive, which is a much easier sell than a new cluster.

The numbers to carry: 69.6% is the maximum reported saving, not the average; 31% is the data fraction in the medical QA experiment, not a universal constant; the models were 7B and 13B, not frontier-scale; and the evaluation was eight benchmarks on coding and medical data. Treat 10 hours → 3 hours as the paper's illustration of the percentage (10 × 0.304 = 3.04), not a promise about your workload. The speedup implied by 69.6% is 3.29× — 1 / (1 − 0.696) — which is the right way to compare it against any other efficiency claim you see this week.

The catch: selection is not free, and 31% is not a law

Four caveats, stated plainly:

Scoring every example costs compute. Computing a learning-effect score for each training example — typically via gradient or influence-style signals — is itself a pass over the data. The 69.6% saving is net of whatever the paper's pipeline cost on its workloads, but on your workload the arithmetic is: net saving = 6.96 hours − selection cost, on a 10-hour run. If selection is expensive and your run is short, the method can lose. Always measure end-to-end, including selection, before cutting data.
"Nearly the same performance" is benchmark-scoped. Eight benchmarks on coding and medical tasks do not cover everything a fine-tuned model will meet. Rare skills are precisely the ones a balanced sample is designed to protect — but a cluster the scorer misjudges is a skill silently dropped. For safety-critical domains, a benchmark pass is not a deployment clearance.
The fraction will vary. A dataset with little redundancy (carefully curated, diverse by construction) has less to gain; a bloated, web-scraped dataset has more. If your data is already deduplicated and balanced, expect a smaller win. The 31% figure is a result on KAIST's datasets, not a dial setting.
This is fine-tuning, not pretraining. CluSTER targets adapting an existing model to a domain. Whether influence-clustered sampling scales to pretraining — where the "skills" are far more numerous and the data far messier — is an open question the paper does not answer.

None of that diminishes the result. It reframes it: CluSTER is not "throw away 69% of your data." It is "measure what each example teaches, keep one good teacher per lesson, and stop paying GPUs to re-learn the same lesson." That is a discipline most fine-tuning pipelines currently lack — they treat data as fuel to burn, not as lessons to curate.

Takeaways

CluSTER (KAIST, EMNLP 2026, arXiv 2609.12584) cuts fine-tuning time by up to 69.6% — 10 hours to about 3 — while keeping performance nearly unchanged, by training on about 31% of the data.
The mechanism is four steps: score examples by learning effect, cluster by similar effect, sample balanced across clusters, allocate balanced across GPUs. The allocation step is part of the method, not plumbing.
Group by learning effect, not topic. Two examples with different words can teach the same skill; text-similarity deduplication misses that redundancy entirely.
69.6% is a maximum, 31% is one experiment's fraction, 7B/13B on coding and medical data is the tested regime. The implied speedup is 3.29×. Re-measure on your own workload, including the cost of scoring.
Selection cost is the break-even. Net saving = time saved − selection cost. Short runs and already-curated data gain less; bloated datasets gain more.
Check it yourself. The companion notebook recomputes the headline arithmetic (3.04 hours, 3.29× speedup), demos stratified sampling that keeps all five toy skills at 31%, and shows balanced GPU dealing — plus the break-even calculation.

Sources: KAIST news release (Nov 2, 2026), Knowridge coverage (Nov 3, 2026), and the paper "Clustering-Based Balanced Sampling and Allocation with Data Parallelism for High-Performance Fine-Tuning" (EMNLP 2026 main.342, arXiv 2609.12584). All headline figures are the authors' reported results, not independently replicated; toy diagrams are labelled as illustrations.

Companion notebook: the runnable tutorial for this post — download it here (open in Colab/Jupyter).