Facebook had 1000+ Memcached servers.
They added one more.
Billions of cache entries got invalidated at once. Thundering herd. Database crushed. Partial outage.
They switched to consistent hashing. Problem solved.
Here's the thing: every time you say "let's distribute data across N servers" in a system design interview, the interviewer is waiting for you to explain how.
If your answer is hash % N, you've already failed the question. This post is why, and what the actual fix looks like.
The naive approach
Setup: 4 cache servers.
Key "user:123" → hash("user:123") % 4 = 2 → goes to Server 2.
Simple. Works fine. Until you scale.
Now add a 5th server:
hash("user:123") % 5 = 3 → goes to Server 3.
Same key. Different server. The data didn't move, but your lookup logic now points at the wrong place — so effectively, it has to move.
The math that breaks production
When you go from N to N+1 servers, roughly N/(N+1) of all your keys remap to a different server.
For N = 100, that's 99% of your keys moving for the crime of adding a single server.
Every one of those keys is now a cache miss. All of them hit the origin server at once. That's the thundering herd — and it's exactly what took Facebook's Memcached layer down.
Consistent hashing brings that number from ~100% down to roughly 1/N. Adding a server only displaces a small, predictable slice of your keyspace instead of nearly all of it.
The actual idea
Instead of hash % N, you place both your servers and your keys on a circular hash space — 0 to 2³² − 1. This is the "hash ring" you'll hear mentioned in basically every system design interview that touches sharding, load balancing, or distributed caching.
Once servers and keys share that ring, adding or removing a node only affects the keys sitting in its immediate neighborhood — not the whole dataset.
I go through the full ring mechanics, virtual nodes, and how this shows up in real interview questions (Redis Cluster, DynamoDB, CDN routing) in the complete writeup here 👉 Consistent Hashing — full breakdown
If you're prepping for system design interviews, this is one of those concepts that separates a pass from a "we'll get back to you."
What's your go-to explanation for consistent hashing when it comes up in an interview? Curious how others build the intuition.
Top comments (0)