Imagine we're building an e-commerce application.
We use Redis to cache product details so that our application doesn't have to query the database every time someone opens a product page.
Everything works well until one product suddenly goes viral.
A product that normally receives 1,000 requests per second now receives 100,000 requests per second.
Our Redis cluster has multiple nodes. Some of them are barely doing any work. Yet one node is struggling to keep up.
Why?
The answer lies in a concept called hot keys.
In this article, we'll understand how Redis distributes keys, why a single key can become a bottleneck, and how we can distribute the load without simply adding more Redis nodes.
1. Understanding How Redis Cluster Distributes Keys
Before understanding hot keys, we need to understand how Redis Cluster distributes data.
Redis Cluster uses a mechanism called hash slots.
It has a fixed number of 16,384 hash slots, numbered from 0 to 16,383.
When we store a key in Redis, the cluster calculates its hash slot using the following formula:
HASH_SLOT = CRC16(key) mod 16384
For example:
product:101 → Slot 1200
product:102 → Slot 7000
product:103 → Slot 14000
The cluster then maps these slots to Redis nodes.
Imagine we have three primary nodes:
Redis Cluster
┌─────────────────────────┐
│ Node A │
│ Slots 0–5460 │
└─────────────────────────┘
┌─────────────────────────┐
│ Node B │
│ Slots 5461–10922 │
└─────────────────────────┘
┌─────────────────────────┐
│ Node C │
│ Slots 10923–16383 │
└─────────────────────────┘
Each node owns a range of slots.
But here's something important:
A slot is not a storage container with a fixed number of keys.
A single slot can contain thousands, millions, or more keys.
The slot is simply a logical partition that determines which node is responsible for a key.
A Redis node's actual storage capacity depends on its available memory and configuration.
2. What Happens When We Read a Key?
Suppose our application executes:
GET product:101
The Redis Cluster client calculates the hash slot for the key.
GET product:101
│
▼
Calculate slot
│
▼
Slot 1200
│
▼
Which node owns Slot 1200?
│
▼
Node A
│
▼
Return value
The client can use the cluster's slot-to-node mapping to send the request directly to the correct node.
If the request reaches the wrong node, Redis Cluster can respond with a MOVED redirection.
The client can then update its topology information and route subsequent requests correctly.
Notice that Redis doesn't search through all the nodes to find our key.
It calculates the slot from the key and determines the responsible node.
3. The Hot Key Problem
Now let's return to our e-commerce application.
Suppose our Redis cluster has three nodes.
Initially, traffic is distributed reasonably well:
Node A → 10,000 requests/sec
Node B → 10,000 requests/sec
Node C → 10,000 requests/sec
Suddenly, product:101 becomes extremely popular.
product:101 → 100,000 requests/sec
The key belongs to Slot 1200, which is owned by Node A.
Now the traffic looks like this:
Redis Cluster
┌─────────────────────────┐
│ Node A │
│ │
│ product:101 │
│ │
│ 100,000 requests/sec │
│ │
└─────────────────────────┘
┌─────────────────────────┐
│ Node B │
│ │
│ Normal traffic │
│ │
└─────────────────────────┘
┌─────────────────────────┐
│ Node C │
│ │
│ Normal traffic │
│ │
└─────────────────────────┘
Node A is under pressure while Node B and Node C may still have plenty of capacity.
This is a hot key.
A hot key is a key that receives a disproportionately large amount of traffic compared to other keys.
What actually breaks?
Several things can happen:
- CPU utilization increases on the hot node.
- Network bandwidth becomes a bottleneck.
- Request latency increases.
- P99 latency becomes significantly worse.
- Requests may time out.
- Application retries may amplify the traffic.
A Redis node may have enough memory to store a key but still struggle to serve millions of requests for it.
This brings us to an important distinction:
Storage capacity and request-serving capacity are two different things.
4. Why Adding More Redis Nodes Doesn't Solve It
Our first instinct might be to add more Redis nodes.
After all, if one node is overloaded, shouldn't adding more nodes distribute the traffic?
Not necessarily.
Redis Cluster distributes keys through hash slots.
Adding a new node allows us to redistribute slots across more nodes. However, a single key still belongs to one slot, and that slot has one primary owner at a time.
For example:
Before adding Node D:
Node A → product:101
Node B → product:102
Node C → product:103
After adding Node D:
Node A → product:101
Node B → product:102
Node C → product:103
Node D → Other keys
Even after adding Node D, product:101 may still belong to Node A.
We could migrate its slot to Node D, but that would only move the bottleneck.
Before:
Node A → 100,000 requests/sec
After moving the slot:
Node D → 100,000 requests/sec
The problem isn't necessarily the number of Redis nodes.
The problem is that all requests for one key are directed to the same primary.
Redis Cluster distributes keys, not individual requests for a key.
5. Can We Create Multiple Copies of the Same Key?
This is where things get interesting.
Suppose we have a hot key:
product:101
We could create multiple copies:
product:101:0
product:101:1
product:101:2
The application can distribute reads across these copies.
But there's a catch.
Creating multiple keys on the same Redis node doesn't solve the node-level bottleneck.
For example:
Node A
│
├── product:101:0
├── product:101:1
└── product:101:2
Even if requests are distributed equally across these three keys, Node A still handles all the traffic.
We have distributed requests across keys, but not across machines.
To achieve meaningful load distribution, the copies must be stored on independent Redis nodes.
Distributing the Copies Across Nodes
Now imagine we distribute the copies across three different primary nodes:
Application
│
100,000 requests/sec
│
┌───────────┼───────────┐
▼ ▼ ▼
Node A Node B Node C
Copy 0 Copy 1 Copy 2
Assuming an even distribution:
| Node | Key | Requests/sec |
|---|---|---|
| Node A | product:101:0 |
33,333 |
| Node B | product:101:1 |
33,333 |
| Node C | product:101:2 |
33,334 |
Now the traffic is distributed across three independent nodes.
This technique is called hot key replication or application-level hot-key sharding.
The application decides which copy to read from.
However, Redis Cluster doesn't automatically create or distribute these copies for us. We need to ensure the keys map to slots owned by different nodes.
6. The Consistency Problem
Distributing reads is only half the problem.
What happens when the product price changes?
Initially:
Node A → product:101:0 → ₹1,000
Node B → product:101:1 → ₹1,000
Node C → product:101:2 → ₹1,000
Now the price changes to ₹1,200.
We need to update all three copies:
Node A → ₹1,200
Node B → ₹1,200
Node C → ₹1,200
If Node A is updated first, but Node B and Node C haven't been updated yet, different users might see different prices.
This creates a consistency problem.
There are several ways to handle it:
Option 1: Update all copies
When the source data changes, update every Redis copy.
This works well when the data changes infrequently.
Option 2: Invalidate all copies
Instead of updating each copy, delete them.
The next request can repopulate the cache from the source of truth.
However, this introduces the possibility of a cache stampede if many requests arrive simultaneously.
Option 3: Use versioning
Store a version alongside the cached data.
When the product changes, increment the version and ensure that readers use the appropriate version.
This can help avoid serving outdated copies, although it introduces additional coordination.
The right approach depends on how frequently the data changes and how much staleness our application can tolerate.
For product descriptions, a little staleness might be acceptable.
For product prices, inventory, or other correctness-sensitive data, we need to be much more careful.
7. Other Ways to Handle Hot Keys
Hot-key replication is not the only solution.
Depending on the workload, we can use several approaches.
Local Application Cache
If product details rarely change, we can cache them inside the application process.
Application
┌─────────────┐
Request ───────►│ Local Cache │
└──────┬──────┘
│ Cache miss
▼
Redis
Many requests can now be served without reaching Redis.
The trade-off is that every application instance maintains its own cache, which introduces invalidation and staleness concerns.
Read Replicas
Redis replicas can serve read traffic when the application and client are configured to route reads to them.
However, Redis replication is asynchronous by default.
Replicas may serve stale data, and replication itself consumes resources.
Request Coalescing
Request coalescing is particularly useful when a hot key expires.
Imagine 10,000 requests arrive simultaneously for an expired product cache.
Without request coalescing:
10,000 requests
│
▼
10,000 cache misses
│
▼
10,000 database queries
With request coalescing:
10,000 requests
│
▼
One database query
│
▼
Cache populated
│
▼
All requests receive the result
This prevents a cache stampede.
However, request coalescing primarily addresses concurrent cache misses. It doesn't eliminate the Redis hot key problem when the key is already cached and every request still reaches Redis.
8. Hot Key vs. Cache Stampede vs. Cache Penetration
These problems are related, but they are not the same.
| Problem | Description | Common solution |
|---|---|---|
| Hot key | One existing key receives too much traffic | Local cache, replication, read replicas |
| Cache stampede | Many requests simultaneously miss an expired key | Request coalescing, locking, TTL jitter |
| Cache penetration | Requests repeatedly ask for nonexistent keys | Negative caching, Bloom filters |
| Node overload | One Redis node receives too much traffic | Better distribution, hot-key mitigation |
Understanding these distinctions helps us choose the right solution rather than applying the same caching strategy everywhere.
Conclusion
Redis Cluster is excellent at distributing data across multiple nodes.
But distributing data and distributing traffic are not always the same thing.
A cluster can have plenty of available memory and processing capacity while one node struggles because a single key receives an overwhelming amount of traffic.
Adding more nodes helps distribute different keys.
Moving a slot changes which node owns a key.
But neither approach automatically distributes the reads for one hot key across multiple primary nodes.
For that, we need to consider techniques such as application-level caching, read replicas, or hot-key replication.
The fundamental idea is simple:
When one key becomes the bottleneck, we need to distribute the reads for that key itself—not just add more nodes to the cluster.
Top comments (0)