Common Cache Production

Cache Stampede, Hot Keys, and Cache Penetration

Caching is one of the most common ways to improve application performance and reduce database load.

But as traffic grows, caching introduces its own problems.

Three common ones are:

• Cache Stampede
• Hot Keys
• Cache Penetration

Let's understand each one with simple examples.

---

1. Cache Stampede

A cache stampede (also called the thundering herd or dogpiling problem) happens when a popular cached value expires and many requests try to fetch the same data from the database at the same time.

Instead of protecting the database, the cache suddenly becomes the reason the database gets overloaded.

When and Why Does It Happen?

Normally, we put a cache such as Redis in front of our database:

Client
   ↓
Application
   ↓
  Cache
   ↓
 Database

Cached data usually has a Time To Live (TTL) so that it doesn't stay stale forever.

A cache stampede can happen when:

1. High Concurrency

Thousands of users request the same piece of data.

For example:

GET homepage
GET homepage
GET homepage
GET homepage
...

2. Simultaneous Expiration

The cached value reaches its TTL and expires.

Cache
product:123 → EXPIRED

3. Race Condition

The first request goes to the database to retrieve the data.

But before that request finishes and updates the cache, thousands of other requests also see a cache miss.

Request 1 ──→ Cache MISS ──→ Database
Request 2 ──→ Cache MISS ──→ Database
Request 3 ──→ Cache MISS ──→ Database
Request 4 ──→ Cache MISS ──→ Database
...
Request 10000 ─→ Cache MISS ──→ Database

Now thousands of requests are doing the same database query.

This can overload the database and potentially cause it to slow down or crash.

---

How Do We Solve a Cache Stampede?

The main goal is simple:

Don't allow thousands of requests to perform the same database work simultaneously.

There are several approaches.

1. Locking

When the cache expires, the first request acquires a lock.

Other requests wait briefly instead of querying the database.

Request 1 → Cache MISS → Acquire Lock → Database
Request 2 → Cache MISS → Wait
Request 3 → Cache MISS → Wait
Request 4 → Cache MISS → Wait

The first request retrieves the data and updates the cache:

Database
   ↓
Update Cache
   ↓
Release Lock

The waiting requests can then read the newly populated cache.

---

2. Stale-While-Revalidate

Instead of making users wait for fresh data, we temporarily serve the existing stale value.

At the same time, a background process refreshes the cache.

User Request
     ↓
Stale Cache
     ↓
Return Immediately

Background Process
     ↓
Database
     ↓
Update Cache

This works well when serving slightly outdated data is acceptable.

---

3. External Pre-Warming

Another approach is to refresh the cache before it expires.

For example, a scheduled job can periodically load popular data:

Cron Job
   ↓
Database
   ↓
Update Cache

This means user traffic doesn't have to be the thing that triggers the cache refresh.

---

2. Hot Keys

A hot key happens when a single cache key receives a very large number of requests.

For example:

product:123

Imagine we have:

• 10 application servers
• 100,000 user requests
• One Redis server

The load balancer distributes requests across the application servers:

                Load Balancer
                 /    |    \
                /     |     \
            App 1   App 2   App 3 ... App 10
               \       |       /
                \      |      /
                    Redis

The application servers can scale horizontally.

But if all requests need the same Redis key:

product:123

then all application servers send requests to Redis.

App 1 ──┐
App 2 ──┤
App 3 ──┤
App 4 ──┤
...     ├──→ Redis
App 10 ─┘

The problem is that one Redis node is receiving a huge amount of traffic for the same key.

The Redis CPU or network capacity can become the bottleneck.

---

Solution 1: L1 / L2 Cache

One of the simplest solutions is to introduce a local cache.

We can think of the caches as:

L1 → Application Server Local Memory
L2 → Redis

The application checks L1 first.

If the data isn't there, it checks Redis.

Request
   ↓
L1 Cache
   ↓
 Cache Hit? ── Yes → Return
   │
   No
   ↓
Redis
   ↓
Store in L1
   ↓
Return

Without L1 Cache

Suppose we have:

• 10 application servers
• 100,000 requests

Without L1:

100,000 requests
       ↓
100,000 Redis calls
       ↓
Redis becomes the bottleneck

With L1 Cache

Each application server can keep the hot data in local memory for a short period, for example 5 seconds.

Initially:

App 1 → Redis
App 2 → Redis
App 3 → Redis
...
App 10 → Redis

After that, each server has the value locally.

The remaining requests can be served directly from memory:

100,000 requests
       ↓
   L1 Cache
       ↓
Most requests served locally

So instead of sending every request to Redis, each application server can serve the majority of requests from its own memory.

Why Is L1 So Effective?

1. No Network Overhead

Reading from local memory is much faster than making a network call to Redis.

Local Memory → Very fast
Redis        → Network round trip

2. Application Servers Scale Horizontally

You can add more application servers behind a load balancer.

This helps distribute user traffic across multiple machines.

A single hot Redis key, however, can still concentrate traffic on one Redis node.

---

Solution 2: Key Splitting

Another approach is to split a hot key into multiple keys.

In Redis Cluster, a single key is mapped to a single shard.

So instead of:

product:123

we can create multiple copies:

product:123:1
product:123:2
product:123:3
...
product:123:100

When a request arrives, the application chooses one of the keys.

For example:

Request 1 → product:123:17
Request 2 → product:123:42
Request 3 → product:123:8
Request 4 → product:123:91

Now the traffic can be distributed across different Redis cluster nodes.

The Idea

Instead of:

100,000 requests
       ↓
product:123
       ↓
One Redis shard

we can have:

100,000 requests
       ↓
 ┌─────┼─────┐
 ↓     ↓     ↓
Key 1 Key 2 Key 3 ... Key 100

This spreads the read traffic instead of hammering a single Redis key.

---

Solution 3: Read Replicas

If you're using a Redis primary-replica architecture, another option is to add read replicas.

The basic idea is:

              Primary
             /   |   \
            /    |    \
       Replica Replica Replica

Writes go to the primary:

Application
     ↓
 Primary

Reads can be distributed across replicas:

Application
     ↓
 ┌───┼────────┐
 ↓   ↓        ↓
R1   R2       R3

This allows the system to distribute a large number of read requests across multiple Redis nodes.

---

3. Cache Penetration

Cache penetration happens when requests repeatedly ask for data that doesn't exist.

For example:

GET /users/999999

If the user doesn't exist:

Request
   ↓
Redis
   ↓
Cache MISS
   ↓
PostgreSQL
   ↓
User NOT FOUND

The problem becomes serious when thousands of requests are made for invalid IDs:

/user/999999
/user/837462
/user/123456
/user/555555
...

Every request misses the cache and reaches the database.

Request 1 → Redis MISS → PostgreSQL → NOT FOUND
Request 2 → Redis MISS → PostgreSQL → NOT FOUND
Request 3 → Redis MISS → PostgreSQL → NOT FOUND
...
Request 10000 → Redis MISS → PostgreSQL → NOT FOUND

Unlike a cache stampede, the problem here isn't that valid cached data expired.

The problem is that the requested data doesn't exist in the first place.

---

Solution: Bloom Filter

A Bloom Filter is a memory-efficient data structure that can quickly tell us whether an item might exist in a dataset.

The application can check the Bloom Filter before querying Redis or PostgreSQL.

Request
   ↓
Bloom Filter
   ↓
Could this ID exist?
   /          \
 No            Yes
 ↓              ↓
Return        Redis
Not Found       ↓
             PostgreSQL

For example, suppose we have these users:

1
2
3
4
5

The Bloom Filter knows about these IDs.

Now a request comes for:

999999

The Bloom Filter can say:

999999 → Definitely does not exist

So we can stop the request immediately without hitting Redis or PostgreSQL.

Important Bloom Filter Property

A Bloom Filter can have false positives, but it does not have false negatives.

In simple terms:

Bloom Filter → "Definitely doesn't exist"
        ↓
     Stop here

Bloom Filter → "Might exist"
        ↓
   Check Redis
        ↓
   Check Database

So if the Bloom Filter says an ID doesn't exist, we can safely reject the request.

If it says the ID might exist, we continue normally.

Why Is This Useful?

Imagine:

1,000,000 invalid requests

Without a Bloom Filter:

1,000,000 requests
        ↓
      Redis
        ↓
  PostgreSQL

With a Bloom Filter:

1,000,000 requests
        ↓
   Bloom Filter
        ↓
Most invalid requests stopped here
        ↓
Only possible valid requests
        ↓
      Redis
        ↓
   PostgreSQL

This can significantly reduce unnecessary traffic to Redis and PostgreSQL.

Bloom Filter is especially useful when the database contains a large number of IDs and the system receives many requests for IDs that don't exist.

---

Cache Stampede vs Hot Key vs Cache Penetration

These three problems are related, but they are not the same.

Problem	Main Issue	Typical Solutions
Cache Stampede	Many requests hit the database after cache expiration	Locking, stale-while-revalidate, pre-warming
Hot Key	One cache key receives too many requests	L1 cache, key splitting, read replicas
Cache Penetration	Requests repeatedly ask for data that doesn't exist	Bloom Filter, negative caching, validation

Simple Way to Remember

Cache Stampede:

The cache expires and everyone goes to the database.

Hot Key:

Everyone keeps asking Redis for the same key.

Cache Penetration:

Everyone keeps asking for data that doesn't exist.

Understanding the difference helps you choose the right solution when designing a high-scale system.