The thundering herd problem happens when a highly cached data item expires, forcing millions of concurrent requests to bypass the cache and slam your database at once. You can solve this by implementing cache locking (or single-flighting), which permits only the first request to rebuild the cache while holding the others in queue.
When I browse Reddit or a major news platform and suddenly get a 504 Gateway Timeout, my mind immediately jumps to system architecture. It is often not a coordinated DDoS attack causing the outage, but rather a self-inflicted system failure: the thundering herd problem. Let's look at how this failure mode works and how I approach fixing it.
What is the thundering herd problem?
The thundering herd problem occurs when a heavily requested cache key expires, forcing all incoming traffic to query your database simultaneously. Instead of reading fast, temporary data from memory, millions of concurrent reads bypass the cache and hit the persistent storage layer all at once.
I visualize this using a physical analogy. Imagine a stadium concourse as your cache and the database as a single turnstile gate. While the concourse is open, thousands of visitors move freely. But the moment that cache key expires, it is as if the main gate vanishes.
The entire stadium crowd tries to squeeze through that single turnstile at the exact same millisecond. The database's CPU spikes to 100%, and it stops accepting any connections—taking the whole application down.
Why does cache expiration cause database failure?
When a cache key expires, the application registers a cache miss and routes the query to the database to rebuild the cache. If thousands of requests arrive during the fraction of a second it takes to fetch and write that data back, they will all hit the database at the same time.
I always remind engineers that caches like Redis or Memcached serve data in microseconds because they operate in memory. Databases, being the disk-backed source of truth, take milliseconds to process queries.
If you have a high-traffic microservice, even a tiny 50-millisecond window where the cache is empty is plenty of time for thousands of concurrent requests to pile up. The database gets overwhelmed trying to run the exact same heavy query thousands of times concurrently, resulting in connection pool exhaustion.
How do you solve the thundering herd problem?
You solve this by implementing cache locking, which is also known as mutex locking or the single-flight pattern. When a cache miss occurs, the system locks that key so only the first request queries the database, while forcing all subsequent concurrent requests to wait until the cache is repopulated.
Instead of letting every request hit the database, I establish a gatekeeper. The first request that encounters the expired cache grabs a lock, queries the database, writes the fresh data to the cache, and releases the lock.
Meanwhile, the other thousands of pending requests wait for a few milliseconds. Once the lock is released, they are served directly from the newly updated cache without ever touching the database.
Which tools support thundering herd protection?
Many modern reverse proxies and caching layers have built-in directives designed to serialize cache-miss requests automatically. Configuring these native features is usually much safer and simpler than trying to write your own custom distributed locking mechanisms.
| Technology | Mechanism | Configuration Strategy |
|---|---|---|
| NGINX |
proxy_cache_use_stale / proxy_cache_lock
|
Serves stale cache data to incoming requests while a single backend request updates the cache. |
| Redis | Distributed Locks (Redlock) | Allows application nodes to synchronize cache population using lightweight keys. |
| Go (Singleflight) |
sync.Mutex group |
Merges duplicate concurrent requests into a single execution shareable by all callers. |
FAQ
Is the thundering herd problem the same as a cache stampede?
Yes, "cache stampede" and "thundering herd" are different names for the same architectural failure mode where expired keys trigger an unmanageable wave of database queries.
Can we prevent this by using longer cache TTLs?
No, increasing the TTL only reduces how often the event occurs; it does not solve the underlying vulnerability. When the key eventually expires under load, the system will still crash unless you have locking or background updates in place.
What is background cache regeneration?
It is an alternative strategy where background workers or cron jobs proactively update cache keys before they expire. Active traffic always reads from the warm cache and never triggers a synchronous database query on a miss.
Top comments (0)