Caching is supposed to make systems faster. Here's the surprising part: a cache can sometimes make a system slower — and in extreme cases, bring it down entirely.
Picture this. Your application normally handles 10,000 requests/sec, and your database comfortably handles 500 queries/sec, because the cache is absorbing almost everything. Then one popular cache entry expires. Suddenly all 10,000 requests miss at once, all 10,000 become database queries, and the database was never sized for that. CPU climbs, connection pools fill up, response times rise, requests start timing out — and then clients retry, which sends the database more traffic, not less.
That's how a small cache event turns into a production incident. This post covers the failure modes behind it: cache stampede, thundering herd, hot keys, cache penetration, cache avalanche, and the retry storms that make all of them worse — plus the standard defenses for each.
1. Cache stampede
Suppose product:123 has a 10-minute TTL and thousands of users are requesting it. For ten minutes, every request is a hit. Then the TTL expires, and if 5,000 requests arrive in roughly the same instant, every single one of them sees a miss and goes straight to the database.
The cache was protecting the database. Then, for one bad moment, it stopped.
2. Why this is so dangerous
Numbers make this concrete. Say normal traffic is 20,000 requests/sec with a 99% hit ratio — the database sees roughly 200 requests/sec, which is nothing.
Now one popular key expires. The same 20,000 requests/sec are still arriving, but now all of them miss — the database goes from 200 requests/sec to roughly 20,000. Nothing about the traffic changed. One cache entry going cold multiplied database load a hundred times over, and that's exactly the kind of jump that turns a healthy database into a struggling one within seconds.
3. The thundering herd problem
Thundering herd is the general name for this shape of problem: a large number of requests are waiting on the same resource, and when it becomes available (or in this case, unavailable), they all rush it simultaneously. The database isn't necessarily slow — the problem is that far too many requests are doing the exact same expensive work at the exact same moment.
4. The simple fix: let one request do the work
If a thousand requests miss the same key at once, there's no reason for a thousand identical database queries. Instead, let one request load the data while everyone else waits for that single result.
This is usually called request coalescing or single flight. It's a genuinely simple idea with an outsized effect: we're not making the database faster, we're just stopping 999 requests from doing work that's already being done.
5. Locking
One way to implement coalescing is a distributed lock: on a miss, try to acquire a lock for that key; whoever gets it queries the database, populates the cache, and releases the lock; everyone else waits, then reads the now-populated cache once it's released.
It works well, but a distributed lock brings its own list of things to get right — lock expiration, deadlocks, a process crashing while holding the lock, who owns the lock, and what retry behavior looks like for the requests waiting on it. Don't reach for distributed locking as a default; reach for it once you've confirmed you need it.
6. Randomized TTL — add some jitter
Here's a much cheaper technique for a related problem. If 100,000 cache entries were all created around the same time with a flat one-hour TTL, a lot of them expire together an hour later. Add a small random offset instead:
TTL = 60 minutes + random(0–5 minutes)
Now expirations spread across a window instead of landing on a single instant. This is TTL jitter — simple to add, and surprisingly effective at preventing synchronized expiry from becoming synchronized load.
7. Hot keys
Different problem, same root cause. Imagine 10 million cached keys — plenty of spread on paper — but 80% of traffic is for exactly one of them, say product:iphone. That single key is now a hot key, and it doesn't matter how many other keys exist; the traffic isn't evenly distributed across them.
8. Why hot keys are dangerous
If a single cache node handles 50,000 requests/sec but one key alone is getting 200,000 requests/sec, that key becomes a bottleneck regardless of how much headroom the rest of the cluster has.
This matters most in distributed caches specifically, because adding more nodes doesn't fix it — the hot key still maps to the same node it always did. You've spread the average load across the cluster; the one key that's actually the problem never moved.
The usual responses: keep the hottest data in application-local memory too, so it doesn't have to reach the distributed cache on every request; replicate the hot key across multiple nodes instead of pinning it to one; apply request coalescing specifically to that key; or, if the data doesn't change often, simply give it a longer TTL to reduce how often it needs refreshing at all.
9. Cache penetration
A different failure shape entirely. Suppose users request product:999999999 — a product that doesn't exist. Every request misses the cache, queries the database, gets NOT FOUND, and the next request does the exact same round trip. The cache isn't helping, because there's nothing valid to cache.
This is cache penetration, and it's particularly nasty when it's driven by scraping, enumeration, or a bug generating IDs that were never real to begin with.
10. Cache the "not found" result
The fix is almost too simple: cache the negative result too.
product:999999999 → NOT_FOUND
TTL = 60 seconds
Now the next request for that ID is a cache hit, even though the answer is "it doesn't exist" — no database call needed. The one thing to watch: don't cache a negative result for hours, since the object might legitimately get created shortly afterward. Negative caching generally wants a short TTL specifically because "not found today" and "will never exist" are different claims.
11. Cache avalanche
Now combine the timing problem with scale. If thousands of entries were all loaded around the same time with the same TTL, they can all expire together — a cache avalanche. It can also be triggered by a cache cluster failure, a mass invalidation, an application restart, a deployment, or a network blip.
Whatever the trigger, the shape is the same: cache failure leads to a cache-miss explosion, which leads to a database traffic explosion, which leads to database slowdown, application slowdown, timeouts, retries, and then even more traffic than before. That's a cascading failure, and it's the same underlying mechanics as a stampede — just triggered by breadth instead of one popular key.
12. The retry problem makes it worse
Here's the part that turns a bad moment into a genuine outage: when a request times out, the client retries. Retry, timeout, retry, timeout — instead of reducing load on a struggling system, the system generates more of it.
The database didn't fail once here. It failed, and then got asked to fail again, faster, by the same clients that just timed out. This is why caching problems and retry storms are dangerous specifically together — a resilient system needs bounded retries, exponential backoff, jitter on the backoff itself, timeouts, circuit breakers, and rate limits. Caching is only one part of the resilience story; the retry behavior around it is just as load-bearing.
13. Cache warming
Rather than letting the application start with a cold, empty cache, you can proactively load the data you already know will be popular. Before Black Friday, preload the popular products. Before a major product launch, ticket sale, or marketing campaign, warm the keys you already know will spike. You don't want your first million users of the day to be the ones who happen to warm your cache for you.
14. Stale-while-revalidate, again
We covered this in Part 5 as an invalidation technique — it's just as useful here as a stampede defense. Instead of deleting an expired value immediately and forcing every subsequent request to wait on the database, serve the existing value while refreshing it in the background:
User → stale value → response immediately
↓
background refresh → cache updated
This trades a small amount of freshness for a large amount of availability and latency stability. As always, whether that trade is acceptable is a business question, not a technical one.
15. Don't treat the cache as a black box
The single biggest observability mistake is only watching the hit ratio. A 99% hit ratio looks great in isolation, but it doesn't tell you what happens during the other 1%, or what happens to that number under stress.
Also watch: miss rate, eviction rate, memory usage, latency, key distribution, hot keys specifically, connection count, errors, and timeouts. Above all, watch what happens to database traffic when cache performance degrades — that's usually the single most important signal in the whole system, because it's the one that tells you whether your cache is a performance optimization or a hidden single point of failure.
16. A more useful mental model
Most people design a cache by thinking Request → Cache → Database and stop there. The more useful model asks what happens at the miss branch specifically — is there a lock, a coalescing layer, a TTL strategy, something standing between "everyone missed at once" and "everyone queries the database at once"?
The goal was never just "make the cache fast." It's "make the system behave predictably when the cache is slow, empty, overloaded, or unavailable." That reframing is worth more than any individual technique in this post.
17. Five problems worth remembering
If you keep nothing else from this post, keep these five, matched to their standard fix:
- Cache stampede — many requests miss the same key at once. Fix with request coalescing, locking, or stale-while-revalidate.
- Hot key — one key receives disproportionate traffic. Fix with local caching, replication, a longer TTL, or coalescing.
- Cache penetration — requests repeatedly ask for data that doesn't exist. Fix with negative caching, input validation, or Bloom filters at large scale.
- Cache avalanche — many keys expire or disappear together. Fix with TTL jitter, cache warming, staggered expiration, and a resilient fallback path.
- Retry storm — failures cause aggressive client retries. Fix with exponential backoff, jitter, bounded retries, circuit breakers, and rate limiting.
The bigger lesson
"Put frequently used data in memory and everything gets faster" is true — right up until it isn't. At production scale, the real questions are what happens when the cache expires, what happens when 10,000 requests miss at the same instant, what happens when one key gets disproportionately popular, what happens when the cache goes down entirely, and what happens when clients start retrying into all of that.
Those questions are what separate a cache that works in a demo from a caching architecture that survives production. And the principle underneath all of them is the same one this whole series keeps returning to: a cache should protect your database, not become another single point of failure.
Has your system ever hit one of these five in production — and which one? Stampede and retry storms tend to travel together in my experience; curious if that matches what others have seen.
Top comments (0)