Cache Stampede: The Failure Mode Hiding Behind a 99% Hit Rate

The problem

A cache hit rate of 99% sounds like a solved problem. Most dashboards stop there. But I've watched a service with a 99.2% hit rate take its database down twice in one quarter — both times during completely unremarkable traffic. No spike, no bad deploy, no slow query anywhere else in the system. The pattern both times: a single popular cache key expired, and for about 400ms, every one of the ~3,000 requests/sec hitting that key sailed straight past the cache and landed on the database at once.

That's a cache stampede (also called a thundering herd). The 0.8% miss rate isn't spread evenly across time — it clusters at the exact moment a hot key's TTL runs out, because that's when every concurrent request checking that key misses simultaneously.

Why it happens

Caches expire keys independently, but request traffic doesn't arrive independently — it's driven by whatever's popular right now. When a key is hot enough that dozens or hundreds of requests check it within the same 50-100ms window, a normal TTL expiry doesn't produce one cache miss. It produces all of them, at once, because nothing coordinates "who's responsible for recomputing this."

Every one of those requests independently decides the same thing: not in cache, better go compute it. If "compute it" means a database query, you just turned one popular cache key into N simultaneous identical queries. If computing the value is itself expensive — a join, an aggregation, a call to a slower downstream service — the database doesn't see N requests it can absorb one at a time. It sees N copies of your worst query, simultaneously, on a key that was supposed to be protecting it from exactly this.

The 99% hit rate number hides this completely, because it's an average over the whole time window. It doesn't tell you the 1% is bursty, correlated, and concentrated on your hottest keys — the ones where a stampede does the most damage.

What to do about it

Four fixes, roughly in order of effort:

Jitter your TTLs. If every replica of a key expires at exactly the same computed time, you've synchronized your own stampede. Add randomness — ttl = base_ttl + random(0, base_ttl * 0.1) — so expiries spread out instead of landing in the same window.

Request coalescing (single-flight). When a key misses, the first request acquires a lock (in-process, or distributed via Redis SETNX) and recomputes. Every other request that misses on the same key while the recompute is in flight waits for that one result instead of triggering its own. Go's singleflight package and most modern cache client libraries have this built in — usually a one-line wrap around your fetch function, not a rewrite.

Probabilistic early recomputation. Instead of waiting for a hard expiry, let requests probabilistically recompute a key slightly before it expires, with the probability rising as the expiry approaches. One request refreshes the value early; everyone else keeps serving the still-valid cached copy. This avoids the "everyone queues behind one lock" latency spike that pure single-flight can still cause on the very first miss.

Never let a cache miss mean "block until we have a fresh value." For keys expensive enough to matter, serve a slightly stale value while a background refresh runs, instead of making the request wait on a synchronous recompute. Stale-while-revalidate turns your worst case (recompute latency, on the critical path, times N concurrent requests) into your best case (cached latency, every time).

Key takeaways

• A high average hit rate can hide a stampede — look for miss clustering on your hottest keys, not just the overall rate.
• The failure isn't the cache missing. It's every concurrent request treating that single miss as its own personal problem to solve, independently, at the same instant.
• Jittered TTLs stop you from synchronizing your own failure.
• Single-flight / request coalescing turns N simultaneous recomputes into 1.
• For expensive keys, serve stale-while-revalidate instead of blocking on a synchronous recompute — a cache miss should never be visible to the end user as added latency.