When a hot cache key expires, every in-flight request misses at the same instant and all of them recompute the same value against the same database. That is a cache stampede, and it shows up as a latency cliff on a fixed interval rather than a gradual degradation. The fixes, in the order I would apply them: jitter your TTLs, add single-flight locking so only one worker recomputes, serve stale while revalidating in the background, and — if you need the last bit of smoothness — recompute probabilistically before expiry.
What does a cache stampede actually look like in the logs?
The tell is periodicity. Not "the database is slow under load" but "the database is fine, then pinned, then fine again, every five minutes." If your hot key TTL is 300 seconds and your p99 graph has teeth exactly 300 seconds apart, you are looking at your own TTL, not a traffic pattern.
On the application side you get a burst of timeouts against one endpoint, and the errors sit downstream of the cache rather than in it. In a Postgres-backed service the burst reads as connection pressure — pool checkout timeouts, or the server refusing connections outright:
FATAL: sorry, too many clients already
In a Go or Node service the same event surfaces as a wall of context-deadline errors that all reference the same query. Redis itself looks healthy throughout: hit rate dips for a second or two, memory is flat, INFO commandstats shows nothing unusual. That combination — healthy cache, spiking origin, fixed interval — is the fingerprint.
Log the cache-miss path with the key name and count misses per key per second. A stampede is not "many misses"; it is many misses on one key inside one recompute window.
Why doesn't adding cache capacity or raising the hit rate fix it?
Because hit rate is the wrong number. Suppose one key serves 500 requests per second and takes 200 ms to recompute. The moment it expires, roughly 100 requests arrive before the first recompute finishes, and each sees an empty slot and starts its own. Your steady-state hit rate can be 99.8% and the incident still happens, because the damage is done by concurrency inside a 200 ms window, not by the volume of misses over an hour.
This is also why a bigger Redis instance, a longer TTL, or a warm-up job on deploy do not help. A longer TTL makes stampedes rarer and worse — more accumulated traffic per expiry, and a colder recompute when it lands. Warm-up jobs are counterproductive if they populate thousands of keys in a loop with identical TTLs, since you have just synchronized all of them to expire in the same second.
The takeaway: a stampede is a concurrency-control bug in your cache-miss path, not a capacity problem.
Fix 1: Jitter every TTL, without exception
This is a one-line change and it dissolves the synchronized-expiry class of the problem entirely. Never write a fixed TTL:
import random
def ttl_with_jitter(base_seconds, spread=0.1):
# 60s base -> a value uniformly in [54, 66]
return int(base_seconds * random.uniform(1 - spread, 1 + spread))
Ten percent spread on a 60-second TTL smears a bulk warm-up across a 12-second window instead of one instant. It does nothing for a single very hot key — that key still has one expiry moment, and everyone still piles into it — which is why jitter is necessary but never sufficient.
Fix 2: Single-flight, so only one worker recomputes
Single-flight means the first request to notice the miss takes a short-lived lock and does the work; everyone else either waits for it or serves something else. In Redis the lock is SET key value NX EX seconds:
import json, time, random
import redis
r = redis.Redis(decode_responses=True)
def _store(key, value, fresh_for, keep_stale_for):
fresh = fresh_for * random.uniform(0.9, 1.1)
entry = {"value": value, "fresh_until": time.time() + fresh}
r.set(key, json.dumps(entry), ex=int(fresh + keep_stale_for))
def get_cached(key, compute, fresh_for=60, keep_stale_for=600, lock_ttl=10):
lock_key = f"lock:{key}"
raw = r.get(key)
if raw is not None:
entry = json.loads(raw)
if time.time() < entry["fresh_until"]:
return entry["value"]
# Stale but usable: one caller refreshes, everyone else serves stale.
if r.set(lock_key, "1", nx=True, ex=lock_ttl):
try:
value = compute()
_store(key, value, fresh_for, keep_stale_for)
return value
finally:
r.delete(lock_key)
return entry["value"]
# Cold miss: nothing to serve, so losers wait briefly for the winner.
if r.set(lock_key, "1", nx=True, ex=lock_ttl):
try:
value = compute()
_store(key, value, fresh_for, keep_stale_for)
return value
finally:
r.delete(lock_key)
for _ in range(20):
time.sleep(0.05)
raw = r.get(key)
if raw is not None:
return json.loads(raw)["value"]
return compute()
Two details bite people here. The lock needs its own TTL, or a worker that gets OOM-killed mid-recompute leaves the key permanently unrefreshable. And r.delete(lock_key) in the finally block is technically unsafe: if the recompute overruns lock_ttl, the lock has already expired and been taken by someone else, so you just deleted their lock. The correct version stores a random token in the lock and deletes it with a small Lua script that compares the token first. For a 10-second lock around a 200 ms query the exposure is small — but it is why single-flight code that "works fine" in staging occasionally double-computes in production.
In Go, do not write this by hand: golang.org/x/sync/singleflight collapses duplicate calls, with the caveat that it is per-process, so ten pods still produce ten recomputes.
Single-flight turns N concurrent recomputes into one, but the N-1 losers on a cold miss are still waiting.
Fix 3: Stale-while-revalidate, which is the one that removes the cliff
The code above is already stale-while-revalidate: the entry carries a logical freshness timestamp and the Redis TTL is set much longer. Between fresh_until and physical expiry, readers get an instant cached response while exactly one worker refreshes in the background. Nobody blocks, so there is no cliff — the origin sees one query per refresh interval regardless of traffic.
The cost is correctness: you knowingly serve data up to keep_stale_for seconds old when refreshes fail. For a dashboard aggregate or a search facet count, that is free. For anything a user just wrote and expects to see, it is not — those keys need explicit invalidation on write and stale windows measured in seconds.
At the HTTP layer you may not need application code at all. Varnish coalesces concurrent requests for the same object by default and can serve stale content while fetching a fresh copy; nginx does the narrower version with proxy_cache_lock on; plus proxy_cache_use_stale updating;. Both apply only to full HTTP responses keyed by URL, so neither helps the internal fragment caches that usually cause this.
The rule of thumb: if you can name a tolerable staleness window for a key, stale-while-revalidate is strictly better than locking alone.
Fix 4: Probabilistic early recomputation
If you cannot serve stale — say the value is a signed token or a count that must never go backwards — each reader can independently decide to refresh slightly early, with probability rising as expiry approaches. This is the XFetch approach from the 2015 paper on optimal probabilistic cache stampede prevention, and it is about six lines:
import math, random
def should_refresh_early(delta_seconds, ttl_remaining, beta=1.0):
"""delta_seconds = how long the last recompute took."""
return delta_seconds * beta * -math.log(random.random()) >= ttl_remaining
Store delta alongside the value when you write it. Expensive values start volunteering for refresh earlier; cheap ones wait. Raising beta above 1.0 makes refreshes more eager. I have shipped this, but it is the fix I reach for last: it adds a tunable nobody will remember the meaning of in six months, and it only helps when stale-serving is off the table.
Which fix should you actually apply?
| Approach | Origin load at expiry | Reader waits? | Serves stale? | Main cost |
|---|---|---|---|---|
| Jittered TTL | Unchanged per key | Yes | No | Nothing — always do it |
| Single-flight lock | 1 recompute | Yes, on cold miss | No | Lock-expiry edge cases |
| Stale-while-revalidate | 1 recompute | No | Yes | Bounded staleness |
| Probabilistic early refresh | 1 recompute, before expiry | Rarely | No | An opaque tuning knob |
For most services the answer is the middle two together — exactly what the code above does: single-flight on cold misses, stale-while-revalidate on warm ones, jitter on every write.
FAQ
What is a cache stampede?
A cache stampede (also called a thundering herd or dog-piling) happens when a cached value expires and many concurrent requests miss simultaneously, each recomputing the same value against the origin. The result is a burst of identical queries that can saturate the database even though overall hit rate stays high.
How do I prevent a Redis cache stampede?
Use SET lock:<key> <token> NX EX <seconds> so only one worker recomputes a given key, store the entry with a logical freshness timestamp shorter than its Redis TTL so other readers serve slightly stale data instead of blocking, and add ±10% jitter to every TTL so bulk-populated keys never expire in the same second.
Does increasing the TTL fix a cache stampede?
No. A longer TTL makes stampedes less frequent but more severe, because more traffic accumulates behind each expiry and the recompute is colder. The concurrency of the miss is what hurts, not how often it occurs.
Bottom line
If you have a periodic latency cliff and a healthy-looking cache, add jitter today and single-flight this week. If the value tolerates being a few seconds old — most aggregates, counts, and rendered fragments do — go straight to stale-while-revalidate, the only option here where readers never wait on a recompute. Reserve probabilistic early refresh for values that genuinely cannot be served stale, and log misses per key so the next incident takes one query to diagnose instead of an hour.
Top comments (0)