You added Redis in front of an expensive query, and it worked beautifully. The dashboard that took 800 ms now returns in 3 ms. The database CPU dropped from 70% to 10%. Everyone is happy.
Then, one afternoon, the database CPU spikes to 100%, latency goes vertical, and the incident channel lights up. The graphs show a sharp, periodic pattern: calm, spike, calm, spike. Nothing deployed. Traffic didn't change.
What happened is a cache stampede (also called the thundering herd or dogpile problem). A hot key expired, and for the next few hundred milliseconds, every request that wanted it missed the cache at the same moment and ran the expensive query itself.
This post explains exactly how that happens and builds up a set of defenses in Node.js with ioredis, from a ten-line fix to a production-grade helper.
Anatomy of a stampede
Here's the classic cache-aside code:
async function getLeaderboard() {
const cached = await redis.get('leaderboard');
if (cached) return JSON.parse(cached);
const data = await db.query(EXPENSIVE_SQL); // takes 800 ms
await redis.set('leaderboard', JSON.stringify(data), 'EX', 60);
return data;
}
Perfectly reasonable. Now watch what happens at the moment the key expires, with 500 requests per second hitting this endpoint:
t=0.000 key expires
t=0.001 request 1: cache miss → starts the 800 ms query
t=0.003 request 2: cache miss → starts the 800 ms query
t=0.004 request 3: cache miss → starts the 800 ms query
...
t=0.800 request 1 finishes, writes the cache
(by now ~400 requests have missed and started the same query)
The query takes 800 ms, so for 800 ms every request is a miss. At 500 req/s that's ~400 identical expensive queries launched against the database in under a second, instead of one. The database slows down under that load, which makes the queries take longer, which widens the miss window, which launches more queries. It's a feedback loop.
Related problems with similar names
People mix these up, but the fixes differ:
| Problem | What happens | Typical cause |
|---|---|---|
| Stampede / thundering herd | Many concurrent misses on one hot key | A popular key expires |
| Avalanche | Many different keys expire together, or the cache is gone | Same TTL set in a batch; Redis restart or failover |
| Penetration | Requests for data that doesn't exist always miss and hit the database | Missing IDs, scanners, bad input |
We'll address all three, since they feed each other.
Defense 1: Single-flight (collapse concurrent misses in one process)
The cheapest fix: if a load for a key is already in progress in this process, wait for it instead of starting another. Store the in-flight promise in a map.
const inflight = new Map<string, Promise<unknown>>();
async function singleFlight<T>(key: string, fn: () => Promise<T>): Promise<T> {
const existing = inflight.get(key);
if (existing) return existing as Promise<T>;
const promise = fn().finally(() => inflight.delete(key));
inflight.set(key, promise);
return promise;
}
async function getLeaderboard() {
const cached = await redis.get('leaderboard');
if (cached) return JSON.parse(cached);
return singleFlight('leaderboard', async () => {
const data = await db.query(EXPENSIVE_SQL);
await redis.set('leaderboard', JSON.stringify(data), 'EX', 60);
return data;
});
}
Now 400 concurrent misses in one process become one database query, and all 400 callers share its result. Because Node.js is single-threaded, the check-then-set on the map has no race.
The limitation is in the name: it's per process. With 20 instances you get up to 20 concurrent loads per expiry, not 400, a big improvement but not a guarantee. For that, you need coordination through Redis.
Defense 2: A distributed lock so only one process rebuilds the value
Use Redis itself to elect one loader across the whole fleet. SET key value NX PX ttl sets the key only if it doesn't exist, atomically:
import { randomUUID } from 'node:crypto';
async function tryLock(key: string, ttlMs: number): Promise<string | null> {
const token = randomUUID();
const ok = await redis.set(`lock:${key}`, token, 'PX', ttlMs, 'NX');
return ok === 'OK' ? token : null;
}
// Release only if we still own the lock. Compare-and-delete must be atomic, so use Lua.
const RELEASE = `
if redis.call('get', KEYS[1]) == ARGV[1] then
return redis.call('del', KEYS[1])
else
return 0
end`;
async function unlock(key: string, token: string) {
await redis.eval(RELEASE, 1, `lock:${key}`, token);
}
Two details matter:
-
The random token + Lua release ensures you never delete someone else's lock. Without it, if your work outlasts the lock's TTL, another process acquires the lock, and your delayed
DELreleases their lock. - The lock TTL is a safety net, not a timer. It must exceed your worst-case load time, so a crashed process can't hold the lock forever.
Then the miss path becomes:
async function getWithLock<T>(key: string, loader: () => Promise<T>, ttlSec: number): Promise<T> {
const cached = await redis.get(key);
if (cached) return JSON.parse(cached);
const token = await tryLock(key, 10_000);
if (token) {
try {
const value = await loader();
await redis.set(key, JSON.stringify(value), 'EX', ttlSec);
return value;
} finally {
await unlock(key, token);
}
}
// Someone else is loading. Wait briefly for their result instead of piling onto the database.
for (let i = 0; i < 40; i++) { // ~2s max
await sleep(50);
const filled = await redis.get(key);
if (filled) return JSON.parse(filled);
}
return loader(); // gave up waiting: load it ourselves
}
Is this a "correct" distributed lock? Not in the strict sense. A paused process can outlive its lease, and a single Redis node can fail over mid-lock. But for cache filling that doesn't matter: the worst case is that two processes load the same value and write the same result. You're using the lock to reduce duplicate work, not to guard a critical section, so a simple SET NX PX is enough. (If you're protecting something where duplicates are harmful, that's a different problem and needs fencing tokens or a database constraint, not a cache lock.)
The downside: waiters are blocked. Everyone who missed is stuck polling for up to the load time. The next defense removes that wait entirely.
Defense 3: Stale-while-revalidate (never make users wait)
The insight: for most cached data, slightly old data is better than a slow response. So don't delete the value when it "expires". Keep it around, mark it stale, and refresh it in the background while still serving the old copy.
Store two expirations: a soft TTL (when it becomes stale) and a hard TTL (when Redis actually deletes it), longer by a grace window.
interface Entry<T> {
v: T;
softExp: number; // epoch ms: after this, the value is stale and should be refreshed
}
async function set<T>(key: string, value: T, softTtlMs: number, graceMs: number) {
const entry: Entry<T> = { v: value, softExp: Date.now() + softTtlMs };
await redis.set(key, JSON.stringify(entry), 'PX', softTtlMs + graceMs);
}
On read:
-
Fresh (
now < softExp): return it. -
Stale (past
softExpbut still in Redis): return it immediately, and if you win the lock, refresh in the background. - Missing (past the hard TTL): load synchronously. This is the only path where someone waits.
Now the expensive load happens off the request path. Users see 3 ms responses all the time, and the database sees one refresh per interval. As a bonus, if the refresh fails (the database is struggling), you keep serving the stale copy, a "stale-if-error" safety net that turns a database outage into slightly old data rather than errors.
Defense 4: Probabilistic early refresh (no locks needed)
Another approach avoids coordination entirely: have requests randomly refresh the value shortly before it expires, with probability that rises as expiry approaches. By the time the key would expire, someone has almost certainly already refreshed it.
This is the XFetch algorithm from the paper Optimal Probabilistic Cache Stampede Prevention (Vattani et al.). Store how long the value took to compute (delta), and on each read decide:
// Refresh early if: now - delta * beta * ln(random()) >= expiry
function shouldRefreshEarly(entry: { softExp: number; delta: number }, beta = 1): boolean {
return Date.now() - entry.delta * beta * Math.log(Math.random()) >= entry.softExp;
}
Math.log(Math.random()) is negative, so the subtraction pushes "now" forward by a random amount. The expression is scaled by delta:
-
Expensive values (large
delta) start refreshing earlier, because they need more lead time. - Cheap values refresh almost right at expiry.
-
beta(default 1) tunes aggressiveness. Higher means earlier refreshes and more cache churn.
Used alone, XFetch can still let a few concurrent requests decide to refresh simultaneously, but instead of hundreds it's a handful. Combined with single-flight or a lock, it's very effective.
Defense 5: Prevent synchronized expiry with TTL jitter
If you warm 10,000 keys at deploy time with EX 3600, they all expire in the same second an hour later. That's the avalanche pattern. The fix is one line: add randomness.
const jitter = (ms: number, spread = 0.1) =>
Math.round(ms * (1 - spread + Math.random() * spread * 2)); // ±10%
await redis.set(key, value, 'PX', jitter(60 * 60 * 1000));
Keys written together now expire spread across a window instead of all at once. It's trivial, it costs nothing, and it should be your default for every cached key.
Defense 6: Negative caching (stop penetration)
If a client requests /users/999999999 (an ID that doesn't exist), the cache never has it, so every request goes to the database, forever. Scanners and bugs generate these constantly.
Cache the absence too, with a short TTL:
// Wrap values so that "null" is a legitimate cached result, distinct from "key not in cache"
const entry = { v: user ?? null, softExp: Date.now() + (user ? 60_000 : 10_000) };
Use a shorter TTL for "not found" (seconds, not minutes) so a newly created record shows up quickly, or invalidate that key on create. This is why the entry wrapper above stores v inside an object. A raw null would be indistinguishable from a miss.
For very large, hostile key spaces, a Bloom filter in front of the cache can reject IDs that definitely don't exist, but start with negative caching.
Putting it together: a reusable helper
Here's a compact implementation combining single-flight, Redis lock, stale-while-revalidate, XFetch, jitter, and a safe fallback if Redis is unhealthy:
// cache.ts
import { randomUUID } from 'node:crypto';
import type Redis from 'ioredis';
interface Entry<T> { v: T; softExp: number; delta: number }
export interface CacheOpts {
ttlMs: number; // how long the value is considered fresh
staleMs?: number; // extra time it may be served stale while refreshing (default: 5x ttlMs)
lockMs?: number; // lock safety TTL: must exceed worst-case load time
waitMs?: number; // how long a cold-miss waiter polls for another loader's result
}
const sleep = (ms: number) => new Promise((r) => setTimeout(r, ms));
const jitter = (ms: number) => Math.round(ms * (0.9 + Math.random() * 0.2));
const RELEASE = `
if redis.call('get', KEYS[1]) == ARGV[1] then return redis.call('del', KEYS[1]) else return 0 end`;
export function createCache(redis: Redis, log = console) {
const inflight = new Map<string, Promise<unknown>>();
async function tryLock(key: string, ms: number) {
const token = randomUUID();
const ok = await redis.set(`lock:${key}`, token, 'PX', ms, 'NX');
return ok === 'OK' ? token : null;
}
const unlock = (key: string, token: string) =>
redis.eval(RELEASE, 1, `lock:${key}`, token).catch(() => {});
async function load<T>(key: string, loader: () => Promise<T>, o: Required<CacheOpts>): Promise<T> {
const started = Date.now();
const v = await loader();
const delta = Date.now() - started; // how long the load took (for XFetch)
const softTtl = jitter(o.ttlMs);
const entry: Entry<T> = { v, softExp: Date.now() + softTtl, delta };
await redis.set(key, JSON.stringify(entry), 'PX', softTtl + o.staleMs).catch((err) =>
log.error('cache set failed', err),
);
return v;
}
async function read<T>(key: string, loader: () => Promise<T>, o: Required<CacheOpts>): Promise<T> {
let raw: string | null;
try {
raw = await redis.get(key);
} catch (err) {
log.error('redis get failed; loading directly', err);
return loader(); // fail open (see "When Redis is down")
}
const entry: Entry<T> | null = raw ? JSON.parse(raw) : null;
const now = Date.now();
// 1. Fresh and not selected for early refresh: serve it
if (entry && now < entry.softExp && now - entry.delta * Math.log(Math.random()) < entry.softExp) {
return entry.v;
}
// 2. Stale or about to expire: serve the old value, one refresher updates it in the background
if (entry) {
const token = await tryLock(key, o.lockMs).catch(() => null);
if (token) {
load(key, loader, o)
.catch((err) => log.error('background refresh failed; serving stale', err))
.finally(() => unlock(key, token));
}
return entry.v;
}
// 3. True miss: one caller loads; the others wait briefly for the result
const token = await tryLock(key, o.lockMs).catch(() => null);
if (token) {
try { return await load(key, loader, o); } finally { await unlock(key, token); }
}
const deadline = Date.now() + o.waitMs;
while (Date.now() < deadline) {
await sleep(50 + Math.random() * 50);
const filled = await redis.get(key).catch(() => null);
if (filled) return (JSON.parse(filled) as Entry<T>).v;
}
return load(key, loader, o); // waited long enough: load it ourselves
}
return function cached<T>(key: string, loader: () => Promise<T>, opts: CacheOpts): Promise<T> {
const o: Required<CacheOpts> = {
staleMs: opts.ttlMs * 5,
lockMs: 15_000,
waitMs: 3_000,
...opts,
};
// Single-flight inside this process
const existing = inflight.get(key);
if (existing) return existing as Promise<T>;
const p = read<T>(key, loader, o).finally(() => inflight.delete(key));
inflight.set(key, p);
return p;
};
}
Usage:
const cached = createCache(redis);
app.get('/leaderboard', async (_req, res) => {
const data = await cached('leaderboard:v1', () => db.getLeaderboard(), {
ttlMs: 60_000, // fresh for ~1 minute
staleMs: 5 * 60_000, // may be served stale for up to 5 more minutes while refreshing
});
res.json(data);
});
A walkthrough of what the helper does:
-
Fresh hit: returns immediately (the common path, one Redis
GET). - Stale or nearing expiry: returns the old value, and the one process that wins the lock refreshes in the background. XFetch randomly pulls some refreshes forward, so expiry rarely arrives "cold".
- Cold miss: one caller across the fleet loads; the others poll briefly. If they time out, they load themselves rather than failing.
- TTL jitter is applied to every write.
- Single-flight collapses concurrent in-process callers before Redis is even consulted.
- Lock released safely via the Lua compare-and-delete.
Because the value is wrapped in an entry object, caching null (negative caching) just works. Use a shorter ttlMs when the loader returns "not found."
Notes on the choices:
-
lockMsmust exceed your worst-case loader time, or a slow load will let a second loader in. It's still only a duplicate load, not corruption. -
Version your keys (
leaderboard:v1) so you can invalidate by bumping the version when the shape changes. Old entries age out on their own. - Don't cache errors. The helper only writes on success, so a failing loader leaves the previous (stale) value in place.
When Redis itself is down
The try/catch around redis.get makes the helper fail open: if Redis is unreachable, it calls the loader directly. That keeps the app working, but it's also a recipe for a stampede, because now every request goes to the database. Single-flight limits it to one load per key per process, but with many distinct keys the database still takes the full force.
Add backstops at the database edge:
-
A concurrency limit on loaders (a semaphore or
p-limit) so the database never sees more than N concurrent cache-fill queries - A circuit breaker around the loader: when it's failing or slow, fail fast or serve stale rather than piling on
-
Aggressive timeouts on Redis calls (
commandTimeout,connectTimeoutinioredis) so a hung Redis doesn't hang every request -
Load shedding (
503withRetry-After) on non-critical endpoints during the incident
Treat Redis as an optimization that may vanish at any moment, and make sure the database survives that moment.
Warming and deploys
Cold caches cause stampedes too, right after a Redis restart, failover, flush, or a deploy that bumps a key version.
- Warm critical keys before taking traffic (a startup script that loads the top N keys), or populate them from a background job
- Roll out gradually. Deploying all instances at once with a new key prefix means all of them miss simultaneously
- Refresh-ahead for known-hot keys: a scheduled job that recomputes the leaderboard every 30 seconds means requests never trigger the load at all
For a few well-known hot keys, refresh-ahead via a scheduler is often simpler and more predictable than any request-driven scheme.
Test it: prove the stampede is gone
Don't trust the cache until you've watched it under concurrent load. Count how many times the loader actually runs:
import { it, expect } from 'vitest';
it('runs the loader once for 200 concurrent cold misses across 2 "processes"', async () => {
const cachedA = createCache(redis); // separate in-process maps simulate two instances
const cachedB = createCache(redis);
await redis.flushdb();
let loads = 0;
const loader = async () => { loads++; await sleep(300); return { top: 'alice' }; };
const calls = Array.from({ length: 200 }, (_, i) =>
(i % 2 ? cachedA : cachedB)('leaderboard:test', loader, { ttlMs: 60_000 }),
);
const results = await Promise.all(calls);
expect(results.every((r) => r.top === 'alice')).toBe(true);
expect(loads).toBe(1); // not 200
});
it('serves stale and refreshes once after the soft TTL', async () => {
// set ttlMs small, wait past it, fire N requests, assert they return instantly
// with the old value and that the loader ran exactly once in the background
});
Also run a load test (autocannon or k6) against a staging environment with a short TTL (say 5 seconds) so that you hit expiries constantly, and watch the database's query rate and connection count. A healthy setup shows a flat line with small, steady refresh queries rather than periodic spikes.
What to monitor
- Hit rate, split into fresh hits, stale hits, and misses
- Loader concurrency per key. Spikes mean a stampede is forming
-
Loader duration (it's also your
delta, and shows when queries are slowing) - Lock contention: how often callers fail to get the lock, and how often waiters time out and load anyway
- Database queries per second and its correlation with cache expiry times
- Redis latency and errors, since the fail-open path is your riskiest moment
Choosing defenses
| Technique | Prevents | Cost | Notes |
|---|---|---|---|
| Single-flight (in-process) | Stampede within one process | Tiny | Always do it, but it's per-process |
Redis lock (SET NX PX) |
Stampede across the fleet | A Redis round trip; waiters block | Duplicate work is tolerable, so a simple lock suffices |
| Stale-while-revalidate | User-visible latency on refresh | Serves slightly old data; more memory | Best user experience; adds stale-if-error |
| XFetch (early refresh) | Synchronized expiry on hot keys | Slightly more refreshes | No locking; combine with single-flight |
| TTL jitter | Avalanche (mass expiry) | None | Use on every key |
| Negative caching | Penetration | Small memory | Short TTL for "not found" |
| Refresh-ahead job | Everything for known hot keys | A scheduler | Simple and predictable |
| Loader concurrency limit + breaker | Database overload if everything else fails | Small | The last line of defense |
Checklist
- ✅ Collapse concurrent misses with single-flight
- ✅ Use a lock (
SET NX PX+ Lua release) so one process per key rebuilds - ✅ Serve stale while revalidating, so users never wait on a refresh
- ✅ Add TTL jitter to every key
- ✅ Cache "not found" with a short TTL
- ✅ Make the lock TTL longer than the worst-case load time
- ✅ Bound database concurrency, and use timeouts and a circuit breaker for when Redis fails
- ✅ Warm caches before cutover, and roll deploys gradually
- ✅ Test that N concurrent misses cause exactly 1 load
- ✅ Monitor loader concurrency and the stale/fresh/miss split
Key takeaways
- A cache stampede is a feedback loop. When a hot key expires, every concurrent request misses and runs the expensive query, which slows the database and widens the miss window.
- Single-flight is the cheapest fix, but it only works within one process. Use a Redis lock for fleet-wide coordination.
- Stale-while-revalidate is the best user experience. Serve the old value and refresh in the background, and you also get stale-if-error for free.
- Probabilistic early refresh (XFetch) avoids expiry cliffs without coordination.
- Add TTL jitter everywhere and cache misses for nonexistent keys to stop avalanches and penetration.
- Plan for Redis being down. Fail open only if the database has its own concurrency limits and breakers.
- Prove it with a test that fires hundreds of concurrent cold requests and asserts the loader ran once.
Have you been hit by a stampede, or found a pattern I didn't cover? Share it in the comments.
Further reading: the paper "Optimal Probabilistic Cache Stampede Prevention" (Vattani, Chierichetti, Lowenstein), the Redis docs on SET and distributed locks, and the ioredis README.
Top comments (0)