How We Eliminated Cache Stampedes in Node.js & Bun with Single-Flight Coalescing

Every engineer who has scaled a high-traffic backend has lived through this nightmare:

You have an expensive endpoint (e.g., aggregating leaderboard data or rendering a personalized dashboard). To protect your database, you cache the result with a 60-second TTL.

Everything runs smoothly until the key expires while receiving 1,500 requests per second.

Within a 50-millisecond window, hundreds of concurrent requests see a cache miss. All of them immediately hammer the primary database simultaneously to recompute the same data. Your connection pool saturates, CPU spikes to 100%, latency explodes, and your server crashes.

This is the thundering herd / cache stampede problem.

---

The Traditional (Fragile) Workarounds

1. Probabilistic Early Expiration (XFetch algorithm): Recomputes the value before it expires based on an exponential probability. While effective, it wastes backend CPU cycles recomputing keys that may never be accessed again.
2. Distributed Mutex Locks (Redlock): Acquiring a distributed lock before computing. This introduces noticeable network roundtrip latency and runs the risk of deadlocks if a worker dies mid-flight.

---

The Zero-Overhead Alternative: Single-Flight Promise Coalescing

Instead of locking distributed infrastructure, we can coalesce concurrent requests in-process at the execution runtime layer.

When Key A is requested and misses the cache:

1. Check if an execution promise for Key A is currently in-flight.
2. If YES: Do not touch the database. Return the existing promise and await it.
3. If NO: Register a new in-flight promise, execute the database fetcher, cache the result, resolve all awaiting callers, and deregister the promise in a finally block.

// How KVDock implements zero-overhead single-flight coalescing:
const running = this.inFlight.get(key);
if (running) {
  this.metrics.stampedeCoalesced++;
  return (await running) as T;
}

const run = (async () => {
  try {
    const cached = await this.get<T>(key);
    if (cached !== null) return cached;
    const fresh = await fetcher();
    await this.set(key, fresh, options);
    return fresh;
  } finally {
    this.inFlight.delete(key);
  }
})();

this.inFlight.set(key, run);
return (await run) as T;

---

⚡ Real-World Benchmark: 1,000 Concurrent Requests

We ran a controlled benchmark under 1,000 concurrent requests to an uncached endpoint with 25ms simulated database query latency:

Metric	Traditional Caching	KVDock (getOrSet)	Net Improvement
Incoming Requests	1,000	1,000	Same Load
Database Queries Executed	1,000 queries (Herd)	1 query (Single-Flight)	99.90% DB load reduction
Coalesced Waiting Callers	0	999	Zero lock contention
Connection Pool Exhaustion	💥 Saturated	🟢 Baseline Stable	Zero connection contention

---

Open Source & Polyglot

We built this into KVDock — an open-source, zero-dependency caching engine with multi-tier storage (L1 RAM + L2 Persistent):

• Zero runtime dependencies in core
• Multi-tier caching with automatic backfill
• Cross-runtime support across Node.js, Bun, Deno, and Cloudflare Workers
• Native polyglot implementations in Rust (kvdock-rs), Go (kvdock-go), Swift (kvdock-swift), and Kotlin (kvdock-kt)

---

⭐ Explore the code & star on GitHub: github.com/zisam-glitch/kvdock

👤 Follow the maintainer: @zisam-glitch for low-latency systems engineering deep dives!