Single-flight caching: how one hot key stops taking the database down

On podbor-minuta.ru some database queries are heavy, so we keep their results in a cache: apartment lists, aggregate numbers for listing pages. While the value is in the cache, everything is fast. The trouble starts the moment it expires.

One day under load we saw a batch of 500 responses timing out. The database was alive, but queries to it suddenly queued up and hit the connection limit. The answer was in how we refreshed the cache.

The logic was naive: if the value is not in the cache, go to the database, compute it, put it in the cache. For a single request that is correct. But imagine the cache has just expired and in the same millisecond two hundred requests arrive for the same page. Each one sees an empty cache and each one goes to the database to compute the same thing. Two hundred identical heavy queries at once. The connection pool drains, the remaining requests get no connection, they wait and drop on timeout.

This is called a cache stampede. Caching itself does not help here, because the hole opens exactly when the value is missing, and that is exactly when the load arrives.

The fix is the idea of "one for all". If a computation for a key is already running, everyone else who comes for the same key does not start their own, they wait for that same result. One query goes to the database, and two hundred callers get the same value.

In practice this is done with a shared in-flight promise. We store not only finished values but also promises of results that are still in progress. The first caller creates a promise and puts it in a map by key. The others see that the promise already exists and simply await it.

const inFlight = new Map<string, Promise<unknown>>();

async function getOrSet<T>(key: string, compute: () => Promise<T>): Promise<T> {
  const cached = cache.get(key);
  if (cached !== undefined) return cached as T; // leave the fast hit path untouched

  const running = inFlight.get(key);
  if (running) return running as Promise<T>; // already computing - await the same result

  const promise = compute()
    .then((value) => {
      cache.set(key, value);
      return value;
    })
    .finally(() => {
      inFlight.delete(key); // release the key no matter what
    });

  inFlight.set(key, promise);
  return promise;
}

Three things matter here. First: the cache-hit path stays fast, we added nothing to it, no maps and no extra checks. Second: we clear the in-flight map in finally, not in then. If the computation fails, the key still has to be released, otherwise the next caller will wait forever on a dead promise. Third: this works inside one process. If you run several replicas of the service, each collapses its own concurrent calls, but there is no protection between replicas. For our load that is enough, because the stampede happens inside one process on a hot key. If you need it stricter, you add a shared lock through Redis, but that is a different cost and a different complexity.

After this change the same load stopped taking the database down. Two hundred requests for a hot key produce one database query instead of two hundred, and the connection pool no longer drains for no reason.

What we took from this. A cache does not protect you from load by itself. The dangerous point is the miss, and if many identical requests arrive at that moment, you get a stampede. It is fixed not by a bigger pool and not by a longer timeout, but by letting concurrent computations of one key share a single unit of work. The fast path must stay untouched, and the key must be released even on an error.

The site is podbor-minuta.ru, daily price monitoring for Moscow new builds.