Our System Crashed at 14:22: It Wasn't the Database

TL;DR: During peak concurrent load, our 5-minute cache unexpectedly evicted keys early due to lock contention. The solution was migrating to asynchronous stale-while-revalidate with an in-memory semaphore. The diff took 18 lines and cut our p99 latency from 1.8s down to 240ms.

What Broke in Production

Last Tuesday, our primary streaming endpoint started throwing sporadic timeouts.
Measured impact:

• p99 Latency: jumped from 280ms to 3,400ms.
• 504 Error Rate: reached 4.2% across an 18-minute window.
• Connection Pool: 100% saturated.

What We Thought Happened (The Wrong Hypothesis)

Our initial instinct was to blame upstream LLM provider throttling. It looked like classic HTTP 429 backpressure. We restarted Celery workers, but within 90 seconds the pool was choking again.

The Actual Root Cause

The culprit was an internal thundering herd problem. When 300 concurrent requests hit an expired cache key at second 300, every single worker triggered the identical upstream recomputation query at the same instant.

[Request A] ──┐
[Request B] ──┼─► [Expired Cache Key] ──► 300 simultaneous upstream calls
[Request C] ──┘

The Code Fix

Instead of recomputing synchronously inside the request thread, we implemented non-blocking lock acquisition that serves stale data while a single detached coroutine refreshes the cache in the background:

import asyncio

async def get_with_revalidation(cache, lock, key: str, factory_coro):
    data, expired = await cache.get_stale(key)
    if expired and not await lock.is_locked(key):
        asyncio.create_task(revalidate_background(cache, lock, key, factory_coro))
    return data or await factory_coro()

async def revalidate_background(cache, lock, key: str, factory_coro):
    async with lock.acquire(key):
        fresh = await factory_coro()
        await cache.set(key, fresh, ttl=300)

When NOT to Use This

If your system handles strict financial balances or ledger transactions where 2-second stale reads cause double spends, do not use stale-while-revalidate. In our case, serving model metadata and prompt routing rules, the trade-off is safe and highly recommended.

Reproduction & Benchmarks

We documented the full Locust load-test harness and synthetic workload in our open engineering runbook:

• Test methodology: GitHub/BeefAPI Gateway
• Production implementation: gateway de alta resiliência e medição de tokens para LLMs.

Comments

[Mike Dabydeen]

The diagnosis is the hard part and you got it: the herd was the cause and the cache was only where it surfaced. Four things about the code as posted, roughly in order of what they would cost you.

The check and the acquire are not atomic. not await lock.is_locked(key) and async with lock.acquire(key) are separated by a task spawn, so under the same 300 concurrent requests a lot of them can see an unlocked key and each schedule a revalidation. Those then serialise on the lock rather than firing together, which changes the shape of the load and not the volume, since upstream still gets called once per task. The line that fixes it goes inside the lock: after acquiring, read the key again and return without calling the factory if somebody already refreshed it. Single flight is the second check, not the lock.

Cold keys still take the old path. return data or await factory_coro() means that when there is no stale value at all, every concurrent request computes inline, which is the original incident. That state is not rare: every deploy that clears the cache, every eviction under memory pressure, and the first traffic to any new key. Stale while revalidate protects you from expiry, not from absence. Separately, data or treats an empty dict or a zero as a miss, so a legitimately falsy cached value recomputes on every request forever.

An in-memory semaphore is per process. You restarted Celery workers, so there is more than one process and probably more than one pod, and a lock local to a process divides the herd by the worker count rather than reducing it to one. The single detached coroutine is one per worker. That may well be fine at your scale, but it is worth writing down, because the number of upstream calls under the fix is now a capacity input that moves every time you scale out.

The last one is small and bites at the worst moment. asyncio.create_task with no reference kept can be collected mid flight. The Python docs are explicit that the loop holds only weak references and that a task not referenced elsewhere "may get garbage collected at any time, even before it's done". The failure mode is a revalidation that silently never finishes, so the key stays stale past its TTL and nothing logs anything. Keep the task in a set and discard it in a done callback, or use a TaskGroup.

One question on the numbers. Is the 240ms p99 measured on a warm cache, and do you have the equivalent for the first minute after a deploy? That second number is the one your fix does not currently move.