Our System Crashed at 14:22: It Wasn't the Database

Cogumellum — Mon, 14 Sep 2026 02:59:18 +0000

Our System Crashed at 14:22: It Wasn't the Database

TL;DR: During peak concurrent load, our 5-minute cache unexpectedly evicted keys early due to lock contention. The solution was migrating to asynchronous stale-while-revalidate with an in-memory semaphore. The diff took 18 lines and cut our p99 latency from 1.8s down to 240ms.

What Broke in Production

Last Tuesday, our primary streaming endpoint started throwing sporadic timeouts.
Measured impact:

p99 Latency: jumped from 280ms to 3,400ms.
504 Error Rate: reached 4.2% across an 18-minute window.
Connection Pool: 100% saturated.

What We Thought Happened (The Wrong Hypothesis)

Our initial instinct was to blame upstream LLM provider throttling. It looked like classic HTTP 429 backpressure. We restarted Celery workers, but within 90 seconds the pool was choking again.

The Actual Root Cause

The culprit was an internal thundering herd problem. When 300 concurrent requests hit an expired cache key at second 300, every single worker triggered the identical upstream recomputation query at the same instant.

[Request A] ──┐
[Request B] ──┼─► [Expired Cache Key] ──► 300 simultaneous upstream calls
[Request C] ──┘

The Code Fix

Instead of recomputing synchronously inside the request thread, we implemented non-blocking lock acquisition that serves stale data while a single detached coroutine refreshes the cache in the background:

import asyncio

async def get_with_revalidation(cache, lock, key: str, factory_coro):
    data, expired = await cache.get_stale(key)
    if expired and not await lock.is_locked(key):
        asyncio.create_task(revalidate_background(cache, lock, key, factory_coro))
    return data or await factory_coro()

async def revalidate_background(cache, lock, key: str, factory_coro):
    async with lock.acquire(key):
        fresh = await factory_coro()
        await cache.set(key, fresh, ttl=300)

When NOT to Use This

If your system handles strict financial balances or ledger transactions where 2-second stale reads cause double spends, do not use stale-while-revalidate. In our case, serving model metadata and prompt routing rules, the trade-off is safe and highly recommended.

Reproduction & Benchmarks

We documented the full Locust load-test harness and synthetic workload in our open engineering runbook:

Test methodology: GitHub/BeefAPI Gateway
Production implementation: gateway de alta resiliência e medição de tokens para LLMs.

DEV Community: Cogumellum

Our System Crashed at 14:22: It Wasn't the Database

Our System Crashed at 14:22: It Wasn't the Database

What Broke in Production

What We Thought Happened (The Wrong Hypothesis)

The Actual Root Cause

The Code Fix

When NOT to Use This

Reproduction & Benchmarks