The Cache Hit Ratio Was Fine, but the Burst Still Melted the Database

When I started profiling a small FastAPI service, the first chart suggested the cache was doing its job. Single requests were fast, the hit ratio sat above 90 percent, and the database showed only a modest load during ordinary traffic. The problem appeared only when a release or a retry storm sent twenty requests for the same tenant at once, and then the database CPU climbed while the cache hit ratio stayed high enough to look innocent. That contradiction became the debug trail.

The metric that lied to me

I kept the reproduction small because the failure depended on timing, not on the size of the data. The service cached tenant project lists for thirty seconds, and the cache looked healthy in the access log. But a high hit ratio only tells you what happened after a key was already warm; it does not tell you how many cold-start requests turned into database work at the same moment.

Run this minimal repro and watch the counter climb:

import threading
import time
from concurrent.futures import ThreadPoolExecutor
from fastapi import FastAPI
from fastapi.testclient import TestClient

app = FastAPI()
cache = {}
cache_lock = threading.Lock()
db_calls = 0

def read_from_db(tenant_id):
    global db_calls
    db_calls += 1
    time.sleep(0.1)  # a slow query, shortened for the repro
    return [{'tenant': tenant_id, 'projects': ['checkout', 'inventory']}]

@app.get('/projects/{tenant_id}')
def projects(tenant_id):
    key = f'projects:{tenant_id}'
    now = time.monotonic()
    with cache_lock:
        entry = cache.get(key)
    if entry and entry['expires'] > now:
        return {'source': 'cache', 'projects': entry['projects']}

    rows = read_from_db(tenant_id)
    with cache_lock:
        cache[key] = {'expires': time.monotonic() + 30, 'projects': rows}
    return {'source': 'db', 'projects': rows}

client = TestClient(app)

def hit(_):
    return client.get('/projects/tenant-a')

with ThreadPoolExecutor(max_workers=20) as pool:
    responses = list(pool.map(hit, range(20)))

print('db_calls:', db_calls)
print('db responses:', sum(1 for r in responses if r.json()['source'] == 'db'))

The output shows db_calls: 20, even though every request after the first one could have waited for the same result. The cache did not reduce cold-start concurrency; it only deduplicated later reads after the first response finally landed in the dictionary.

The race is in the lazy load

The first request checked the cache, missed, released the lock, and went to the database. The other nineteen requests did exactly the same thing because nobody had marked the key as being loaded. That pattern is a cache stampede, and it can turn a harmless cache expiry into a burst of identical queries. The health dashboard was not broken; it simply answered a different question than the burst was asking.

Before fixing anything, I made the failure measurable. If a change did not move that db_calls counter from twenty toward one, it was not solving the race.

Fix the missed cache with a per-key singleflight

The fix is to let one caller do the work while the others wait for that work to finish. I used MonkeyCode's free model access to review a per-key lock against a process-wide lock before committing to one. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Here is the smallest singleflight I would trust in a threaded FastAPI process:

class SingleFlight:
    def __init__(self):
        self.lock = threading.Lock()
        self.inflight = {}

    def do(self, key, fn):
        with self.lock:
            if key in self.inflight:
                event, box = self.inflight[key]
                should_run = False
            else:
                event, box = threading.Event(), {}
                self.inflight[key] = (event, box)
                should_run = True

        if should_run:
            try:
                box['result'] = fn()
            except Exception as exc:
                box['error'] = exc
            finally:
                event.set()
                with self.lock:
                    self.inflight.pop(key, None)
        else:
            event.wait()

        if 'error' in box:
            raise box['error']
        return box['result']

Wire it into the endpoint so the cache miss path calls the database through the flight map:

flight = SingleFlight()

@app.get('/projects/{tenant_id}')
def projects(tenant_id):
    key = f'projects:{tenant_id}'
    now = time.monotonic()
    with cache_lock:
        entry = cache.get(key)
    if entry and entry['expires'] > now:
        return {'source': 'cache', 'projects': entry['projects']}

    def load():
        return read_from_db(tenant_id)

    rows = flight.do(key, load)
    with cache_lock:
        cache[key] = {'expires': time.monotonic() + 30, 'projects': rows}
    return {'source': 'db', 'projects': rows}

Run the same burst test and db_calls drops to one. The other callers wait on the event and receive the same result, so the database sees a single query while the cache stays useful afterwards.

Verify it in a second environment, not just on one laptop

Thread scheduling can hide a race if you only run the test on the same machine that wrote the code. I ran the script on MonkeyCode's free server option as a second environment, because a different container changes the timing enough to expose whether the lock is actually shared across workers. The result stayed consistent in that environment: twenty requests and one database call instead of twenty.

The takeaway is not that a singleflight is always the correct fix. It is that a cache miss is still a control-flow decision, and control flow under concurrency needs an explicit owner.

Decision table before you copy this

Situation	Tool
One process, cache misses on the same key under a burst	In-process per-key singleflight
Multiple replicas sharing one Redis cache	Redis SET NX or a distributed lock
Stale data is acceptable during a refresh	Stale-while-revalidate
Many different keys are missing at once	Connection pooling, not a per-key flight map
Very low traffic and a cheap query	Keep the code simple; skip the lock

What this still does not fix

• A local singleflight only works inside one process. Multiple Gunicorn workers or multiple replicas each get their own flight map.
• It does not help when the burst is spread across many different tenant keys.
• It shares one result with every waiter, so it cannot satisfy callers that genuinely need a newer value.
• The lock adds bookkeeping, so it is wasted work when the database query is faster than the coordination.
• A long-running load keeps waiters blocked unless you add an outer timeout and decide what the fallback should be.

Who should avoid this pattern

Keep the simple cache if your traffic is low, your query is cheap, or you already have a distributed atomic cache layer doing the same coordination. Also avoid a local singleflight if you run many replicas but do not pair it with a shared lock, because that creates the illusion of protection while every replica still stamps the database once. The lock is a targeted tool for a narrow cold-start race, not a general performance patch.

The dashboard was not wrong; it was answering a different question than the burst was asking. A hit ratio tells you what happened after a key was warm, not whether all of the cold-start traffic collapsed into one query. Reproduce the cold path, count the database calls, and only then decide whether a lock is worth the complexity.