The database was struggling. The API said the user didn't exist.
That is the part of OpenRouter's February outages worth looking at closely.
The reported failure involved cache connectivity and the database lookups behind API-key authentication. When the cache stopped serving those lookups, the database had to absorb work it normally wouldn't see. Requests started failing with 401 User not found.
There is an important distinction here: an authentication failure and an authentication infrastructure failure are not the same thing. If a database timeout becomes a 401, a client may treat a temporary service failure as an invalid credential. Retries, alerts and incident diagnosis can all head in the wrong direction.
The capacity problem is just as interesting. A database can appear comfortably provisioned because a cache absorbs most of the read traffic. That doesn't mean the database can survive the workload when the cache disappears. Recovery can create its own traffic spike, precisely when the dependency underneath is least able to handle it.
This is why cache-failure testing shouldn't stop at checking whether the application stays online. It should also examine database concurrency, fallback request rates, cache repopulation, retry behavior and the errors returned to clients. A fallback path that works for a few misses may behave very differently when every request takes it at once.
There is also a subtle observability problem here. The first outage coincided with a denial-of-service attack, which initially complicated the investigation. When the caching failure recurred in February without the same accompanying attack, the underlying cause became clearer. Multiple problems can overlap; temporal correlation alone doesn't establish root cause.
The remediation needs to address both load and semantics. Circuit breakers and controlled cache repopulation can limit the burst reaching the database. Returning 503 Service Unavailable rather than 401 Unauthorized when an infrastructure failure prevents an authentication lookup helps clients distinguish an unavailable service from invalid credentials.
A useful test for any cache-backed authentication system: invalidate the cache under realistic concurrency and measure the database's behavior during recovery. How quickly does load rise? Are concurrent misses coalesced? Do retries amplify the spike? Can the service shed work before the database saturates?
Those answers reveal more about the system's failure tolerance than its cache-hit ratio during normal operation.
The full postmortem reconstructs both outages, separates published facts from illustrative capacity calculations, and identifies what the available evidence does—and doesn't—establish about the remediation.
Read teh full teardown here: https://buildopsy.com/system-design/openrouter-cache-thundering-herd-postmortem.html
Top comments (1)
401 for a timeout is the nasty bit. the client reads it as a dead credential, not a down service.