Today's blog is mostly about the common ways cache misses have happened in our system, and how we resolved them.
Cache misses — and the system crashes they cause — are something you will face very often when you implement caching in a system. Until you face them, there's no way to know how the system can collapse without them.
1. Key simply doesn't exist yet (cold miss)
First-ever request for that data — nothing wrong here, just an empty cache.
Strategy: Cache-aside (lazy loading) — read from the DB on a miss, populate the cache, then return. This is the default pattern for a reason: it only caches what's actually requested. The upside here is that the request is usually new and not yet a common one, so you can afford not to overthink it.
2. Key exists but fails to deserialize
The bytes are there, but a schema change, version drift, or a bad write left the value unreadable. This one's sneaky — it looks like a miss, but it's actually a read failure being swallowed.
Strategy: Version your cache keys alongside your data schema (e.g., user:v2:123) so old, incompatible values naturally stop being looked up instead of throwing on read. Also log deserialization failures separately from true misses — conflating them hides real bugs. We faced this issue with a failing Jackson deserialization, which made the system go completely haywire.
3. TTL expiry — and the stampede risk on high-value keys
A key ages out normally. That's fine for low-traffic keys. It's dangerous for hot keys: if thousands of concurrent requests hit the same expired key at once, they all miss simultaneously and hammer the DB together.
Strategy:
- Mutex/lock on rebuild — the first request past the miss recomputes the value; others wait or get a stale value.
- Logical expiration — never let the TTL actually delete the key; instead, store an internal
expires_at. On read, if it's expired, serve the stale value immediately and refresh it asynchronously in the background. [This is something we currently implement.] - Jittered TTLs so hot keys don't all expire in lockstep with each other.
4. Request for a resource that doesn't exist (cache penetration)
Someone queries an ID that isn't in the DB either — so nothing ever gets cached, and every repeat of that query bypasses the cache and hits the DB. This can be accidental, or an active exploit (bots probing random IDs).
Strategy:
- Cache the negative result too, with a short TTL, so repeated misses for the same nonexistent key stop reaching the DB. This is negative caching, and we currently have this implemented.
- Bloom filter in front of the cache/DB to cheaply reject obviously invalid keys before they ever reach the backend.
5. Entire cache cluster (server + replicas) went down
This one is severe. Thanks to the holiday-season timing, we were able to resolve it without much of a crisis — but every single request across the system ends up hitting the DB at once. This is a full avalanche, not a localized miss. In our system, we didn't have read sentinels set up properly.
Strategy:
- High availability for the cache layer itself (Sentinel/Cluster with automatic failover), so one node dying isn't a total outage.
- Circuit breaker / rate limiting in front of the DB as a last line of defense, so a cache outage degrades gracefully instead of taking the DB down too.
- A local in-process L1 cache in front of Redis can cushion the blow even if L2 is fully down.
6. Network timeout to the cache
The cache is healthy and has the key, but the client times out or the connection drops before it gets the response. Depending on your fail-open/fail-closed policy, this often silently falls through to the DB, even though the data technically wasn't missing.
Top comments (0)