Most of this series has been about what happens when caching works: request, hit, fast response. But production doesn't stay in that state forever. Redis becomes unavailable, a network path breaks, a node crashes, connections exhaust, a deployment clears the cache, a config mistake causes mass expiration, a region goes dark. Eventually, cache goes down.
The question that actually matters is what happens to the application next. A poorly designed system turns a Redis outage into database overload into an application outage, in seconds. A well-designed one turns it into degraded performance, a protected database, and an application that stays up. That gap is the entire subject of this post.
1. The most important principle
A cache should usually be an optimization, not the source of truth. If Redis disappears and the database still holds the authoritative data, Application → Database remains a valid path even with Redis gone. That's the theory. The practice is harder: what happens when 100,000 requests/sec were relying on Redis catching most of them, and it suddenly can't?
2. The cache failure avalanche
Under normal conditions, 100,000 requests/sec against a 95% hit ratio means the database sees roughly 5,000/sec — comfortable. The instant Redis fails, all 100,000 requests fall through, because there's no cache left to catch any of them. That's not a proportional increase. It's the full, unfiltered load the cache was built specifically to prevent, arriving all at once.
3. Why the database can take the whole system down with it
If the database comfortably handles 10,000 requests/sec but Redis was normally absorbing enough that the database only saw a fraction of 100,000 total traffic, a sudden 100,000-request flood is 10x what the database can actually process. From there the chain writes itself: the database slows down, application requests queue, connection pools fill, threads wait, timeouts rise, clients retry, and the retries add more load on top of an already-struggling system. This is a cascading failure — the cache didn't just fail, its failure propagated.
4. Retries make this worse, not better
A request that times out against Redis and retries twice more has turned one logical request into three actual ones. Multiply that across 100,000 requests and you've turned 100,000 requests into 300,000 — against a dependency that was already failing before the retries even started. Retries sound protective. Against an already-overloaded system, they're an amplifier.
5. Timeouts decide how long the damage takes to start
A 10-second Redis timeout sounds conservative, but it means 100,000 requests can all be waiting on Redis simultaneously, holding threads and connections the whole time. A short timeout — 50ms, not 10 seconds — lets a request fail fast and move to its fallback instead of sitting there. The principle holds regardless of the exact number: a dependency that's already failing shouldn't be allowed to hold your application hostage while it does.
6. Fail open vs. fail closed
When Redis is unavailable, should the application keep going without it, or refuse the operation entirely? Both are legitimate answers, and which one is correct depends entirely on what the cache was actually doing.
A product description can fail open — skip the cache, hit the database, the user gets a slightly slower but correct response. Rate limiting is the opposite case: if you can't verify whether a user has exceeded their limit, silently allowing every request through removes a protection you were relying on. Failing closed there — rejecting or restricting the request — is the safer default. Neither choice is universally right; the purpose behind the cache is what decides.
7. Cache as optimization vs. cache as infrastructure
This distinction matters more than it first appears. When Redis is purely making reads faster, its disappearance still leaves Application → Database intact. But Redis is often doing more than caching — sessions, rate limiting, distributed locks, queues, coordination. At that point it's not a cache anymore, it's infrastructure, and its failure needs to be handled with the seriousness of losing infrastructure, not the shrug you'd give a slow cache. Don't assume every Redis dependency has a database fallback waiting behind it — some of them don't have one at all.
8. Circuit breakers
Sending every one of 10,000 failing requests to a Redis that's already down accomplishes nothing except adding load to something already struggling. A circuit breaker recognizes the failure pattern and stops sending traffic to Redis at all, routing everything straight to the fallback instead.
CLOSED is business as usual. OPEN means Redis isn't contacted at all — every request goes straight to its fallback. HALF-OPEN is the recovery test: one request is allowed through after a cooldown period, and its outcome decides whether the circuit reopens fully or closes back to normal. The point of the whole mechanism isn't detecting that Redis failed — it's not re-detecting the same failure 100,000 times a second while Redis is trying to recover.
9. Graceful degradation
The system doesn't need to be perfect. It needs to remain useful. If Redis goes down and an e-commerce homepage normally shows product info, search, checkout, recommendations, and personalized offers, taking down the entire page because recommendations can't load is a self-inflicted wound. Product information, search, and checkout can stay up; recommendations and personalized offers can show a fallback or simply not render. The user still has a working site.
10. Sorting data into critical, important, and optional
This classification should directly shape your fallback strategy. Critical data — authorization, payment validation, certain security controls — means the operation genuinely cannot proceed safely without it. Important data — recommendations, personalization, analytics — means the operation continues with reduced functionality. Optional data — a trending widget, recently viewed items, decorative content — the user may not even notice its absence. Knowing which bucket a piece of data falls into, before the incident happens, is what makes graceful degradation possible instead of theoretical.
11. Serving stale data on purpose during an outage
If a local, slightly-old copy exists, serving it can beat going all the way to the database. Trending products that are 10 minutes stale — the user probably never notices. An account balance served 10 minutes stale is a completely different kind of problem. The decision about which data gets this treatment during an outage is the same staleness-tolerance question from Part 13, just applied under failure conditions instead of normal ones.
12. Cache miss and cache failure are not the same thing
A miss means the cache is working correctly and simply doesn't have this key — GET product:123 returning nil is expected, routine behavior. A failure means you couldn't reliably talk to the cache at all — a connection timeout is not the same category of event, and treating it identically to a miss hides a real problem. A miss needs a database query. A failure may need a fallback, a circuit breaker trip, an alert, and traffic protection — a much bigger response than "just go to the database this once."
13. Don't keep trying to write to a cache that's already down
Here's a subtle waste: Redis is unhealthy, the application correctly falls back to the database, and then still tries Redis SET to repopulate the cache — which also fails, on every single request, for no benefit. Once the system knows Redis is unhealthy, it can stop attempting cache writes entirely until the circuit breaker's test request confirms Redis is actually back.
14. Protecting the database is the actual priority during an outage
Connection pool limits, concurrency limits, request queues, rate limiting, circuit breakers, load shedding, and fallback responses all exist to keep the database from being the thing that turns a cache incident into a full outage. Deliberately capping how many requests reach the database — say, 10,000 out of 100,000 — and returning a controlled response to the rest sounds harsh compared to "let everything through." But 10,000 succeeding while 90,000 get a controlled failure is a materially better outcome than all 100,000 requests taking down the database and leaving zero requests succeeding.
15. Load shedding is a deliberate design choice, not a failure
If database capacity is 10,000 req/sec and incoming traffic is 50,000, deliberately routing 10,000 through and rejecting or deferring the other 40,000 is what keeps the system available at all. A resilient architecture treats "no" as a legitimate, planned response — not a symptom of something broken.
16. Recovery can recreate the exact problem you just survived
Redis comes back online with an empty cache, and every application instance immediately tries to refill it — which, at scale, is just the original stampede happening again, this time triggered by recovery instead of failure. Cache warming, request coalescing, jittered expiration, and rate-limited rebuilding — the Part 7 stampede defenses — apply just as much to the moment a cache comes back as to the moment a popular key expires.
17. Redis Cluster failure and its own subtleties
Replication lets a cluster fail over from a failed primary to a replica, reducing downtime — but failover isn't instantaneous or invisible. There can be a failover delay, transient connection errors, temporarily unavailable keys, and replication lag even after the new primary takes over. Applications still need to handle these as real, if brief, failure conditions rather than assuming clustering makes failure disappear.
18. Network failure can look exactly like Redis failure
Redis can be perfectly healthy while your application simply can't reach it — a network partition looks identical to a dead Redis from the application's point of view. This is why monitoring Redis server health alone isn't sufficient: Redis server health: GOOD and Application-observed Redis latency: BAD can both be true simultaneously, and only the application's own perspective catches the second one.
19. Run the capacity math before an incident forces you to
At 50,000 requests/sec and a 98% hit ratio, the database normally sees roughly 1,000/sec — well within a 5,000/sec capacity. If Redis fails, the database faces the full 50,000/sec: 50 times its normal load, and 10 times more than it can actually process. This is exactly the kind of calculation that belongs in an architecture review, done deliberately with real numbers, rather than discovered for the first time during the incident itself.
20. What an incident actually looks like end to end
Watched together rather than separately, these five numbers tell a single causal story: cache hit ratio collapses, database traffic spikes in direct response, CPU saturates under that load, and application latency follows immediately after. Knowing "Redis is at 80% CPU" in isolation tells you far less than seeing this whole chain — which is why cache miss rate correlated against database QPS is one of the most valuable signals a caching system can expose.
21. Don't retry everything, indefinitely
The failure mode to avoid: retry, fail, retry, fail, retry, fail, with no bound and no growing delay. The alternative is bounded retries, short timeouts, exponential backoff, jitter on that backoff, and a circuit breaker that eventually stops the attempts altogether. The goal was never "keep trying until it works" — it's "give the dependency one reasonable chance without piling onto its failure or ours."
22. Cache failure needs to actually be tested
Most test suites, per Part 10, cover the cache working. Far fewer cover it not working: Redis unavailable, Redis slow, connection exhaustion, a node failing outright, elevated network latency, an empty cache, mass expiration, a slow database, an unavailable database. The question every one of those scenarios needs answered isn't "does this pass" — it's "does the application remain useful."
23. Chaos testing, for systems that can support it
Mature systems can go further and deliberately inject a Redis failure in a controlled environment, then observe: does traffic actually move to the database, does the database survive it, do circuit breakers open when they should, do alerts fire, does recovery behave the way it's supposed to. The goal was never "prove nothing fails" — that's not a realistic goal for any real system. It's "know exactly how the system behaves when something does," ahead of the moment you need that knowledge.
The golden rule
Never assume a cache outage is harmless just because the cache isn't your source of truth. The cache may be absorbing 95%, 98%, or more of your total traffic — remove it suddenly, and the database inherits all of it at once. The question to ask when designing any cache is "what happens if this disappears right now" — and the answer should come from a calculation, not a guess.
The bigger lesson
Caching gets introduced to make a system faster. Once a system becomes dependent on that cache, caching also becomes a resilience question — and the failure chain (cache failure → misses → database overload → application slowdown → timeouts → retries → more load → cascading failure) is entirely predictable in shape, even if the trigger varies.
A resilient system breaks that chain with timeouts, circuit breakers, fallbacks, concurrency limits, load shedding, graceful degradation, cache warming, and real observability — not as separate features bolted on, but as one coordinated resilience strategy. The goal was never making the cache impossible to fail. That's not achievable. The goal is making the application capable of surviving the failure when it happens — and that distinction is what separates a caching implementation from a production-grade caching architecture.
Has your system been through a real cache outage — and looking back, was the actual failure mode the one you'd already planned for, or something the runbook hadn't considered?
Top comments (0)