Your Redundancy Math Assumes Independence. Production Doesn't.

The problem

Every redundancy calculation you've ever been handed — RAID fault tolerance, N+1 replica math, "three nines times three nines," retry-until-success — leans on one silent assumption: that the failures being multiplied together are independent events.

They usually aren't. And the gap between the math and reality is exactly where the 2 a.m. pages come from.

Why it happens

Take a RAID array everyone treats as "safe because it tolerates one drive failure." The textbook argument: if a single drive has a 2% annual failure rate, the odds of two specific drives failing at once are roughly that number squared — vanishingly small. Nobody loses sleep over a one-in-thousands event.

Here's the math on what actually happens once a drive fails and the rebuild starts:

Per-drive rebuild-window failure prob (independent):  0.00011  (~0.01%)
P(any second failure during rebuild), independent:     0.00077  (~0.08%)
Correlated conditional second-failure prob range:      0.15 - 0.35  (15-35%)
Ratio vs independent model:                            ~200x - 450x

The independent model says "any second failure during the rebuild window" is roughly a 1-in-1,300 event. But a rebuild isn't a quiet passive period — it's a full-surface read of every remaining drive, all at once, under sustained load. If those drives came from the same manufacturing batch (they almost always did — you bought them together), they share the same wear curve, the same firmware, the same thermal environment. The rebuild doesn't just wait for an independent second failure to happen to arrive — it actively creates the shared stress event that triggers one. Field data on large-array rebuilds puts that conditional risk in the 15-35% range, not 0.08%. That's not a rounding error; it's a 200-450x miss.

This is the same failure of imagination that produces retry storms (a shared timeout misconfig means retries aren't independent attempts, they're a synchronized herd), that makes "three nines of compute times three nines of network" wildly optimistic (both often go down together because they share a rack, a switch, or a cloud AZ), and that quietly invalidates a lot of fan-out latency modeling (slow servers cluster around shared causes — a noisy neighbor, a GC pause propagating through a connection pool — rather than failing as independent coin flips).

The pattern: multiplying probabilities is only valid when the events don't share a cause. In production, almost everything shares a cause if you look one layer up.

What to do about it

You can't out-math correlation, but you can design against it:

1. Ask "what do these share?" before you trust the math. Same batch of drives, same AZ, same upstream dependency, same deploy, same on-call engineer who fat-fingers the same runbook step — any shared factor is a channel for correlated failure. Redundancy without diversity is theater.

2. Break the shared cause deliberately. Stagger drive purchase batches. Spread replicas across AZs and, for anything critical, across providers. Jitter your retries so a shared trigger doesn't produce a synchronized herd. Canary and stagger deploys instead of pushing to 100% at once.

3. Test the correlated case, not just the independent one. Chaos-test by killing the shared dependency, not just a random node. If your DR plan has never survived "the AZ that holds your primary and your replica both go dark," you don't actually know your real fault tolerance — you know your spreadsheet's.

4. Treat rebuild/recovery windows as the highest-risk moment, not a safe one. RAID rebuilds, replica re-syncs, and cache warm-ups all impose the exact kind of correlated stress that turns a "safe" single failure into a double one. Budget extra caution (and extra monitoring) for that window specifically.

Key takeaways

• Redundancy math (RAID, N+1, retries, five-nines stacking) is usually independence math wearing a trench coat.
• Correlated failure modes — shared batch, shared rack, shared dependency, shared deploy — can push real risk 100-400x above the naive calculation.
• Recovery windows (rebuilds, re-syncs) don't just wait for a second failure; they actively manufacture the shared stress that causes one.
• The fix isn't better arithmetic. It's designing for diversity of failure cause, and testing the correlated scenario deliberately.