It's easy to explain how a system works when everything goes right. The harder — and more
valuable — question is: what happens when it does not?
Distributed systems don't fail in infinite ways — they fail in a small number of recognizable
patterns, and the same ones keep appearing in postmortems at every company. Once you can name a
pattern, you can design against it before it ever becomes an incident. None of these need exotic
infrastructure to hit — most show up the first time a system carries real traffic.
1. Single Point of Failure
Any component on the request path with no redundancy — one load balancer, one primary
database, one message broker, one auth service. When it dies, everything behind it becomes
unreachable, even though those instances are perfectly healthy.
Clients ──► [Load Balancer] ──► App Server 1, 2, 3 (all healthy)
✗ dies
Clients ──► ✗ ──► unreachable, despite 3 healthy app servers
Fix: Redundancy + automatic failover. Run at least two instances of everything on the critical
path, put health checks in front of them, and spread instances across availability zones so one
datacenter problem can't remove an entire tier. Failover must be tested — a standby that's
never taken traffic is a guess, not a plan.
A useful habit when reviewing any architecture: trace each request path and ask which single box,
if deleted, stops all traffic. That's the box to fix first.
2. Cascading Failure
One slow dependency rarely stays contained. A database slows from 20ms to 5s → the service calling
it holds its (fixed-size) thread pool open waiting → it stops answering its own callers → they
block too. The slowness travels backwards up the call chain until the whole system looks down,
even though only one component actually degraded.
Client → Service A → Service B → Service C → DB (slow: 20ms → 5s)
↑ threads held open
↑ B's threads exhausted, stops responding
↑ A's threads exhausted, stops responding
Fix: Containment.
- Aggressive timeouts — a caller gives up rather than waiting forever.
- Circuit breakers — stop sending traffic to a dependency that's already failing.
- Bulkheads — give each dependency its own connection pool, so one sick downstream can't consume every thread the service owns.
3. Retry Storm
Retries are usually good — they stop being good when everyone retries at the same moment. A
service comes back after a blip; every client that failed during the outage retries at once, often
multiple times each. The recovering service now sees several times its normal load, falls over
again, generating a fresh round of retries. The retry logic becomes the outage.
3 clients × 3 retries each = 9x load on a service that just barely recovered → falls over again
Fix:
- Exponential backoff with jitter — spreads attempts across a window instead of synchronizing on the same second.
- Retry budget — caps retries as a percentage of total traffic.
- Circuit breaker — clients stop retrying a service that's clearly down.
4. Cache Stampede
Caches hide how much load your database really receives. When one popular key expires, every
request that would have been a cache hit becomes a miss at the same instant. All of them run
the same expensive query against the database simultaneously — which suddenly takes the full read
volume it's been shielded from for months.
Cache key expires → 1,000 concurrent requests all miss → all 1,000 hit the DB with the same query
Fix:
- Request coalescing (single-flight) — the first miss recomputes the value; the rest wait on that in-flight result instead of each triggering their own DB query.
- Refresh hot keys shortly before expiry — keeps them warm, avoiding the miss entirely.
- Add jitter to TTLs — stops thousands of keys from expiring together in the first place.
5. Hot Partition
Sharding assumes traffic spreads evenly across shards. Real traffic doesn't. Pick a
low-cardinality shard key (country, tenant, day) and one shard receives most of the writes
while others idle. That shard saturates first — and adding more shards doesn't help, because the
skew follows the key, not the machine count.
Shard key = country → Shard 1 (US): 92% of writes | Shard 2, 3: idle
Fix:
- Choose a high-cardinality, evenly-distributed key — e.g., a hash of the user ID.
- For genuinely "celebrity" entities that are hot no matter what: salt the key into sub-partitions, or give those entities dedicated capacity.
- Track per-shard metrics, not cluster averages — an average hides the one node that's on fire.
6. Replication Lag
Read replicas make reads cheap, but they're always slightly behind the primary. Under load, that
gap grows from milliseconds to seconds. A user updates their profile, the write commits on the
primary, the next read lands on a lagging replica — the change appears to have vanished. Nothing
is technically broken, but the bug report is still real.
Client writes → Primary (committed) → Replica (3s behind) → Client reads → gets STALE value
Fix: Be explicit about which reads can tolerate staleness.
- Read-your-writes routing — route to the primary for a short window after a write by that same user.
- Track replication position — only serve a read from a replica that has caught up past the write's position.
- Analytics/feed reads can safely stay on replicas — they don't need freshness.
7. Duplicate Processing
Any retried request may execute twice. The first attempt reaches the server and succeeds, the
response is lost to a timeout, and the client retries the same operation. For a read, harmless.
For a payment, an email send, or an inventory decrement, the user is charged twice and the
ledger is wrong.
Client → charge $100 → Server processes, charges $100 → response lost (timeout)
Client retries → charge $100 → Server charges $100 AGAIN → $200 total charged
Fix: Idempotency. Client attaches a unique idempotency key to each logical operation; server
records that key with the result. When a request arrives carrying a key it's already processed,
the server skips the work and returns the original response — same request, same outcome, any
number of attempts.
8. Queue Backlog
Queues absorb bursts, which is exactly why they hide problems. If producers publish faster than
consumers process — even by a small margin — the backlog grows continuously. Every health check
stays green while the age of the oldest message climbs from seconds to hours, and by the time
anyone notices, the data being processed is badly out of date.
Producers: 10,000 msgs/sec | Consumers: 2,000 msgs/sec → queue grows by 8,000/sec, forever
Fix: Make the imbalance visible and bounded.
- Alert on consumer lag and oldest-message age, not just queue length alone.
- Autoscale consumers on lag.
- Bound the queue so it applies backpressure, and shed low-value work under sustained overload instead of accepting everything and falling further behind.
9. Poison Message
One malformed message can stop an entire queue. The consumer picks it up, throws, and never
acknowledges it. The broker redelivers it, the consumer throws again, and the loop repeats
forever. Throughput drops to zero while every valid message behind the bad one waits — and the
crash loop can take the whole consumer group down with it.
[bad msg] [msg2] [msg3] ... → consumer crashes on bad msg → redelivered → crashes again → forever
↑ msg2, msg3... never processed
Fix:
- Cap delivery attempts, then move the message to a Dead Letter Queue so the consumer can continue past it.
- Alert on anything landing in the DLQ.
- Keep enough context on the message to fix the underlying bug and replay it later.
10. Split Brain
During a network partition, two nodes can both conclude they are the primary. Each accepts
writes from the clients that can still reach it, and the two copies of the data diverge. When the
partition heals, there's no correct way to merge them — both sides accepted writes assuming they
were authoritative.
Network partition:
Node A (thinks it's primary) ← writes from Client 1
Node B (thinks it's primary) ← writes from Client 2
→ data diverges, no clean merge once partition heals
Fix: Make leadership require a majority.
- Quorum-based writes — only the side holding more than half the nodes can commit.
- Consensus-based leader election (e.g., Raft) — prevents two leaders being elected in the same term.
- Fencing tokens — let storage reject writes from a leader whose lease has already expired.
Patterns Worth Pairing Together
A few of these failure modes naturally show up as a package when you propose a design:
- Add a cache → think about stampede protection up front.
- Add sharding → name your shard key and why it won't go hot.
- Add retries → pair them with backoff and jitter.
- Add replicas → decide which reads can tolerate lag.
Anyone can draw boxes on a diagram — the useful skill is explaining how those boxes behave on
the worst day of the quarter.
Quick Reference Table
| # | Failure Mode | One-line cause | One-line fix |
|---|---|---|---|
| 1 | Single Point of Failure | No redundancy on a critical-path component | Redundancy + tested automatic failover |
| 2 | Cascading Failure | Slow dependency exhausts caller's thread pool, propagates upstream | Timeouts, circuit breakers, bulkheads |
| 3 | Retry Storm | Synchronized retries multiply load on a just-recovered service | Exponential backoff + jitter, retry budget, circuit breaker |
| 4 | Cache Stampede | Popular key expires, all misses hit DB simultaneously | Request coalescing, early refresh, jittered TTLs |
| 5 | Hot Partition | Low-cardinality shard key skews load to one shard | High-cardinality key, salting, per-shard metrics |
| 6 | Replication Lag | Replica behind primary, stale read after a write | Read-your-writes routing, replication-position tracking |
| 7 | Duplicate Processing | Retried request re-executes a non-idempotent side effect | Idempotency keys |
| 8 | Queue Backlog | Producer rate > consumer rate, backlog grows silently | Lag-based alerting/autoscaling, bounded queue, load shedding |
| 9 | Poison Message | Malformed message crash-loops the consumer forever | Capped attempts → Dead Letter Queue |
| 10 | Split Brain | Partition causes two nodes to both act as primary | Quorum writes, consensus leader election, fencing tokens |
Summary
Distributed systems fail in a recurring, nameable set of ways: missing redundancy, failures
propagating through tight coupling (timeouts/thread pools), synchronized client behavior (retries,
cache expiry), uneven data/traffic distribution, staleness from replication, non-idempotent
side effects, and disagreements about leadership during partitions. Each one has a well-understood
fix — redundancy, timeouts/circuit breakers/bulkheads, backoff with jitter, request coalescing,
better shard keys, staleness-aware read routing, idempotency keys, lag-based backpressure, dead
letter queues, and quorum-based consensus. Designing a system well means assuming each of these
will eventually happen, and deciding in advance how the system should behave when it does.
Top comments (0)