Hello, I'm Maneshwar, and I'm building LiveReview — a blast-radius aware AI code review built for your business-critical systems. Star us to help devs discover the project, give it a try, and share your feedback to help improve the product.
How does Amazon stay up during Black Friday when a normal server would be on fire by 9am? Why do banks rarely lose a transaction even when a data center loses power mid-transfer?
The honest answer is not "they don't have failures." Everyone has failures.
Networks partition, disks die, servers get evicted by a bored Kubernetes scheduler for no reason anyone can explain.
Reliability isn't about preventing that.
It's about the system still doing the right thing while it's happening.
There are seven ideas that keep showing up whenever you dig into how large systems actually stay reliable.
I went through and a couple of them connect straight back to Kademlia and XOR distance, which I wrote about a while back.
Small world.
1. The CAP Theorem: pick your poison
CAP says a distributed system can give you at most two of these three, at the same time:
- Consistency: every node sees the same data at the same time
- Availability: every request to a non-failing node gets a response
- Partition tolerance: the system keeps working even when the network splits
Here's the part people skip past: partitions are not optional.
Cables get cut, switches die, cloud regions lose connectivity to each other.
Partition tolerance isn't a feature you choose, it's a fact of networked life.
So in practice, CAP quietly becomes "choose consistency or availability, for the duration of the partition."
Two real systems, two real answers:
- Google Spanner picks consistency. It uses atomic clocks and GPS-synced time (the actual TrueTime API) to keep transactions linearizable across continents. During a partition, the majority side keeps serving reads and writes, the minority side drops to read-only until things heal.
- Amazon DynamoDB picks availability. It keeps accepting writes during a partition and resolves conflicting updates afterward using timestamps. You always get a response. Sometimes it's a slightly stale one.
Neither is wrong. A bank ledger wants Spanner's paranoia.
Your Twitter feed does not need to be linearizable, it needs to not go down.
Here's what "consistency" actually looks like on the wire: every node holding the exact same value at the exact same time, no stragglers, no stale reads sneaking through.
That guarantee is expensive, which is exactly why not everyone pays for it.
Now flip to availability. Here the system cares more about answering something than answering the latest thing.
Notice the older values (t-1, t-2) still floating around some nodes below. That's the trade Dynamo makes on purpose, every request gets a response, even if it's a slightly out-of-date one.
And then there's the partition itself, the thing that forces the choice in the first place.
Below, a chunk of the ring has gone unreachable (the red X nodes). The surviving majority partition keeps talking to itself in teal, while the minority is cut off entirely, that's the moment Spanner would start rejecting writes and Dynamo would keep serving them.
Put those three together and here's the decision every single node is quietly making the instant a partition happens, boiled down to one flowchart.
If it can reach a quorum, life goes on as normal. If it can't, it has to pick a lane, reject the request or serve something possibly stale, and that one branch is the entire CAP theorem in practice.
2. Eventual consistency: it'll get there, chill
Eventual consistency makes a deceptively small promise: if you stop writing, every replica will eventually agree. That's it. No promise about when.
That sounds sloppy until you realize what it buys you.
A write can return immediately without waiting for every replica to confirm, which is why Amazon's shopping cart lets you add an item even if a couple of backend servers are having a bad day.
Your cart write doesn't block on all of them.
The obvious follow-up question: what happens when two replicas get conflicting updates? Three common answers:
- Last write wins (LWW): pick whichever update has the newer timestamp. Simple, and simple to lose data with, since "newer" depends on whose clock you trust.
- CRDTs (conflict-free replicated data types): data structures that are mathematically guaranteed to converge no matter what order updates arrive in. No coordination needed, the math just works out.
- Application-defined merges: you write the merge logic yourself, using whatever your business actually needs (e.g. "cart merge = union of items, not last-write-wins").
def merge_last_write_wins(local, remote):
# simplest possible conflict resolution: newer timestamp survives
return remote if remote.timestamp > local.timestamp else local
DynamoDB typically converges within milliseconds under normal load, which is why "eventual" feels instant almost all the time and only bites you during actual network weirdness.
Here's the shape of a typical eventually-consistent write path: client talks to a server, server talks to a primary, and the primary is the one source of truth everything else copies from.
Nothing below the primary is guaranteed to be caught up the instant you write, it's guaranteed to get there.
Zoom into that last hop and you can see the replication itself happening as four discrete steps: the write lands on the primary, gets acknowledged back to the caller, and only then gets pushed out to the replicas sitting behind it.
That gap between step 2 (caller gets its answer) and steps 3-4 (replicas actually catching up) is the entire "eventual" in eventual consistency, and it's usually measured in milliseconds, not minutes.
And yes, on a bad day that gap can stretch a lot further than milliseconds, which is basically this entire meme.
3. Load balancing: the bouncer at the door
Load balancers spread incoming requests across servers, and the "simple" part of that sentence is doing a lot of lying.
There are two layers to know:
- Layer 4 balancers route on IP address and TCP/UDP port. Fast, because they never look inside the packet.
- Layer 7 balancers read HTTP headers, URLs, even the request body, and route based on that. Smarter, more expensive.
And routing algorithms have gotten past plain round robin:
- Least connections: send the new request to whichever server currently has the fewest active connections
- Least response time: same idea, but also factors in how fast each server has been responding recently, so a technically-idle-but-slow server doesn't get flooded
Load balancers themselves need to not be a single point of failure, so they're usually deployed as a primary/secondary pair with a heartbeat, failing over in milliseconds if the primary drops.
Consistent hashing (up next) is often what keeps a given client landing on the same backend server every time, which matters a lot if that server is holding session state in memory.
At its simplest, this is the whole picture: two clients, one load balancer, three servers, and a routing decision made per request.
Peek inside the load balancer itself and it's just a process terminating TCP/UDP connections and forwarding them onward, usually bound to one IP and port that every client hits.
And here's round robin specifically doing its thing over a few requests, cycling evenly through Server A, B, and C regardless of how loaded any of them actually are.
That "regardless of load" part is exactly why least-connections and least-response-time exist as smarter alternatives.
4. Consistent hashing: don't reshuffle the whole deck
Quick problem statement: you've horizontally scaled your data across N nodes.
Now you want to add or remove a node without moving nearly all of your data around.
Plain modular hashing (node = hash(key) % N) fails at this spectacularly. Change N by one, and almost every key maps to a different node.
That's a full data migration triggered by adding a single server.
Consistent hashing fixes this with a neat trick: put both the nodes and the keys on the same circular hash ring.
- Hash each node to get its position on the ring
- To find where a key lives, hash the key and walk clockwise until you hit the first node
- Replicate to the next N-1 nodes clockwise, for redundancy
Add a node, and it only takes over keys from its immediate neighbor. Remove one, and its keys shift to the next node over.
Instead of remapping practically everything, you move roughly K/N keys, where K is total keys and N is node count.
DynamoDB and Cassandra both lean on exactly this.
If "hash space" and "ring" and "closest node wins" sound familiar, that's because Kademlia is solving a strikingly similar problem for peer discovery, just with XOR distance instead of clockwise distance on a ring. Different metric, same underlying move: stop routing through a central authority, let structure do the work.
Here's the ring itself: every server hashed onto a fixed position, and every key just walking clockwise until it finds a server to land on.
And here's the actual assignment happening for a handful of keys across three nodes, with the K/n math spelled out, four keys, three nodes, so each node ends up owning roughly one and a third keys' worth of the ring.
5. Circuit breakers: fail fast, on purpose
Here's a failure mode that's sneakier than it sounds: one slow service starts a cascade. Service A calls Service B, B is struggling, so A's requests start piling up waiting on B, A's own threads exhaust, and now A is down too, even though A's own code was fine.
Circuit breakers stop this by giving up on purpose, fast. Three states:
- Closed: requests flow through normally
- Open: requests fail immediately, without even attempting the call, buying the failing service time to recover
- Half-open: a few test requests are let through to check if the service has recovered
if failure_rate > threshold:
state = OPEN # stop calling, fail fast
elif state == OPEN and timeout_elapsed:
state = HALF_OPEN # let a few test requests through
elif state == HALF_OPEN and test_requests_succeeded:
state = CLOSED # back to normal
elif state == HALF_OPEN and test_requests_failed:
state = OPEN # nope, not yet
Netflix popularized this pattern hard with Hystrix (now in maintenance mode, but the pattern lives on in things like resilience4j), and it's basically table stakes in microservices now.
The core insight is almost counterintuitive: a fast failure is a feature, because a slow failure quietly drains resources from everything downstream of it.
Here's all three states sitting side by side against the same upstream/downstream pair, so you can see exactly what changes as the breaker trips: closed lets traffic through both ways, open blocks it outright, half-open cracks the door back open just a little.
And here's the same three states redrawn as an actual state machine, with the transitions labeled.
This is the version worth memorizing, since it's basically the spec for how you'd implement one yourself.
Same state machine, this time as a plain flowchart if you just want the transition logic without the visual states:
6. Rate limiting: no, you may not send 10,000 requests a second
Rate limiting caps how many requests a client can make in a given window, protecting you from both overload and outright abuse. There's more than one way to do it (token bucket, leaky bucket, fixed window, sliding window), and the differences matter enough that I already wrote a whole separate post walking through all of them, token buckets included:
7. Monitoring: you can't fix what you can't see
Everything above is useless if you find out it broke from a tweet. Modern observability leans on four signal types:
- Metrics: numeric time series, CPU, request rate, error counts
- Logs: structured records of individual events, with context
- Traces: the path a single request takes across your whole distributed system
- Events: notable one-off occurrences, deployments, config changes
The hard part isn't collecting this, it's alerting on it without drowning yourself. Static thresholds are fine until your traffic pattern changes and they start firing constantly for no real reason.
Better systems use statistical anomaly detection that learns what "normal" looks like and flags deviations from it, and combine signals into composite alerts (high CPU and rising errors and slow responses, not any one alone) to cut noise.
The actual north star here is the SLO, a measure of what your users actually experience, not a graph that only makes sense to the person who drew it.
You probably don't need all seven
A web app serving a few thousand users doesn't need consistent hashing.
It almost certainly doesn't need a hand-rolled circuit breaker library either.
These seven patterns aren't a checklist you tick off on day one, they're tools you reach for once you have actual evidence you need them, not because a blog post (this one included) told you to.
Start simple. Measure everything. Add complexity only when the evidence, not the fear, tells you to. That's the whole game.
Your team's attention is limited, and the deluge of AI-generated code is making it harder to keep production code safe without slowing you down.
I'm building LiveReview, a blast-radius aware AI code review built for your business-critical systems.
Instead of presenting every diff with equal emphasis, LiveReview scores each change by blast radius — how far its impact reaches through your call graph — so you can focus attention where it actually matters.
Spend code review effort where business risk is highest — not spread evenly across every diff.
Try LiveReview on your codebase:
Top comments (0)