This is Part 13 and Finale part of my "From One User to One Million" series, where we'll build an understanding of System Design by following a simple application as it grows from a single user to millions. Instead of memorising technologies, we'll learn why they exist by solving real problems as they appear.
We've spent twelve articles asking the same question in different forms.
How do we make the system handle more?
More traffic. More data. More reads. More writes. More users, spread across more of the world, asking for more things, all at once.
Every time we hit a limit, we found what was too concentrated and we distributed it. One server became many. One database became a primary with replicas. One giant dataset became shards. One application became services. Content moved from one origin to edge locations near the users who needed it.
The architecture we've built can scale. It can absorb traffic spikes. It can serve millions of users simultaneously across multiple continents.
But Part 12 ended by asking something we've never asked before.
We've built a system that can handle scale. But what happens when part of it simply stops working?
That's the question this article answers. And it turns out, it changes how you think about everything.
--
Section 1: What Happens When a Server Dies?
Let's start with the most ordinary failure imaginable.
An application server dies. Not a slow leak, not a gradual degradation. It just disappears. The process stops. The machine goes dark.
If that server is the only one running the application, the answer is simple and bad.
User
|
v
Server ❌
The application is down.
Users get errors.
Nothing works until someone notices and restarts it.
But this is actually a problem we already solved, back in Part 4. We added more application servers and put a load balancer in front of them. We weren't thinking about failure at the time. We were thinking about traffic.
.-- App Server 1 ❌
User --> Load Balancer --> App Server 2
'-- App Server 3
Server 1 dies. The load balancer, which periodically checks whether each server is healthy, notices that Server 1 isn't responding. It stops sending traffic there. Requests go to Server 2 and Server 3 instead.
The users connected to Server 1 at the moment it failed might experience a brief error or a dropped request. But the overall system keeps running. A few seconds later, everything looks normal again.
This is the most basic form of redundancy: having more than one of something, so that when one fails, the others carry on.
We added redundancy to the application layer for performance reasons. It turns out that same redundancy also makes the system more resilient to failure. The two benefits came together for free.
But redundancy only helps if the thing that fails isn't the only path through the system.
--
Section 2: The Single Point of Failure
Now let's make the failure harder.
The load balancer is routing traffic across three healthy application servers. The cache is working. The message queue is draining normally.
Then the database fails.
App Server 1 --.
App Server 2 --+--> Database ❌
App Server 3 --'
It doesn't matter that we have three application servers. Every single one of them depends on that database. Without it, they can't read user data. They can't save orders. They can't authenticate logins. They can receive requests just fine. They have nowhere to send them.
Three healthy servers. Zero working application.
This is what engineers call a single point of failure: any component whose failure alone is enough to bring the entire system down.
The question that should now become second nature is this: if this component disappeared right now, would the entire system disappear with it?
Ask it about every component in your architecture.
The load balancer: if there's only one, and it fails, every user is disconnected. Single point of failure.
The database: if there's only one, and it fails, every application server is blind. Single point of failure.
The message queue: if there's only one and it fails, background work stops accumulating and workers have nothing to process.
The cache: less critical, since a failing cache is survivable if the database can absorb the extra reads. But still worth considering.
Single points of failure are everywhere in early architectures, often invisible because we build for the happy path. Everything is fine until it isn't, and then the single point that failed takes everything with it.
--
Section 3: Don't Let One Failure Take Everything Down
The response to a single point of failure is the same idea as the response to a traffic bottleneck: don't let one instance carry all the weight.
For application servers, we already have this. Multiple servers share the load.
For the database, we also already have this, in a different form. Read replicas were introduced for performance. But a replica is also a copy of the data. If the primary database fails, and there's a replica that's been kept in sync, the situation isn't as permanent as it first appears.
The key question is: what happens to that replica when the primary disappears?
If nothing automated happens, the replica just sits there, receiving no new data, while the application falls over. Engineers have to manually promote the replica to primary, update connection strings, restart services, and redirect traffic. That can take minutes or hours. For most applications, that's an unacceptable outage.
But if the system is designed to handle this automatically, something different happens.
The primary fails. Within seconds, a monitoring process detects the failure. The replica is automatically promoted to become the new primary. Application servers are redirected to the new primary. Traffic resumes.
Users might experience a brief interruption, a few seconds of errors, while the switchover happens. But the application comes back on its own, without anyone being paged at 3am to manually fix it.
This automatic process of moving from a failed component to a healthy replacement is called failover.
Primary Database fails:
Before failover:
App Servers --> Primary ❌
Replica (idle, waiting)
After failover:
App Servers --> New Primary (promoted from replica)
Failover only works if there's something to fail over to. That means redundancy has to be built in before the failure happens, not scrambled together after. You can't provision a replica in the middle of an outage and expect it to help immediately. The replica has to already exist, already be in sync, and already be ready to take over.
This is what it means to design for failure: not fixing things after they break, but building in the capacity to survive breakage before it occurs.
--
Section 4: High Availability and Failover
There's a concept that captures this design philosophy in a phrase: high availability.
A system is highly available when it continues to serve users even when individual components fail. Not because those components never fail. Because the system was built assuming they would.
High availability doesn't mean zero downtime. That's an unrealistic standard. Hardware fails. Software has bugs. Networks have hiccups. Deployments go wrong. Claiming a system will never have a moment of unavailability is almost never true.
What high availability does mean is that failures are absorbed. The system detects them, routes around them, and keeps working. Not every failure mode can be absorbed, but the common ones can be designed for.
The practical mechanisms that make this work are ones we've already touched on:
Health checks let the load balancer and other components know which servers are responding and which aren't. A server that stops answering health checks gets marked as unhealthy and taken out of rotation before users are routed to it.
Automatic failover means that when a primary database goes down, a replica takes its place without waiting for human intervention.
Multiple availability zones mean that if an entire data center loses power or network connectivity, services running in other locations continue serving traffic. Your application doesn't have to be in one physical place.
None of these are exotic techniques. They're standard practice in systems designed to stay up. What makes them effective isn't any individual mechanism. It's the mindset that produces them: the deliberate assumption that any given component can fail at any given moment, and the architecture built around that assumption.
--
Section 5: When the Network Fails Too
So far we've talked about components disappearing entirely. But there's a category of failure that's often harder to handle: partial failure.
A server isn't dead. It's just slow. A network isn't down. It's dropping some packets. A downstream service isn't unavailable. It's responding, but taking five seconds per request instead of fifty milliseconds.
These partial failures are treacherous because they don't trigger the clean detection that a complete outage does. The health check passes. The server responds. But every request that touches it takes five seconds, and those slow requests start backing up, consuming threads, and eventually making the service that depends on it look sick too.
A few techniques exist to contain this kind of failure.
Timeouts are the simplest defense. Instead of waiting indefinitely for a response from a slow service, set a limit. If the response doesn't arrive within 500 milliseconds, give up and return an error. This prevents one slow dependency from holding every request hostage.
Retries can help when a failure is likely to be temporary. A network hiccup that drops one packet is often resolved by trying again. But retries have a catch that matters enough to spend a moment on.
--
Section 6: Retries Can Make Things Worse
Imagine a user clicks "Pay Now." The Order Service calls the Payment Service. The Payment Service processes the payment successfully and sends a response. That response gets lost somewhere in the network. The Order Service never receives it.
From the Order Service's perspective, the payment request timed out. Should it retry?
If it retries, the Payment Service receives a second request to process the same payment. This time it might succeed and send a response that arrives. The payment is processed and the order goes through. But the user's card was charged twice.
This isn't a theoretical edge case. It happens. And the solution requires thinking carefully about what it means to run an operation more than once.
An operation is idempotent if running it multiple times produces the same result as running it once. Some operations are naturally idempotent. Reading a user's profile twice returns the same profile. But charging a payment card twice doesn't produce the same result as charging it once.
Designing retries safely means designing the operations being retried to be idempotent where possible: using unique identifiers for payment requests so the Payment Service can detect duplicates and refuse to process the same payment twice, even if it receives the request multiple times.
This is the same complexity that Part 12 introduced when services started communicating over a network. The failure-handling layer inherits those problems and has to solve them deliberately.
Recovering from failure is itself a design problem.
A related technique is the circuit breaker. If a downstream service is failing consistently, retrying rapidly can make things worse: flooding an already struggling service with repeated requests. A circuit breaker tracks how often calls to a service are failing. Once failures exceed a threshold, it stops sending requests to that service entirely for a period of time, letting it recover rather than hammering it. When the cooldown expires, it tries again cautiously.
These aren't exotic patterns. They're standard tools for building systems that survive the messiness of real-world distributed operation.
--
Section 7: Designing for Failure
At this point it might seem like the answer is: add redundancy everywhere, retry everything, set timeouts on every call, add circuit breakers to every dependency.
But redundancy isn't free.
Every replica is another machine to pay for, another machine to keep in sync, another machine to monitor. Running services across multiple availability zones doubles the infrastructure cost. The operational complexity of managing failover, tracking health across many components, and debugging distributed failures across redundant systems is real and significant.
The right question isn't "how do we achieve maximum redundancy?" The right question is "what failure modes does this specific system need to survive, and what's the cost of not surviving them?"
A small internal tool used by twenty people during business hours can tolerate several hours of downtime. The cost of building high availability into it is almost certainly greater than the cost of the occasional outage. A basic health check and a single database is probably fine.
A global e-commerce platform processing payments around the clock cannot afford minutes of downtime without losing significant revenue and user trust. Multi-region failover, replica databases, circuit breakers, and automated recovery are worth the cost because the alternative is worse.
These are two genuinely different answers to the same question, and both are correct for their context.
Designing for failure means honestly assessing what failures your system needs to survive, how much downtime is acceptable, how much data loss is acceptable, and then building exactly enough resilience to meet those requirements. Not less. But not reflexively more, either.
--
Conclusion
Let's go all the way back to the beginning.
One user. One server. One database.
That was the starting point of this series. A perfectly reasonable starting point for any new application. Simple to understand, simple to build, simple to deploy.
Then the users multiplied, and we started hitting limits.
One server couldn't handle the traffic, so we added more and put a load balancer in front of them.
The database was answering the same questions thousands of times, so we added a cache and stopped making it repeat itself.
The cache needed to stay accurate as data changed, so we built invalidation strategies to keep it honest.
One database couldn't handle all the reads, so we added read replicas and distributed the load.
One database couldn't hold all the data, so we added sharding and split the data across many machines.
Some user requests triggered too much background work, so we added message queues and moved that work off the critical path.
Users in distant regions were waiting too long for content, so we added CDN edge locations and brought the content closer to them.
The application itself became too large and too tightly coupled for many teams to develop independently, so we split it into services that could be deployed and scaled on their own.
And finally, with all those pieces in place, we asked the question that reframes everything: what happens when something fails?
The answer wasn't a new technology. It was a mindset.
Every technique in this series was a response to a specific bottleneck. Load balancers responded to traffic. Caching responded to repeated work. Sharding responded to data volume. Message queues responded to latency. CDNs responded to distance. Microservices responded to organizational coupling.
High availability responds to failure. And it does so by applying the same fundamental principle as every solution before it: don't let any single thing be the only thing standing between your users and a working application.
The complete journey:
One server --> Load Balancers
One query, repeated --> Caching + Invalidation
One database reads --> Read Replicas
One database too large --> Sharding
Users waiting --> Message Queues
Content too far --> CDNs
One codebase --> Microservices
Single point of failure --> Redundancy + Failover
Each column on the right isn't a technology to memorize. It's an answer to a specific question that the growing system forced someone to ask.
That's what system design actually is. Not a catalog of tools. Not a collection of patterns to be applied uniformly. It's the practice of asking what happens when this component can't handle what's being asked of it, and then finding the right response.
The questions change as the system grows. But the habit of asking them stays the same.
What happens when one server isn't enough?
What happens when one database can't keep up?
What happens when data outgrows one machine?
What happens when users are too far from the server?
What happens when one application becomes too hard to change?
And finally: what happens when something fails?
Every one of those questions led somewhere. Not to a perfect system, because perfect systems don't exist. But to a better one, more capable of surviving the pressures that scale inevitably brings.
The goal was never to build a system that never fails.
It was to build one that keeps working when it does.
That's where this series ends. And if you've been reading since Part 1, that's also where the real work begins.
Top comments (0)