DEV Community

Cover image for Reliability & Fault Tolerance: Designing Systems That Survive Failures
Tanu Priya
Tanu Priya

Posted on

Reliability & Fault Tolerance: Designing Systems That Survive Failures

#architecture #softwareengineering #systemdesign

Modern applications are expected to be available almost all the time.

Users don't care if a server crashes, a database becomes unavailable, or a network connection fails—they simply expect the application to work. But in distributed systems, failures are not rare exceptions.

Failures are inevitable.

Servers crash. Networks fail. Databases become unavailable. Deployments go wrong. External APIs time out.

The goal of good system design is not to build a system where failures never happen.

The goal is to build a system that continues working even when failures happen.

This is where Reliability and Fault Tolerance become essential.

What Is Reliability?

Reliability is the ability of a system to perform its intended function correctly and consistently over time.

A reliable system should:

  • Return correct results
  • Handle expected traffic
  • Recover from temporary failures
  • Minimize downtime
  • Avoid data loss

For example, if an application promises 99.9% availability:

Total Time: 30 Days

Allowed Downtime ≈ 43 Minutes
Enter fullscreen mode Exit fullscreen mode

Higher availability means less acceptable downtime.

Availability Approx. Downtime per Month
99% 7 hours 18 minutes
99.9% 43 minutes
99.99% 4 minutes
99.999% 26 seconds

But achieving higher reliability also increases infrastructure complexity and cost.

What Is Fault Tolerance?

Fault tolerance is the ability of a system to continue operating even when one or more components fail.

Imagine a simple architecture:

Users
   │
   ▼
Application Server
   │
   ▼
Database
Enter fullscreen mode Exit fullscreen mode

What happens if the application server crashes?

Application Server 

→ Entire application becomes unavailable
Enter fullscreen mode Exit fullscreen mode

This architecture has a single point of failure.

A fault-tolerant system introduces redundancy:

                    Load Balancer
                         │
              ┌──────────┴──────────┐
              ▼                     ▼
         Server 1               Server 2
             │                     │
             └──────────┬──────────┘
                        ▼
                     Database
Enter fullscreen mode Exit fullscreen mode

If Server 1 fails:

Server 1 

Traffic → Server 2
Enter fullscreen mode Exit fullscreen mode

The system continues serving users.

That is fault tolerance.

Types of Failures in Distributed Systems

Before designing for reliability, we need to understand what can fail.

1. Server Failures

A server can crash because of:

  • Hardware issues
  • Memory exhaustion
  • Application bugs
  • CPU overload
  • Operating system failures

Example:

Server 1 
Server 2 
Server 3
Enter fullscreen mode Exit fullscreen mode

The system should detect the failure and redirect traffic.

2. Network Failures

Networks can experience:

  • Packet loss
  • High latency
  • Connection failures
  • DNS issues
  • Network partitions

A service might appear unavailable even though it is still running.

Service A ──── ❌ ──── Service B
        Network Failure
Enter fullscreen mode Exit fullscreen mode

This is why distributed systems must carefully handle timeouts and retries.

3. Database Failures

Databases can fail due to:

  • Hardware crashes
  • Storage issues
  • Connection limits
  • Overloaded queries
  • Software bugs

A single database creates a major risk:

Application
     │
     ▼
Database
Enter fullscreen mode Exit fullscreen mode

Even if every application server is running, the application may still be unavailable.

4. Dependency Failures

Modern applications often depend on external services.

For example:

Application
     │
     ├── Payment Service
     ├── Email Service
     ├── Authentication Service
     └── Analytics Service
Enter fullscreen mode Exit fullscreen mode

If one dependency fails, your application should not always fail completely.

For example, an analytics failure should not prevent a user from completing a payment.

Redundancy: Don't Depend on One Component

One of the most important principles of fault tolerance is:

Avoid single points of failure.

Redundancy means having multiple components capable of performing the same function.

For example:

                  Load Balancer
                       │
          ┌────────────┼────────────┐
          ▼            ▼            ▼
       Server 1      Server 2      Server 3
Enter fullscreen mode Exit fullscreen mode

If one server fails:

Server 1 
Server 2 
Server 3
Enter fullscreen mode Exit fullscreen mode

The remaining servers continue handling requests.

Redundancy can exist at multiple levels:

  • Application servers
  • Databases
  • Load balancers
  • Data centers
  • Network connections

Database Replication

Databases are critical, so they often use replication.

A common architecture looks like this:

                 Application
                      │
              ┌───────┴───────┐
              ▼               ▼
           Primary         Replica
          Database        Database
Enter fullscreen mode Exit fullscreen mode

The primary database handles writes:

Write Request
     │
     ▼
Primary Database
Enter fullscreen mode Exit fullscreen mode

The data is then replicated to other databases:

Primary
   │
   ├────→ Replica 1
   │
   └────→ Replica 2
Enter fullscreen mode Exit fullscreen mode

If the primary database fails, one replica may become the new primary.

This process is called failover.

Failover: Switching When Something Fails

Failover is the process of automatically switching traffic or operations from a failed component to a healthy backup.

Example:

        Before Failure

Application
     │
     ▼
Primary Server ✅
Enter fullscreen mode Exit fullscreen mode

After failure:

Primary Server ❌
        │
        ▼
Backup Server Becomes Active ✅
Enter fullscreen mode Exit fullscreen mode

The same concept can be used for:

  • Application servers
  • Databases
  • Network infrastructure
  • Data centers

A good failover mechanism should be:

  • Fast
  • Automatic
  • Reliable
  • Regularly tested

Because a backup system that has never been tested may fail exactly when you need it most.

Retries: Handling Temporary Failures

Not every failure is permanent.

Sometimes a request fails because of:

  • Temporary network issues
  • Short service overload
  • Brief database unavailability

In these cases, retrying the request can help.

Example:

Request ❌
   │
Retry 1 ❌
   │
Retry 2 ❌
   │
Retry 3 ✅
Enter fullscreen mode Exit fullscreen mode

However, retries must be implemented carefully.

A naive retry strategy can make an outage worse.

Imagine 1 million failed requests immediately retrying:

1M Requests
    │
    ▼
Service Overloaded ❌
    │
    ▼
1M Immediate Retries
    │
    ▼
Even More Overload 💥
Enter fullscreen mode Exit fullscreen mode

This is known as a retry storm.

Exponential Backoff

Instead of retrying immediately, systems often use exponential backoff.

Example:

Retry 1 → Wait 1 second
Retry 2 → Wait 2 seconds
Retry 3 → Wait 4 seconds
Retry 4 → Wait 8 seconds
Enter fullscreen mode Exit fullscreen mode

This gives the failing service time to recover.

A more realistic approach also adds jitter, which introduces a small random delay.

Retry Delay = Exponential Backoff + Random Jitter
Enter fullscreen mode Exit fullscreen mode

This prevents millions of clients from retrying at exactly the same time.

Timeouts: Don't Wait Forever

Every external call should have a reasonable timeout.

Bad design:

Application → Waiting forever...
Enter fullscreen mode Exit fullscreen mode

Better design:

Application
     │
     ▼
Call External Service
     │
     ├── Response within 3 seconds → Continue
     │
     └── No response → Timeout ❌
Enter fullscreen mode Exit fullscreen mode

Without timeouts, requests can remain stuck and consume valuable resources.

Timeouts help the system fail fast and recover gracefully.

Circuit Breakers

A circuit breaker prevents an application from repeatedly calling a service that is already failing.

Imagine this:

Application → Payment Service ❌
Application → Payment Service ❌
Application → Payment Service ❌
Application → Payment Service ❌
Enter fullscreen mode Exit fullscreen mode

Instead, after repeated failures:

Circuit Breaker → OPEN 🔴
Enter fullscreen mode Exit fullscreen mode

The application temporarily stops sending requests to the failing service.

Application
     │
     ▼
Circuit Breaker OPEN
     │
     └── Return fallback response
Enter fullscreen mode Exit fullscreen mode

After some time, the system can test whether the dependency has recovered.

Typical circuit breaker states:

CLOSED → OPEN → HALF-OPEN → CLOSED
Enter fullscreen mode Exit fullscreen mode

This prevents a failing dependency from causing a cascading failure across the entire system.

Graceful Degradation

A reliable system doesn't always need to provide every feature during a failure.

Instead, it can continue providing the most important functionality.

For example:

Normal Mode
├── Recommendations
├── Search
├── Payments
├── Notifications
└── Analytics
Enter fullscreen mode Exit fullscreen mode

If the recommendation service fails:

Degraded Mode
├── Recommendations ❌ Temporarily Disabled
├── Search ✅
├── Payments ✅
├── Notifications ✅
└── Analytics ✅
Enter fullscreen mode Exit fullscreen mode

The system is not perfect, but users can still perform critical actions.

This is called graceful degradation.

Health Checks

How does a load balancer know whether a server is healthy?

Using health checks.

For example:

GET /health
Enter fullscreen mode Exit fullscreen mode

A healthy server might return:

{
  "status": "healthy"
}
Enter fullscreen mode Exit fullscreen mode

The load balancer periodically checks each server:

Load Balancer
     │
     ├── Server 1 → Healthy ✅
     ├── Server 2 → Healthy ✅
     └── Server 3 → Unhealthy ❌
Enter fullscreen mode Exit fullscreen mode

Traffic is automatically removed from unhealthy servers.

Health checks are especially important in distributed environments with multiple instances.

Monitoring: You Can't Fix What You Can't See

Reliability also depends on detecting problems quickly.

Important metrics include:

  • Error rate
  • Response time
  • CPU usage
  • Memory usage
  • Database latency
  • Request throughput
  • Failed requests

A monitoring system might detect:

Normal Error Rate: 0.2%

Current Error Rate: 15% 🚨
Enter fullscreen mode Exit fullscreen mode

An alert can then notify engineers before the issue becomes a major outage.

Monitoring turns failures from:

"Users are complaining that the app is down."

into:

"We detected increasing errors and are investigating the problem."

Avoiding Cascading Failures

A cascading failure happens when one failure causes failures in other parts of the system.

Example:

Database Slows Down
       │
       ▼
Application Requests Wait
       │
       ▼
Threads Become Exhausted
       │
       ▼
Application Becomes Unresponsive
       │
       ▼
Entire System Fails 💥
Enter fullscreen mode Exit fullscreen mode

To prevent this, systems use techniques such as:

  • Timeouts
  • Circuit breakers
  • Rate limiting
  • Queues
  • Bulkheads
  • Load shedding

The goal is to isolate failures instead of allowing them to spread.

The Bulkhead Pattern

A ship uses bulkheads to isolate flooded sections so the entire ship doesn't sink.

The same idea can be used in software.

Instead of sharing all resources:

One Shared Resource Pool
Enter fullscreen mode Exit fullscreen mode

You isolate resources:

Payment Resources
Recommendation Resources
Search Resources
Notification Resources
Enter fullscreen mode Exit fullscreen mode

If the recommendation service consumes all its resources, it doesn't necessarily affect payments or search.

This improves fault isolation.

A Reliable System Architecture

A highly reliable architecture might look like this:

                     Users
                       │
                       ▼
                  Load Balancer
                       │
          ┌────────────┼────────────┐
          ▼            ▼            ▼
       App 1         App 2         App 3
          │            │            │
          └────────────┼────────────┘
                       ▼
                Primary Database
                       │
              ┌────────┴────────┐
              ▼                 ▼
          Replica 1         Replica 2
Enter fullscreen mode Exit fullscreen mode

Additional systems may include:

Cache
Queue
Monitoring
Alerting
Backup Storage
Disaster Recovery
Enter fullscreen mode Exit fullscreen mode

Each component contributes to the overall reliability of the system.

Reliability vs Fault Tolerance

These concepts are related but not exactly the same.

Reliability Fault Tolerance
Focuses on consistent operation Focuses on surviving failures
Reduces the chance of failure Reduces the impact of failure
Includes monitoring and testing Includes redundancy and failover
Measures how dependable a system is Measures how well a system handles faults

A highly reliable system should ideally also be fault tolerant.

When Designing for Reliability, Ask These Questions

Whenever you design a system, ask:

What happens if this server crashes?

What happens if the database becomes unavailable?

What happens if a request times out?

What happens if an external API fails?

Can the system retry safely?

Is there a backup?

How does the system detect failures?

Can traffic automatically move to a healthy server?

Can one failing service bring down the entire system?

These questions often reveal weaknesses in a system before they become real outages.

Final Thoughts

Failures are a normal part of distributed systems.

You cannot completely prevent servers, networks, databases, or external services from failing. But you can design systems that expect failure and recover from it gracefully.

The core building blocks of reliable systems include:

  • Retries for temporary failures
  • Timeouts to prevent requests from hanging
  • Redundancy to remove single points of failure
  • Failover to switch to healthy components
  • Circuit breakers to prevent cascading failures
  • Monitoring to detect problems early
  • Health checks to identify unhealthy services
  • Graceful degradation to keep critical features running

The biggest mindset shift in system design is simple:

Don't ask, "Will this component fail?" Ask, "When it fails, what happens next?"

That is the foundation of Reliability and Fault Tolerance.

Key Takeaways

  • Failures are inevitable in distributed systems.
  • Reliability focuses on consistent, dependable operation.
  • Fault tolerance helps systems continue working during failures.
  • Redundancy removes single points of failure.
  • Retries should use exponential backoff and jitter.
  • Timeouts prevent requests from waiting indefinitely.
  • Failover redirects traffic to healthy components.
  • Circuit breakers prevent failing dependencies from causing larger outages.
  • Monitoring and health checks help detect failures quickly.
  • Design systems assuming that something will eventually fail.

*A scalable system handles growth. A reliable system survives failure. The best systems do both. *

Top comments (0)