DEV Community

Cover image for Reliability & Availability
NISCHIT D S
NISCHIT D S

Posted on

Reliability & Availability

#systemdesign #beginners #learning

| What's the Difference?

These terms are often confused, but they measure different things:

  1. Reliability : System performs correctly without failure
  2. Availability : System is operational when needed

Analogy: Think of a car:

  • Reliable = The car doesn't break down during your trip
  • Available = The car is ready to drive when you need it 
A system can be available but unreliable (works but gives wrong results), or reliable but unavailable (works correctly but only 50% of the time).
Enter fullscreen mode Exit fullscreen mode

| Measuring Availability

Availability is typically expressed as a percentage of uptime:

Availability = Uptime / (Uptime + Downtime) × 100%

**Real-world examples:**

Google Search: ~99.99% (four nines)
AWS S3: 99.99% availability SLA
Credit card processing: Often requires 99.999%
Enter fullscreen mode Exit fullscreen mode

| Calculating Composite Availability

For components in sequence (all must work):

A_total = A₁ × A₂ × A₃ × ... × Aₙ
Enter fullscreen mode Exit fullscreen mode

Example: Three services at 99% each in sequence:

0.99 × 0.99 × 0.99 = 97.03%
Enter fullscreen mode Exit fullscreen mode

For components in parallel (any can work):

A_total = 1 - (1 - A₁) × (1 - A₂) × ... × (1 - Aₙ)
Enter fullscreen mode Exit fullscreen mode

Example: Two servers at 99% each in parallel:

1 - (0.01 × 0.01) = 99.99%
Enter fullscreen mode Exit fullscreen mode

| Measuring Reliability

Reliability is measured using these key metrics:

Mean Time Between Failures (MTBF)

Average time the system operates before failing.

MTBF = Total Operating Time / Number of Failures
Enter fullscreen mode Exit fullscreen mode

Example: Server runs for 8760 hours/year with 2 failures

MTBF = 8760 / 2 = 4380 hours
Enter fullscreen mode Exit fullscreen mode

Mean Time To Recovery (MTTR)

Average time to restore the system after failure.

MTTR = Total Downtime / Number of Failures
Enter fullscreen mode Exit fullscreen mode

Lower MTTR = Better availability

Relationship to Availability

Availability = MTBF / (MTBF + MTTR)
Enter fullscreen mode Exit fullscreen mode

Key Insight: You can improve availability by:

  • Increasing MTBF (fail less often)
  • Decreasing MTTR (recover faster)

Often, reducing MTTR is easier and cheaper than increasing MTBF!

| Common Failure Modes

Software Failures

  • Memory leaks - Application crashes over time
  • Bugs - Incorrect behavior under certain conditions
  • Deadlocks - System hangs waiting for resources
  • Resource exhaustion - Disk full, too many connections

Human Errors

Studies show 70-80% of outages are caused by humans:

  • Configuration mistakes
  • Deployment errors
  • Accidental deletions
  • Security misconfigurations

External Failures

  • Network partitions
  • Third-party service outages
  • Power failures
  • Natural disasters

| Patterns for High Availability

1. Redundancy : Eliminate single points of failure by duplicating components.

Types of redundancy:

  • Active-Active: All nodes handle traffic (load sharing)
  • Active-Passive: Standby nodes take over on failure
  • N+1: One extra server for every N servers

2. Health Checks & Monitoring : Detect failures quickly to minimize MTTR.

Health check best practices:

  • Check every 10-30 seconds
  • Use separate endpoint from main API
  • Include dependency checks
  • Set appropriate timeouts

3. Failover Strategies

4. Circuit Breaker Pattern : Prevent cascading failures by stopping calls to failing services.

States:

  • Closed: Normal operation, requests pass through
  • Open: Requests fail immediately (no call to service)
  • Half-Open: Allow limited requests to test recovery
  1. Retry with Exponential Backoff : Handle transient failures gracefully. 
def retry_with_backoff(operation, max_retries=5):
    for attempt in range(max_retries):
        try:
            return operation()
        except TransientError:
            wait_time = (2 ** attempt) + random.uniform(0, 1)
            time.sleep(wait_time)
    raise Exception("Max retries exceeded")
Enter fullscreen mode Exit fullscreen mode 
Key points:

- Start with small delay (1s)
- Double each time (1s, 2s, 4s, 8s...)
- Add jitter to prevent thundering herd
- Set maximum retry count
Enter fullscreen mode Exit fullscreen mode

Real-World Architecture: High Availability Database

This architecture provides:

  • Read scaling via replicas
  • Automatic failover if primary fails
  • Point-in-time recovery from S3 backups
  • Geographic distribution possible

Key Takeaways

  • Availability measures uptime, Reliability measures correctness
  • Each "nine" of availability is 10x harder to achieve
  • Parallel redundancy dramatically improves availability
  • Reducing MTTR is often easier than increasing MTBF
  • Human error causes most outages—automate and use guardrails
  • Circuit breakers prevent cascading failures
  • SLOs should be based on user needs, not engineering pride

Top comments (0)