The 3 AM Wake-Up Call
Two years ago I got paged at 3 AM. The monitoring dashboard was screaming that our payment service was down. I logged into the server, checked the process list — the service was running. Checked the port — it was listening. Checked the logs — requests were coming in and responses were going out.
But payments were failing. Why? Because the database connection pool had silently exhausted, and every request was timing out after 30 seconds.
Here's the infuriating part: we had a /health endpoint. It returned 200 OK with {"status": "alive"}. The load balancer was happy. The process supervisor was happy. Customers were not.
That night I learned the difference between a liveness check and a health check — and it cost us 40 hours of debugging over the following month to untangle the cascading failures.
Liveness ≠ Health
Most teams conflate these two things:
| Liveness Check | Health Check | |
|---|---|---|
| Answers | "Is the process running?" | "Can the process do its job?" |
| Checks | Port is open, HTTP responds | DB connection, upstream APIs, disk space, queue depth |
| Used by | Orchestrator (K8s, systemd) | Load balancer, monitoring, on-call dashboards |
| When it fails | Restart the container | Stop routing traffic to it |
The /health endpoint that returns {"status": "ok"} regardless of whether the database is reachable? That's a liveness check masquerading as a health check. It will lie to your load balancer and make bad situations worse.
The Pattern: 3-Tier Health Checks
After that 3 AM incident I standardized on three tiers:
# health.py — drop this into any service
import time
import json
from typing import Dict, Any
class HealthChecker:
"""3-tier health check for any service."""
def __init__(self):
self._checks = {}
self.start_time = time.time()
def register(self, name: str, check_fn, tier: str = "critical"):
"""Register a health check function.
Tiers:
- critical: service cannot function without this (DB, config)
- degraded: service is impaired (cache miss, secondary API)
- informational: nice to know (queue depth, disk usage)
"""
self._checks[name] = {"fn": check_fn, "tier": tier}
def run(self) -> Dict[str, Any]:
results = {}
overall = "healthy"
for name, check in self._checks.items():
try:
ok, detail = check["fn"]()
results[name] = {
"status": "pass" if ok else "fail",
"detail": detail,
"tier": check["tier"]
}
if not ok and check["tier"] == "critical":
overall = "unhealthy"
elif not ok and check["tier"] == "degraded" and overall == "healthy":
overall = "degraded"
except Exception as e:
results[name] = {
"status": "fail",
"detail": str(e),
"tier": check["tier"]
}
if check["tier"] == "critical":
overall = "unhealthy"
return {
"status": overall,
"uptime_seconds": int(time.time() - self.start_time),
"timestamp": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()),
"checks": results
}
# ---- Usage in your FastAPI app ----
from fastapi import FastAPI, Response
app = FastAPI()
health = HealthChecker()
# Register checks — these are the actual functions you'd implement
health.register("database", lambda: check_db_connection(), tier="critical")
health.register("redis_cache", lambda: check_redis(), tier="degraded")
health.register("disk_space", lambda: check_disk("/var/data"), tier="informational")
health.register("upstream_payment_api",
lambda: check_upstream("https://api.payments.example/health"),
tier="critical")
@app.get("/health")
def health_endpoint():
result = health.run()
status_code = 200 if result["status"] == "healthy" else 503
return Response(
content=json.dumps(result),
status_code=status_code,
media_type="application/json"
)
@app.get("/health/live")
def liveness():
"""Simple liveness — used by K8s livenessProbe."""
return {"status": "alive"}
@app.get("/health/ready")
def readiness():
"""Readiness — used by K8s readinessProbe and load balancer."""
result = health.run()
if result["status"] == "unhealthy":
return Response(content=json.dumps(result), status_code=503)
return result
The key insight: your load balancer should use /health/ready, not /health/live. When the database is down, the process is alive, but it cannot serve traffic. Route requests elsewhere.
What to Check (and What NOT to Check)
After implementing this pattern across 12 services, here's what I learned about which checks are worth it:
Worth Checking
| Check | Tier | Why |
|---|---|---|
| Database connection (ping + simple query) | critical | Most common silent failure |
| Upstream API availability | critical/degraded | Catch cascading failures early |
| Disk space (more than 10 percent free) | degraded | Prevents the "disk full at 2 AM" surprise |
| Message queue depth | informational | Early warning for backpressure |
| Configuration validity | critical | Catch bad deploys before traffic hits them |
| Redis/memcached | degraded | Without cache, service degrades but still works |
NOT Worth Checking
| Check | Why Not |
|---|---|
| Deep application logic | Health check is not an integration test suite |
| Every single downstream service | Pick what's actually critical for THIS service |
| Synthetic transactions on every call | Use a separate probe for that (run every 60s, not on every health check request) |
| Anything that takes over 2 seconds | Health checks are called frequently; slow checks cause cascading timeouts |
The Kubernetes Wiring
If you're on Kubernetes, here's how to wire this up:
apiVersion: apps/v1
kind: Deployment
spec:
template:
spec:
containers:
- name: my-service
# Liveness: is the process alive? Restart if not.
livenessProbe:
httpGet:
path: /health/live
port: 8000
initialDelaySeconds: 10
periodSeconds: 30
timeoutSeconds: 3
failureThreshold: 3
# Readiness: can this pod serve traffic? Stop routing if not.
readinessProbe:
httpGet:
path: /health/ready
port: 8000
initialDelaySeconds: 5
periodSeconds: 10
timeoutSeconds: 5
failureThreshold: 2
The liveness probe checks /health/live — it should never fail unless the process is truly stuck. The readiness probe checks /health/ready — it fails when the database is down, and traffic stops being routed to that pod. The pod stays alive, waiting for the database to come back.
This separation is the single most impactful thing you can do for service reliability.
The 5-Minute Implementation Checklist
-
Create a
health.pymodule — copy theHealthCheckerclass above -
Add
/health/liveand/health/readyendpoints to your service - Register critical dependencies as tier="critical" checks
-
Wire up your load balancer to use
/health/ready, not/health - Test it: kill your database and verify your service returns 503 and gets drained from the load balancer
- Add a dashboard widget showing per-service health status — it'll pay for itself the next time something breaks
One Final Rule
If your health check returns 200 OK but can't actually tell you whether the service is working, you don't have a health check. You have a lie.
A health check that lies is worse than no health check at all — because it gives you false confidence while quietly routing traffic to broken instances.
What does your team's health check actually verify? I'd love to hear what checks have saved you from late-night pages.
Top comments (1)
I completely agree with the distinction between liveness checks and health checks, and I've seen similar issues in my own experience where a service appears to be running but is actually unable to perform its job. The 3-tier health check pattern you've implemented is a great way to provide a more nuanced view of a service's health, and I appreciate the example code you've provided. One thing I'm curious about is how you handle cases where a critical check is failing, but the service is still able to recover on its own after a short period of time - do you have any mechanisms in place to automatically retry failed checks or to alert the team to potential issues?