DEV Community

Cover image for 🔁 One Health Check Turned a 10-Minute Outage Into 40
Kyryl
Kyryl

Posted on

🔁 One Health Check Turned a 10-Minute Outage Into 40

#kubernetes #springboot #observability #java

The database went down for ten minutes. The restart storm it triggered lasted forty.

Every pod's /actuator/health turned red the moment the database became unreachable. Kubernetes read that as "the process is broken" and started killing pods, one after another, across the entire fleet. New pods booted, connected to nothing, failed the same check, and got killed again. The dependency was down for ten minutes. The self-inflicted damage ran four times longer.

The trigger was one line of config that looked like a simplification.

The setup that looks clean

Someone wired both the liveness and the readiness probe to the same endpoint.

livenessProbe:
  httpGet:
    path: /actuator/health
    port: 8080
readinessProbe:
  httpGet:
    path: /actuator/health
    port: 8080
Enter fullscreen mode Exit fullscreen mode

One endpoint, one thing to configure, one thing to reason about. It reads as clean. It is actually a category error, because those two probes are asking completely different questions and Spring's default /actuator/health answers neither of them cleanly.

Two probes, two questions

The distinction is the whole point, so it is worth stating flatly.

Liveness answers: should I kill this process and start a new one? The only honest reason to say yes is that the process is wedged in a way a restart will fix. A deadlock, an unrecoverable internal state, a JVM that is thrashing. Restarting helps only when the problem lives inside the process.

Readiness answers: should I send this pod traffic right now? A pod can be perfectly alive and still be unable to serve, because something it depends on is unavailable. The fix there is not a restart. It is to stop routing requests until the dependency comes back.

/actuator/health by default aggregates everything, including the db health indicator, into one status. Point liveness at it and you have told Kubernetes to kill the process whenever the database is down. But a down database is not a process problem, and no number of restarts will bring it back.

The restart storm, step by step

Here is the loop that ate thirty extra minutes:

  1. The database becomes unreachable.
  2. The db health indicator goes DOWN, so aggregated /actuator/health returns DOWN.
  3. The liveness probe fails. Kubernetes kills the pod.
  4. A fresh pod boots, starts up, and immediately checks its health.
  5. The database is still down, so the new pod's health is DOWN too.
  6. Liveness fails again. Kubernetes kills it again.
  7. Repeat, on every pod, for as long as the dependency stays down.

Now the failure is worse than the original outage. The pods are gone, so even reads that could have been served from cache are gone. Startup load hammers the recovering database the moment it comes back. And your dashboards are a wall of CrashLoopBackOff that makes it look like the application itself is broken, which sends everyone debugging the wrong thing.

What should have happened is boring by comparison. Readiness fails, Kubernetes pulls the pods out of the Service endpoints, traffic stops, the processes keep running. When the database returns, readiness goes green and traffic resumes. No restarts, no cold caches, no thundering herd.

The fix Actuator already ships

You do not need a custom endpoint or a sidecar. Spring Boot Actuator has health groups built for exactly this split.

management:
  endpoint:
    health:
      probes:
        enabled: true
      group:
        liveness:
          include: livenessState
        readiness:
          include: readinessState,db
Enter fullscreen mode Exit fullscreen mode

That gives you two dedicated endpoints:

  • /actuator/health/liveness includes only livenessState, the process-internal signal. No database, no downstream calls. It answers "is this process wedged" and nothing else.
  • /actuator/health/readiness includes readinessState plus the real dependency checks, db here. It answers "can this pod serve traffic right now."

Point the probes at the right endpoints:

livenessProbe:
  httpGet:
    path: /actuator/health/liveness
    port: 8080
readinessProbe:
  httpGet:
    path: /actuator/health/readiness
    port: 8080
Enter fullscreen mode Exit fullscreen mode

Now a database outage trips readiness only. Traffic drains, the pods live, and recovery is automatic. Liveness stays green because the process is, in fact, fine.

The honest trade-off

This is not a set-and-forget win, and pretending it is misses the way the bug comes back.

The config takes five minutes and then nobody looks at it again. That is the problem. Over the next year the service grows: a Redis cache, a Kafka producer, a call to some downstream API. Each of those registers a health indicator, and each is a decision about which group it belongs in. If a new dependency lands in the readiness group, great. If it silently ends up aggregated into liveness, or if someone adds a custom indicator without thinking about probes at all, you have quietly rebuilt the original bug with a new trigger.

The drift is invisible until the next outage. Nobody audits "which health indicators feed which probe" as part of adding a dependency, so the wrong wiring sits there dormant for months. The next time that specific dependency fails, the restart storm returns, and it looks brand new even though it is the same mistake.

The mitigation is process, not code: when you add a dependency with a health indicator, decide its group in the same PR. Treat "which probe does this affect" as part of the definition of done for any new external call.

Has your liveness probe ever quietly grown a dependency check nobody noticed, or has your fleet actually lived through a restart storm like this? Curious how other teams keep the liveness group honest as services accumulate dependencies.

Top comments (0)