Who calls your /health endpoint, what does it check, and what happens when it fails? Those three answers decide whether a slow dependency stays a slow dependency or becomes an outage.
The endpoint itself is simple. The systems consuming it are not, and they do different things with a failure.
Three consumers, three different reactions
A health check is not one mechanism. It is at least three, and mixing them up is where the trouble starts.
A Kubernetes liveness probe decides when to restart a container. The Kubernetes documentation says that if a container fails its liveness probe more times than the configured tolerance, the kubelet restarts that container. Restart is the remedy for a process that is stuck, for example in a deadlock.
A Kubernetes readiness probe decides whether a pod gets traffic. If the readiness probe fails, the EndpointSlice controller removes the pod's IP address from the EndpointSlices of all matching Services. The container is not restarted. It keeps running; it just stops receiving requests until the probe passes again.
An AWS Application Load Balancer health check decides whether a target stays in rotation. Per the ELB documentation, after UnhealthyThresholdCount consecutive failures the load balancer takes the target out of service. It does not restart or replace anything. And there is a caveat worth knowing before you rely on it: if every registered target in all enabled Availability Zones fails health checks at the same time, the load balancer fails open and routes requests to all of them, regardless of health status.
Restarts, removal from endpoints, removal from rotation, and fail-open are four different responses to the same failed HTTP call.
Where the cascade starts
Now make the endpoint deep: before returning 200, it runs a test query against the database and pings a third-party API. The intent is good. The blast radius is not.
The database gets slow for reasons unrelated to your app code. Every instance's deep check starts failing at the same time, because every instance shares that database. What happens next depends on which consumer is wired to that endpoint:
- Wired to a liveness probe: the kubelet restarts healthy application containers across the fleet because a dependency is slow. Restarts add load and cold starts to an already degraded system.
- Wired to a readiness probe or an ALB: every instance leaves rotation at once. With an ALB, once all targets report unhealthy in all enabled zones, the fail-open behavior sends traffic back to all of them, including instances that still cannot reach the database.
Either way, one slow shared dependency has been amplified into an app-wide event. The check did not detect the outage. It created the shape of it.
These reactions are also slower than they look. The same ELB documentation notes that an ALB checks each target about every HealthCheckIntervalSeconds, 30 seconds by default for instance and IP targets, and only removes it after consecutive failures. A failing instance can keep serving errors for minutes before rotation changes. Kubernetes probes have their own configurable failure tolerances.
The opposite failure
The shallow version has its own cost. An endpoint that returns 200 as long as the process runs will keep an instance in rotation while the app cannot serve real requests: deadlocked workers, a broken configuration load, a missing environment variable. Users get errors from instances every check calls healthy.
So deep is dangerous and shallow is blind. That tension is the actual design problem.
How I would wire it
My advice, and it is advice rather than a rule: keep the endpoint that load balancers and probes consume shallow. It should answer "is this process up and able to accept work," not "is every dependency healthy." Put dependency checks on a separate route that monitoring and humans read, where a failure pages someone instead of changing traffic.
If a dependency check must influence traffic, scope it to what the instance can actually do about it. An instance cannot fix a slow database by leaving rotation, and a fleet of instances leaving rotation together is worse than the original slowness.
Then test the behavior rather than assuming it: in a controlled environment, make a dependency slow and watch which probes fail, what the orchestrator and load balancer do, and whether the fail-open path behaves the way the documentation describes.
The short version
A health check is a control loop, not a formality. Kubernetes liveness restarts, readiness removes from endpoints, and an ALB removes from rotation with a fail-open caveat when everything fails together. Point them at an endpoint that checks shared dependencies, and one slow dependency can take down healthy code. Keep the traffic-path check shallow, move deep checks to monitoring, and verify the real behavior in a test you control.
Cover photo by Samuel Ramos, free to use under the Unsplash License.
I write about startups and AI on StartupWiki.

Top comments (0)