A couple of days ago I wrote that a health check should check something real: open a connection to the database, touch a real table, answer 503 if it can't. I still think a health check that only proves the process is running tells you almost nothing.
But there's a second half to that advice, and skipping it is how a health check takes a whole system down.
Ten servers, one database, thirty seconds
Picture ten app servers behind a load balancer. Each one exposes /health, and /health runs a query against the shared database. The load balancer checks every few seconds and pulls any server that fails a few checks in a row.
Now the database has a thirty-second blip. A failover, a connection limit, a slow vacuum. It doesn't matter which.
Every server's health check fails at the same moment, because every server depends on the same database. The load balancer does exactly what it was told: it marks all ten unhealthy and stops sending them traffic.
A few things go wrong at once:
- Requests that never needed the database (static pages, cached reads, a status page) now fail too, because there's no server left in rotation.
- When the database comes back, the servers don't. The load balancer waits for several consecutive passing checks on each one before re-adding it, so the outage outlasts the blip.
- If an auto scaling group replaces instances that fail load balancer checks, it starts killing healthy servers and launching new ones that will fail the same check.
Nothing about any single server was broken. The health check couldn't tell "this server is sick" from "something every server depends on is sick", and it reacted to both the same way.
The failure is correlated
AWS's Builders' Library article on health checks splits them into three kinds: liveness checks (does the process answer?), local health checks (can this server do its own work: disk, threads, config?), and dependency health checks (can it reach the database, cache and other services?). Local failures tend to hit one server at a time. Dependency failures don't. When a shared dependency goes down, it goes down for everyone.
Their advice is to keep the fast-acting load balancer checks local, and to handle the deeper dependency checks somewhere that can react carefully instead of ejecting the whole fleet.
What I'd do instead
Split the endpoint. One route for the load balancer, one for you.
// Routing decision: can THIS server do its own work?
app.get('/ready', async (_req, reply) => {
const ok = !shuttingDown && eventLoopLagMs() < 500;
return reply.code(ok ? 200 : 503).send({ ok });
});
// Visibility: how are the things it depends on?
app.get('/health/deep', async () => ({
db: await probe(() => db.execute(sql`select 1 from tenants limit 1`), 1000),
queue: await probe(() => redis.ping(), 500),
}));
// Every probe has a timeout, so a slow dependency can't hang the check.
async function probe(fn: () => Promise<unknown>, ms: number) {
try {
await Promise.race([fn(), new Promise((_, reject) =>
setTimeout(() => reject(new Error('timeout')), ms))]);
return 'ok';
} catch {
return 'failing';
}
}
The load balancer watches /ready. Dashboards and alerts watch /health/deep. A database blip now pages a person and shows red on a graph, but it doesn't empty the load balancer.
Fail open if deep checks must route traffic. Some load balancers already do this: if every target fails, they route to all of them anyway, on the theory that a fleet-wide failure is more likely a bad check than a fleet of dead servers. Know whether yours does before you rely on it.
Alert on the dependency, not the server. If the database is down everywhere, replacing servers can't fix it. Ejecting them only adds a second outage on top of the first.
Keep the check cheap. A deep check that runs a query every few seconds on every server adds load to the database exactly when it's struggling. Give each probe a short timeout, and cache the result for a few seconds if your fleet is large.
So, should a health check check something real?
Yes, but check the thing this server owns. Route on what one server can fix by being removed. Report on everything else, and let a person, or something that can see the whole fleet, decide what a shared failure means.
A health check is a promise that this server can do its job. Don't make it promise things about the database it can't keep.
Source: Amazon Builders' Library, Implementing health checks.
I write about the backend checks that catch problems before users do. Follow along if that's your kind of thing.
Top comments (0)