A health check is an endpoint or command that reports whether a service is working. Orchestrators, load balancers, and developers use it to decide whether to send traffic, restart a process, or wait.
The most common mistake is a check that always returns success. It says the process is alive, and nothing about whether the service can do its job.
Liveness and readiness
Two questions matter, and they have different answers:
- Liveness: is the process running and responsive? If not, restart it.
- Readiness: can the service handle requests right now? If not, stop sending it traffic, but do not restart it.
Mixing them causes trouble. A service that waits for its database should not be restarted just because the database is slow. It should be marked not ready until the database answers.
Designing a readiness check
A readiness check should test what the service needs to answer a request. That might include a database connection, a required configuration value, or a dependency it cannot work without. Keep it cheap. A check that runs a heavy query every few seconds adds load to the thing it is checking.
A liveness check should be simpler. It should prove the event loop or main thread responds, and nothing more.
Timeouts
A check with no timeout can hang, and a hung check looks like a dead service. Give every check a short limit, and decide what happens when it expires.
Startup order
Readiness solves a common startup problem. A service that starts before its database should report not ready, and the orchestrator should wait for it. Declaring dependencies explicitly and gating startup on readiness avoids a race that otherwise shows up as intermittent failures.
TDK CLI starts services in dependency order and reports readiness. tdk status --json returns data.tilt.readiness, with ready, pending, and failures, so a script or an agent can wait for a clear result rather than guessing. tdk up --json reports ok: false with errors when a resource fails or readiness times out.
Checks in production
In production, health checks feed load balancers and restart policies. Keep the readiness check honest, because an optimistic check sends traffic to a service that cannot answer it. Keep the liveness check narrow, because an aggressive one restarts healthy services under load.
Summary
Separate liveness from readiness. Design readiness around what the service needs to answer requests, keep both checks cheap and time-limited, and use readiness to order startup. A health check is only useful when it fails for the right reason.
Top comments (0)