Originally published on kuryzhev.cloud
A deployment goes out on a Friday afternoon. Within minutes, half the fleet is cycling through CrashLoopBackOff, even though the application code hasn't changed and the database is healthy. The likely culprit is not the code — it's a misconfigured liveness probe checking something it has no business checking. Understanding how kubernetes liveness readiness probes actually work, and where teams get the contract wrong, is the difference between a self-healing cluster and a self-inflicted outage.
What this actually does
Probes are not generic "is my app up" checks. They are a signaling contract between kubelet and the container, and each of the three probe types triggers a distinct, non-interchangeable action.
A liveness probe failure tells kubelet the process is stuck — deadlocked, hung, or otherwise unrecoverable — and the response is to kill and restart the container. A readiness probe failure tells kubelet the pod cannot currently serve traffic, so it is pulled from the Service's Endpoints (or EndpointSlices) — no restart, no disruption to the running process, just traffic rerouted elsewhere. A startup probe gates both of the others: until it succeeds, liveness and readiness checks are suspended entirely, which matters for anything with a slow boot sequence.
These checks run per-container, executed locally by the kubelet on the node the pod is scheduled to — not by the control plane, and not by the API server. Supported mechanisms are exec, httpGet, grpc, and tcpSocket. Two separate knobs control timing, and conflating them is a common source of bad tuning: timeoutSeconds has to exceed the worst-case response time of a single check, while periodSeconds × failureThreshold sets the total detection window before kubelet acts on a failing sequence. With periodSeconds: 10 and failureThreshold: 3, that detection window lands somewhere between roughly 20 and 30 seconds depending on where in the cycle the first failure occurs — treat it as an upper bound, not a fixed countdown. See the official Kubernetes documentation on configuring probes for exact field semantics.
How people use it wrong
The most common failure pattern: pointing liveness and readiness at the same endpoint, or the same underlying logic. If that shared /health handler also checks database connectivity, a slow query or a transient network blip doesn't just mark the pod unready — it kills the container outright. Multiply that across every replica hitting the same database, and a single slow dependency becomes a synchronized restart storm across the entire fleet.
A closely related mistake is designing liveness probes that check downstream dependencies at all — a third-party API, a cache cluster, a message broker. Liveness should answer "is this process itself broken," not "is everything this process depends on currently reachable." Conflating the two turns every upstream outage into a self-inflicted local one.
Watch out for: teams also frequently under-tune initialDelaySeconds for JVM-based or Node.js applications with real cold-start costs — JIT warmup, class loading, dependency injection wiring. If the liveness probe starts checking before the app is genuinely ready, kubelet kills a perfectly healthy-but-still-booting container, and the pod enters CrashLoopBackOff purely from impatience, not a real defect.
Another gotcha: assuming a stalled rollout means the new pods are crashing. A broken readiness probe is a documented failure mode with no restart signal at all — pods stay in Running state and never crash, but they also never flip to Ready, so the rollout appears to hang with no obvious error until someone runs kubectl describe pod.
The correct approach
The dividing line is recoverability. If a failure can resolve itself through retry or backoff without restarting the process, it belongs in readiness. If the process itself is unresponsive and only a restart fixes it, that's liveness territory.
Liveness should be cheap, local, and dependency-free — an in-process handler that returns 200 as long as the main loop is alive, nothing more. Readiness is explicitly allowed, and expected, to check real dependencies: connection pool health, cache reachability, queue lag. Its job is to gate traffic, so it should fail gracefully rather than crash the process when a dependency is temporarily unavailable.
For slow-booting applications, use startupProbe instead of inflating initialDelaySeconds on liveness. It has been GA since Kubernetes 1.20 and exists specifically to prevent premature kills during legitimate long initialization — schema migrations, cache preload, model loading.
The example below shows the three probes working together on one pod:
apiVersion: v1
kind: Pod
metadata:
name: probe-demo
spec:
containers:
- name: app
image: example/app:2026.1
ports:
- containerPort: 8080
# startupProbe: gates liveness/readiness until app finishes slow init
startupProbe:
httpGet:
path: /startupz
port: 8080
failureThreshold: 30 # 30 * 10s = 5 min max startup window
periodSeconds: 10
# liveness: cheap, local, no external dependencies
livenessProbe:
httpGet:
path: /healthz # returns 200 only if process loop is alive
port: 8080
periodSeconds: 10
timeoutSeconds: 2
failureThreshold: 3 # restart after roughly 20-30s of failures
# readiness: allowed to check real dependencies
readinessProbe:
httpGet:
path: /readyz # checks DB pool + cache connectivity
port: 8080
periodSeconds: 5
timeoutSeconds: 2
failureThreshold: 2 # pulled from endpoints after ~10s
successThreshold: 1
lifecycle:
preStop:
exec:
command: ["sh", "-c", "sleep 5"] # allow in-flight requests to drain
terminationGracePeriodSeconds: 30
The preStop hook above assumes the image ships a shell and a sleep binary — it silently fails on distroless or scratch images with no shell at all. Kubernetes 1.29+ exposes a native sleep lifecycle action (lifecycle.preStop.sleep.seconds) that needs neither; check the feature's stability status for the cluster's actual version before relying on it.
successThreshold is fixed at 1 for both livenessProbe and startupProbe — only readinessProbe accepts a value greater than 1, which is useful for requiring several consecutive successes from a flapping dependency before traffic resumes. Tune periodSeconds and failureThreshold against actual traffic tolerance: a latency-sensitive API needs a short detection window, while a batch or stateful workload can afford more patience before Kubernetes acts.
Advanced patterns
Services that implement the standard gRPC health-checking protocol can use the grpc probe type directly instead of running a sidecar or shelling out to grpc_health_probe. The feature was beta and enabled by default in Kubernetes 1.24, and reached general availability in 1.27 — confirm the cluster's actual version before depending on it in a strict compliance path.
A useful graceful-degradation pattern: during a deploy-triggered cache warmup or configuration reload, flip readiness to "not ready" without touching liveness at all. The pod stays alive, stops receiving new traffic, finishes its warmup, then flips back — no restart, no cold-start penalty, no lost in-flight connections.
Combine that with a preStop hook and a deliberate terminationGracePeriodSeconds during rolling updates, but understand what's actually happening: when a pod starts terminating, kubelet marks it not-ready immediately, while removal from the Service's Endpoints and the delivery of SIGTERM (after the preStop hook completes) all proceed asynchronously and in parallel — there's no guaranteed ordering between endpoint removal and SIGTERM. The preStop sleep exists to cover the propagation lag while endpoints update, giving load balancers time to stop routing new connections before the container actually shuts down; it doesn't sequence readiness ahead of anything.
At the fleet level, readiness state feeds into two mechanisms that get conflated in practice. The Horizontal Pod Autoscaler ignores unready pods entirely when computing its scaling metrics, so flapping readiness across a fleet can skew HPA decisions without anyone noticing why. Separately, the ReplicaSet controller prefers to delete unready pods first during voluntary scale-downs, and eviction during a PodDisruptionBudget-gated disruption is decided by how many pods in the set are currently marked ready — not by whether the specific pod being evicted has a request in flight. A pod mid-request can still be selected for eviction if the PDB's ready-pod count permits it; readiness alone doesn't protect an individual request, only the drain sequence (preStop, connection draining, grace period) does that. For broader patterns on tuning cluster-level resilience, see the related posts on kuryzhev.cloud covering Kubernetes rollout and reliability topics.
Performance notes
Probe frequency is not free, though the cost is paid locally by the kubelet rather than by etcd. Kubelet does not push a pod status update to the API server on every successful check — only state transitions (for example, a readiness flip from False to True) get persisted. Aggressive intervals, such as periodSeconds: 1 or 2, across a fleet with thousands of pods add overhead that scales with pod count and probe frequency: more open connections, more scheduling work inside kubelet's probe manager, more contention for CPU on that node.
The probe mechanism matters too. exec-based probes fork a new process inside the container for every single check, which is an inherently heavier operation than httpGet or tcpSocket, both of which kubelet handles without spawning a subprocess. At high pod density that per-check forking is a reasonable place to expect CPU pressure, though the actual magnitude depends on image size and runtime and isn't something to assume without checking node metrics directly. Prefer HTTP or gRPC health endpoints over shell one-liners whenever the application can expose them.
Security note worth flagging: exec probes running shell commands inside the container widen the attack surface if the probe script reads from writable paths or is influenced by environment-controlled input — a compromised writable volume could, in principle, alter probe behavior. A typed HTTP or gRPC health endpoint avoids that class of risk entirely and is generally the more auditable choice.
When triage is needed, the sequence below is the standard first pass for probe-related CrashLoopBackOff or stalled-rollout incidents:
# Triage sequence when pods are stuck in CrashLoopBackOff due to probes
1. kubectl describe pod <name>
→ look for "Liveness probe failed" or "Readiness probe failed" events
2. kubectl get pod <name> -o jsonpath='{.status.containerStatuses[0].lastState.terminated.reason}'
kubectl get pod <name> -o jsonpath='{.status.containerStatuses[0].lastState.terminated.exitCode}'
→ a probe-triggered kill typically shows reason Error and exitCode 137,
but that alone doesn't rule out an OOM kill from elsewhere — cross-check
with the events from step 1
3. Check if liveness checks a downstream dependency
→ if yes, that's the bug: move the dependency check to readiness
4. Check startupProbe presence
→ missing startupProbe + slow boot = false liveness failures during init
5. Verify timing math
→ timeoutSeconds must exceed the worst-case per-request response time
→ periodSeconds * failureThreshold sets the total detection window —
that's a separate number, not the same knob
Decision table:
| Signal | Belongs in |
|------------------------------------|---------------------|
| Process deadlocked/unresponsive | livenessProbe |
| Dependency (DB/cache) unreachable | readinessProbe |
| Slow but legitimate startup | startupProbe |
| Graceful drain before shutdown | preStop + readiness |
Verify current field defaults and version-specific behavior against the Kubernetes API reference for the Probe object, since defaults for timeoutSeconds and periodSeconds have shifted slightly between releases and are worth confirming for the cluster actually in use.
Getting kubernetes liveness readiness probes right is less about memorizing YAML fields and more about respecting the contract: liveness answers "is the process broken," readiness answers "can it serve traffic right now," and startup buys legitimate initialization time without lying to either of the other two. Misconfiguration in this area rarely shows up as a code bug — it shows up as an outage that traces back to a health check nobody reviewed since the app was first deployed.
Top comments (0)