Honey Kiran

Posted on Oct 22 Originally published at honeykiran.substack.com on Oct 21

Your Liveness Probe Is a Kill Switch on a Timer

#kubernetes #devops #platformengineering #sre

Pull up a chair. Today on r/devops, someone posted a thread titled: “Kubernetes probes keep restarting services that just need more time.” I read that title and sighed into my chai, because every engineer who has run Kubernetes for more than a week has watched this exact tragedy.

You deploy your app. The pod goes Running. You smile. Ten seconds later the restart counter ticks to 1. Then 2. Then the status flips to CrashLoopBackOff — Kubernetes throwing its hands up: I tried restarting this thing, it keeps failing, so now everyone waits.

This Substack is reader-supported. To receive new posts and support my work, consider becoming a free or paid subscriber.

You check the logs, expecting a stack trace. There is none. The last line says something innocent like Loading configuration... or Warming up local cache...

Your app was not broken. It was not deadlocked. It was busy getting ready for work. But the cluster saw a container that had not answered yet, assumed it was dead, and shot it.

Slow is not dead. Your cluster just cannot tell the difference until you teach it.

Three probes, three questions, three punishments

In Kubernetes, health checks are called probes. They are not run by some wizard in the control plane. They are run by the kubelet — the agent that lives on every node and babysits the containers assigned to it.

The kubelet runs three probes. They are not interchangeable. Each asks a different question, and each carries a different punishment.

ProbeQuestion it asksWhat happens when it fails Startup Has it finished starting?Nothing — until its whole budget runs out. Then the container is killed and restarted. Liveness Is it deadlocked?The kubelet kills the container and restarts it, subject to the pod’s restart policy. Readiness Can it take traffic right now?Nothing is killed. The pod’s IP leaves the Service’s endpoints, so traffic stops until it recovers.

Readiness is the gentle one: no traffic for you until you feel better. It runs for the container’s whole life, and its punishment costs you a few seconds of capacity, never the container.

Liveness is capital punishment: death and restart. It exists for exactly one situation — an app so deadlocked that no amount of waiting will save it, where the only cure is off and on again.

Attach that punishment to a container that is still booting, and you have not built a health check. You have built a kill switch on a timer.

The 30-second fuse

Why does this happen out of the box? The defaults. Every probe carries the same knobs:

initialDelaySeconds — wait before the first check. Default: 0.
periodSeconds — how often the check repeats. Default: 10.
timeoutSeconds — how long a check may take before it counts as failed. Default: 1.
successThreshold — successes needed to call it healthy. Default: 1.
failureThreshold — failed checks in a row before the punishment lands. Default: 3.

Now add a standard livenessProbe to a slow-booting app and touch nothing else, because defaults feel harmless. Here is the math the kubelet runs. The container starts; the first check fires almost immediately and fails — the port is not open yet, the app is still reading its config. Strike one. Ten seconds later, strike two — it is loading dependencies now. Ten seconds after that, strike three. Roughly thirty seconds into the boot, the kubelet kills the container and starts a fresh one — which boots the same slow way, and gets shot at the same thirty-second mark.

Repeat that a few times and the kubelet starts backing off exponentially between restarts. That is your CrashLoopBackOff.

The classic band-aid is initialDelaySeconds: 60. I dislike it. It is a blind guess wearing a number. If the app boots in 8 seconds, the pod sits unchecked for most of a minute. If a busy node or a slow disk stretches one boot to 65 seconds, the guess fails and the kill switch fires anyway.

A liveness probe should be your last line of defense against a dead process — not a test of whether your app can boot in thirty seconds flat.

The fix: give the boot its own probe

Kubernetes has a probe built for exactly this: the startup probe. Its rule is simple and beautiful — while a startup probe is configured and has not yet succeeded, Kubernetes does not run the liveness or readiness probes at all. They sit on the bench and wait.

The startup probe runs only during boot, and its patience is a budget you compute:

Startup budget = failureThreshold × periodSeconds

The official Kubernetes documentation’s own example sets failureThreshold: 30 and periodSeconds: 10 — 30 × 10 = 300 seconds, a five-minute runway. If your app answers at second 40, the startup probe succeeds once, retires forever, and liveness and readiness take over the watch. If the app truly hangs and burns all thirty checks, only then is it killed.

apiVersion: v1
kind: Pod
metadata:
  name: slow-starter-demo
spec:
  containers:
  - name: app
    image: example-app:1.0
    ports:
    - containerPort: 8080
    startupProbe:
      httpGet:
        path: /healthz
        port: 8080
      periodSeconds: 10
      failureThreshold: 30 # 30 x 10s = 300s to finish booting
    livenessProbe:
      httpGet:
        path: /healthz
        port: 8080
      periodSeconds: 10
      failureThreshold: 3 # strict again — this one is for deadlocks
    readinessProbe:
      httpGet:
        path: /ready
        port: 8080
      periodSeconds: 5

Notice the trick: the startup probe and the liveness probe ask the same endpoint. The startup probe is just the patient version, standing guard over the boot so liveness never has to be lenient. Your liveness stays tight — a genuinely deadlocked app still gets caught and restarted in about thirty seconds. Fast recovery, no friendly fire.

Liveness discipline: never check the database

One more trap, and it is the most dangerous one in this whole topic. Kubernetes gives you four probe mechanisms — httpGet (success means a status from 200 to 399), tcpSocket, exec, and grpc (stable since Kubernetes 1.27). The danger is not the mechanism. It is what your endpoint checks.

The classic mistake: the /healthz handler checks the process, then pings the database, then the cache, then a downstream API — and returns 500 if any of them sulk. Wire that to a liveness probe and walk through a database hiccup with me. The database stops accepting connections for a minute. Every pod’s liveness check now fails. Every kubelet in the cluster does its duty and kills its pods — all of them, at roughly the same time. The fresh pods boot straight back into the same dead database, fail the same check, and get killed again, while their boot storms hammer the very database that was trying to recover.

The Kubernetes documentation warns about this in plain words: incorrect liveness probes lead to cascading failures — restarts under high load, failed client requests, and extra workload dumped on the pods still standing.

So the rule, stated flat: liveness checks only the app itself. Is the process breathing? Is it deadlocked? Nothing outside the container’s own skin. Dependency checks belong in the readiness probe — when the database goes down, readiness fails, traffic drains away, and the pod sits quietly with its connections and memory intact, waiting. Nobody gets shot over a dependency.

The new slow starters

Slow boots are not a legacy problem. They are getting worse.

JVM services warming up and apps hydrating big in-memory caches were already blowing past a thirty-second fuse years ago. Now add the newest tenants of our clusters: model servers. A GPU inference server can spend minutes loading model weights into memory before it can answer a single request — and that is a healthy, normal boot. Drop one behind default liveness settings with no startup probe and it will never serve a request in its life. It will load weights, get shot, load weights, get shot.

The defaults suit a small, fast service. Plenty of what we run in 2026 is neither. Bridging that gap is not the app’s job. It is ours.

The probe audit

Run your manifests against this list:

Every slow starter has a startupProbe . If a boot takes more than a few seconds, liveness never guards the boot window again.
The startup budget covers the worst boot you have measured — on a cold, busy node, not on your laptop. Budget = failureThreshold × periodSeconds.
Startup and liveness share one endpoint. Same /healthz, two patience levels. No extra endpoint to maintain.
Liveness checks nothing external. No database, no cache, no downstream API. If it reaches outside the container, move that check to readiness.
Readiness checks the real dependencies. If the app cannot serve honestly without its database, readiness should know before your users do.
timeoutSeconds is honest. A busy single-threaded event loop can easily blow a 1-second timeout under load — and a timeout counts as a failure.
Heavy exec probes stay off dense nodes. Every exec check forks processes inside the container; on a crowded node that overhead is real. Prefer httpGet, tcpSocket, or grpc.
Alert on restart counts, not just availability. A pod killed and restarted all day can still show green between executions. Restarts are the smoke; go look for the fire.

The takeaway

A kubelet has no emotional intelligence. It holds a stopwatch and a restart button, and it does exactly what your numbers tell it to do. So set the numbers like someone who has read them: a startup probe with a real budget for the boot, a liveness probe that only ever asks “are you deadlocked?”, and readiness carrying every dependency check. Slow is not dead — stop letting a thirty-second fuse treat it that way.

Honey Kiran · DevOps & Platform Engineering

I write about SRE, Kubernetes, Terraform, and platform engineering — the defaults that bite, the incidents that teach, and the guardrails that hold.

What is the worst boot time hiding in your cluster right now — and which probe is guarding it? Tell me in the replies; the slowest honest answer wins.

Disclaimer

This post describes my personal experience with my own systems and accounts, in my own words. It is a first-person account of what I observed and what I do in my own work — not a statement of any company’s intentions, policies, or internal decisions. Where I do not know something, the post says so.
I am not affiliated with, endorsed by, or speaking for any company, product, or vendor named in this post. All product and company names are trademarks of their respective owners and are mentioned only to describe the experience accurately.
Any error messages, documentation excerpts, or figures quoted here are reproduced from my own records or from the public sources linked in the post, believed accurate at the time of writing. Product behavior changes over time; verify against current official documentation before relying on any detail.
Nothing in this post is professional, financial, legal, or security advice for your specific situation. Examples are illustrative — validate in your own environment before running anything in production.
Corrections are welcome. If any company named here publishes an official statement, or if any detail is wrong, I will update this post and note the correction. The goal is an accurate public record, not a verdict.

Examples are illustrative — validate in your own environment before running in production.