Readiness Probes for a Model-Serving Pod That Loads Slowly

#ai #machinelearning #llm #tutorial

A model server that takes four minutes to load weights and has a liveness probe with default settings will be killed at around thirty seconds, restarted, and killed again. The pod reports CrashLoopBackOff, the logs show no error, and nothing has actually crashed.

The CrashLoopBackOff that is not a crash

The default probe settings are documented and short: periodSeconds of 10, failureThreshold of 3, timeoutSeconds of 1, successThreshold of 1 and initialDelaySeconds of 0. A liveness probe on those settings gives a container roughly thirty seconds to answer before the kubelet concludes it is dead and restarts it.

That is a reasonable budget for a web service and hopeless for an inference server, which has to read weights from disk or object storage, move them onto the GPU and often compile or warm up kernels before it can serve anything. The restart loop then makes it worse: each restart re-reads the weights, and on a node pulling from a remote store, several pods restarting together can saturate the very bandwidth they need to finish loading.

The tempting fix is a large initialDelaySeconds on the liveness probe. It works, and it costs you the thing liveness is for: after a four-minute delay, a container that wedges in month two is detected just as slowly, because the delay applies to every restart, not only the first. The startup probe exists precisely to avoid that trade.

Three probes, three questions

Startup — “has it finished starting?” Kubernetes documents that liveness and readiness probes do not run until the startup probe succeeds. Once it succeeds it never runs again for that container.
Readiness — “should it receive traffic?” Failing removes the pod’s endpoint from its Service. It does not restart anything, which makes it the right probe for a temporary condition such as a full request queue.
Liveness — “should it be killed?” Failing past the threshold restarts the container. It should test something a restart would actually fix: a deadlocked request loop, a wedged CUDA context. It should not test a dependency, or one slow downstream service will restart your whole fleet.

Give each one its own endpoint if the server allows it. Pointing liveness at the same handler as readiness means any condition that makes the pod busy also makes it dead.

Setting the startup budget

The startup budget is failureThreshold multiplied by periodSeconds. That product is the maximum time the container gets to start before the kubelet kills it and applies the pod’s restart policy.

Measure the real load time from the logs of a pod that started successfully: the gap between the process starting and the server announcing it is listening. Include the image pull separately — it happens before the container starts, so it is not inside the startup budget, but it is inside your rollout time.
Multiply by a comfortable factor. A cold page cache, a busy node, a slower object-store read and a larger model variant all push the same container past a budget set to its best case.
Express the result as a long failureThreshold with a short periodSeconds, not the reverse. failureThreshold: 60 with periodSeconds: 10 gives ten minutes but notices success within ten seconds of it happening. failureThreshold: 2 with periodSeconds: 300 gives the same ten minutes and wastes up to five minutes of every startup.
Confirm with kubectl describe pod after a deploy: a startup probe that is failing while the model loads is normal and shows as probe failure events without restarts. Restarts mean the budget was too short.

The full probe block

    startupProbe:
      httpGet:
        path: /health
        port: 8000
      periodSeconds: 10
      failureThreshold: 60
      timeoutSeconds: 5

    readinessProbe:
      httpGet:
        path: /health
        port: 8000
      periodSeconds: 5
      failureThreshold: 3
      timeoutSeconds: 5

    livenessProbe:
      httpGet:
        path: /health
        port: 8000
      periodSeconds: 20
      failureThreshold: 3
      timeoutSeconds: 10

timeoutSeconds is raised from its default of 1 on every probe here, and that matters more on a GPU pod than elsewhere. A server saturated with inference work can take longer than a second to schedule the thread that answers an HTTP health check, so a one-second timeout turns heavy load into a restart — the worst possible response to heavy load. Note also that the timeout must be shorter than the period, or probes will overlap.

Two related settings often belong alongside this block. terminationGracePeriodSeconds on the pod should be long enough for in-flight generations to finish, since a long streaming response killed at the default 30 seconds is a truncated answer to a user. And a preStop hook with a short sleep gives the endpoints controller time to remove the pod from its Service before the process stops accepting connections, which removes the burst of connection errors that otherwise accompanies every rollout.

What this changes about rollouts

Readiness gates the rollout. A Deployment will not proceed past its maxUnavailable and maxSurge constraints until new pods are Ready, so a ten-minute startup budget means a rolling update of a large fleet takes a correspondingly long time. Two consequences follow.

First, progressDeadlineSeconds on the Deployment — ten minutes by default — can expire before a slow-loading pod is Ready, marking the rollout failed while it is in fact progressing normally. It needs to exceed the startup budget plus the image pull, not merely match it.

Second, maxSurge on a GPU Deployment means extra GPUs. A surge of one on a four-replica fleet requires a fifth GPU to exist during the rollout, and if it does not, the new pod is Pending, the old pod cannot be retired, and the rollout stalls with no error — the diagnosis is on the insufficient-GPU fix page. On a fixed-size GPU pool, maxSurge: 0 with maxUnavailable: 1 is the setting that actually completes, at the cost of running one replica short throughout.