Health Checks in a Dockerfile for a Model-Serving Container

#ai #machinelearning #tutorial #llm

A model server binds its port in a second and is useless for the next several minutes. Any health check that tests the port is therefore lying, and the lie is expensive: it is what makes an orchestrator route the first request into a process with no weights in memory.

Why the obvious check is wrong

The reflexive check is a TCP connect or a request to /. Both succeed as soon as the HTTP server is listening, which happens before the checkpoint is read from disk, before tensors are moved to the GPU, and before any warm-up pass. For a web application the gap is milliseconds and nobody notices. For a model server it can be minutes, and it is exactly the window in which everything goes wrong.

There is a second failure in the other direction, and it is the reason --start-period exists. Give the container a check with the default timings and a slow load, and the checks fail from the moment the container starts. After the retry budget is exhausted the container is marked unhealthy — and anything watching that status, including a compose depends_on condition or a swarm scheduler, acts on it while the container is doing precisely what it should.

The options and their defaults

Docker’s Dockerfile reference documents five options on HEALTHCHECK:

--interval — time between checks once running. Default 30 seconds.
--timeout — how long a single check may take before it counts as a failure. Default 30 seconds.
--start-period — an initialisation window during which a failure does not count against the retry budget. Default 0 seconds, which is the wrong default for anything that loads a model.
--start-interval — the interval used during the start period, so a container can be probed frequently while starting and infrequently afterwards. Default 5 seconds. This one is newer than the others and needs a recent Docker Engine.
--retries — consecutive failures before the container is marked unhealthy. Default 3.

The check command’s exit status is the whole interface: 0 means healthy, 1 means unhealthy, and 2 is reserved and should not be used. A container has three health states — starting, healthy and unhealthy — and moves out of starting on the first success or after --retries consecutive failures outside the start period.

Docker’s Dockerfile reference

--start-interval was added later than the other four. If your build fails parsing it, the builder or engine predates the option; drop it and accept a coarser probe during startup rather than pinning an old Docker version.

Give the server something true to report

A health check can only be as honest as the endpoint it calls. If your only endpoint is the inference route, the check either does real work on every probe — expensive, and on a GPU it competes with traffic — or it tests nothing. Add an endpoint whose value is set by the loading code:

from fastapi import FastAPI, Response

app = FastAPI()
_model = None

@app.on_event("startup")
async def load_model() -> None:
    global _model
    _model = load_checkpoint("/models/current")   # minutes

@app.get("/healthz")
def healthz(response: Response):
    if _model is None:
        response.status_code = 503
        return {"status": "loading"}
    return {"status": "ok"}

The point is that the flag is set by the same code path that finishes the load, so it cannot drift. A boolean set optimistically at the top of startup is worse than no check at all, because it makes the container look ready earlier than the naive port check would.

If the model is loaded lazily on first request, set the flag after a warm-up call in the startup hook rather than reporting healthy immediately — otherwise the first real request pays the entire load time and probably times out.

Write the HEALTHCHECK

HEALTHCHECK --start-period=10m \
            --start-interval=10s \
            --interval=30s \
            --timeout=5s \
            --retries=3 \
  CMD python -c "import urllib.request,sys; sys.exit(0 if urllib.request.urlopen('http://localhost:8000/healthz').status==200 else 1)"

Set --start-period from a real cold start. Time the container from launch to first successful /healthz on the slowest storage path it will use, then roughly double it. Too short and a cold node is marked unhealthy; too long only delays detection of a genuine failure during startup, which is the cheaper mistake.
Keep --timeout short. The check should be a memory read behind an HTTP handler. If it can take five seconds, it is doing too much, and a slow check under load will fail the container precisely when it is busiest.
Do not call an external service. A check that queries a vector store or a provider API makes your container unhealthy when their dependency is degraded. Health is about this process.
Use an interpreter the image has. The command runs inside the container, and slim, distroless and hardened images routinely have no curl. Python is present in a Python image; curl may not be.
Verify it. Run the container, then docker inspect --format '{{json .State.Health}}' ctr | jq shows the current status and the last five probe results with their exit codes and output. Read the output field — a check failing because the binary is missing looks identical from the outside to one failing because the model has not loaded.

What a health status does not do

Docker records the status. On its own it does not restart the container, does not stop it, and does not remove it from anything. Something else has to act on it, and what that something is differs by platform: compose can gate depends_on on it, Swarm reschedules on it, and a bare docker run does nothing but display it.

Kubernetes ignores it entirely. It does not read the image’s HEALTHCHECK; it runs the probes declared on the pod spec. So the same readiness logic must be expressed twice for a container that runs in both places — as a HEALTHCHECK for local and compose use, and as a readiness probe in the manifest. Point both at the same /healthz endpoint so there is one definition of ready and two callers of it.

One last distinction worth keeping straight, because Docker has one concept where Kubernetes has three. A liveness probe asks “should this be killed”, a readiness probe asks “should this receive traffic”, and a startup probe suppresses the other two while a slow process initialises. A single HEALTHCHECK answers only the middle question well, so write it as a readiness check and do not expect it to catch a wedged process.