Distroless Base Images for a Model-Serving Container

#ai #llm #machinelearning #programming

A distroless image is a container image with a runtime and nothing else: no shell, no package manager, no ls, no curl. For a model server that is a good trade, right up until the first production incident, when every habit you have for looking inside a container stops working at once.

What is actually removed

Google’s distroless project describes its images as containing “only your application and its runtime dependencies”, with no package managers, shells or other programs you would expect in a Linux distribution. What remains in the base variants is glibc, a CA certificate bundle, timezone data, and the /etc/passwd entries for a small number of users. The Debian userland is not there.

The project publishes a ladder of images, and picking the right rung is most of the decision: static for a fully static binary, base for one that needs glibc, cc when C++ runtime libraries are required, and language-specific images for Java, Node and Python 3. Each carries nonroot, debug and debug-nonroot tag variants alongside the default. The distroless repository is the authority on which images and Debian versions currently exist, and that list changes as Debian releases move.

The image names, Debian version suffixes and language versions in that list are the volatile part of this page. Read them from the repository rather than from any article, including this one — a tag that existed a year ago may have been retired.

Why it is worth it for an inference image

The usual argument is image size, and for an inference container that argument is weak: the base is a rounding error next to a CUDA runtime and a set of weights. The arguments that survive are about what an attacker finds after achieving execution, and about what a scanner reports.

No shell means no shell-based next step. A large fraction of the tooling that runs after a remote code execution assumes it can spawn /bin/sh, then reach for curl or wget to fetch a payload. None of that is present. It is not a boundary, but it raises the cost of the second step considerably.
No package manager means the image cannot change. A running container that can apt-get install is a container that can acquire capabilities it was not built with.
The CVE list gets short and stays relevant. Most findings a scanner reports against a full distribution base are in packages the service never invokes, which trains everyone to ignore the report. Removing them means the remaining findings are about code that actually runs. The static variant is around 2 MiB, which is the size of what has to be triaged, not just the size of the download.
It composes with the non-root work. The nonroot tags already set the image user to uid 65532, so the non-root user step becomes a matter of getting ownership right on what you copy in rather than of creating an account.

What breaks the day you switch

Four things, all of them the same underlying fact stated differently.

docker exec fails. You get
```
OCI runtime exec failed: exec failed: unable to start container process: exec: "sh": executable file not found in $PATH: unknown
```
because there is no sh. Same for bash, and same for kubectl exec, which produces the same message wrapped in an API error.
Shell-form ENTRYPOINT and CMD stop working. ENTRYPOINT python -m uvicorn ... is compiled by the builder into an exec of /bin/sh -c, which does not exist. Only the JSON exec form runs. The knock-on effect is that no environment variable in the command line is expanded any more: a command containing --port $PORT passes the four characters to the program. Read the port from the environment inside the program instead.
HEALTHCHECK is usually dead. Almost every healthcheck in circulation is curl -f against a local endpoint, and curl is not there. In Kubernetes this is not a loss, because an httpGet probe is performed by the kubelet from outside the container and needs nothing inside it. Under plain Docker or Compose, either compile a tiny static health binary in the build stage and copy it in, or check liveness from outside.
Runtime installation is gone, permanently. Anything that downloads a model at first request with a helper binary, or that shells out to nvidia-smi, needs that dependency copied in explicitly during the build.

For a Python inference service there is one more consequence worth planning for: the distroless Python image has no pip. Dependencies have to be installed in an earlier build stage into a virtual environment or a target directory, then copied wholesale. Because the distroless images are Debian-derived they use glibc, so wheels built for manylinux work directly — this is a real advantage over an Alpine base, where musl means either a source build or a separate wheel set.

Debugging without a shell

The replacement is not to keep a shell around. It is to attach one from outside, at the moment you need it, to a container that never had one.

In Kubernetes, use an ephemeral container: kubectl debug -it POD --image=busybox:1.36 --target=server -- /bin/sh. The --target flag shares the process namespace of the named container, so ps shows the model server’s processes and /proc/1/root reaches into its filesystem. The pod is not restarted and the image is not modified.
To inspect the filesystem without a running pod at all, use a copy: kubectl debug POD --copy-to=POD-debug --image=busybox:1.36, which builds a new pod from the same spec with an extra container.
Under plain Docker, run a sidecar sharing namespaces: docker run --rm -it --pid=container:my-inference --network=container:my-inference busybox:1.36 sh. Network debugging works immediately because the namespace is shared; the filesystem is reached through /proc.
When you genuinely need the shell inside the image, rebuild against the :debug tag, which adds a busybox shell. Deploy it to reproduce, then go back. This is the escape hatch, not the default, because a :debug image that ends up in production has given back the entire benefit.

Choosing a variant

Start from what the process actually links against, which is a question ldd answers in the build stage before you have thrown the tools away. A statically linked Go or Rust proxy takes static. A CPython process needs the Python variant. Anything that pulls in a C++ runtime — which includes a good deal of native ML tooling — needs cc or a base that carries libstdc++.

The case where distroless is the wrong choice is a GPU container. A CUDA workload needs the vendor’s runtime libraries and the driver interface presented by the container toolkit, and those arrive as a vendor base image built on a normal distribution. Trying to reassemble that on a distroless base is a large amount of work for a small security gain, and the effort is better spent making the CUDA image non-root and read-only. The rule that survives is the one distroless embodies rather than the images themselves: nothing in the runtime layer that the service does not call.