A Non-Root User in a Model-Serving Dockerfile

#ai #machinelearning #tutorial #llm

Adding USER to a Dockerfile takes one line. What takes the afternoon is the permission-denied error that appears the first time the container is run with the model weights on a mounted volume, and that error is entirely predictable once you know which piece of the user identity crosses the boundary.

What running as root actually costs

A container that does not set USER runs as uid 0. That is not the same as root on the host — the default runtime drops most capabilities and applies a seccomp profile — but it is root inside the namespace, which is enough to matter in three concrete ways for an inference service.

Every file the process can reach is writable. A model server that loads weights, a tokenizer and a config from disk has no reason to be able to rewrite any of them. As uid 0 it can rewrite all of them, and so can anything that achieves code execution through it.
A mounted host path is written as root. Anything the container creates on a bind-mounted directory ends up owned by uid 0 on the host, which is how a cache directory becomes something the developer who mounted it cannot delete.
Cluster policy will eventually reject it. The Kubernetes restricted Pod Security Standard requires runAsNonRoot: true, and an image with no USER and no numeric user in its config fails admission with a message about the container having a runAsNonRoot setting and an unset or root user.

Adding the user

Create the account in the final stage, give it a fixed numeric id, and hand ownership over as part of the copy rather than in a separate RUN chown, which would duplicate the whole layer.

# final stage of a multi-stage build
FROM python:3.12-slim

RUN groupadd --gid 10001 app \
 && useradd --uid 10001 --gid 10001 --create-home --shell /usr/sbin/nologin app

WORKDIR /srv
COPY --from=build --chown=10001:10001 /opt/venv /opt/venv
COPY --chown=10001:10001 serve.py /srv/serve.py

ENV PATH=/opt/venv/bin:$PATH \
    HF_HOME=/var/cache/hf \
    PYTHONDONTWRITEBYTECODE=1

RUN install -d -o 10001 -g 10001 /var/cache/hf

USER 10001:10001
EXPOSE 8000
ENTRYPOINT ["python", "-m", "uvicorn", "serve:app", "--host", "0.0.0.0", "--port", "8000"]

Write USER 10001:10001 rather than USER app. The name is resolved through /etc/passwd inside the image, and nothing outside the image can see it; a Kubernetes runAsNonRoot check, a volume’s ownership and a host bind mount all deal in the number. Writing the number in the Dockerfile means the number is the thing you can grep for later.

Why the mounted model directory breaks

This is the step that introduces the failure. The image is correct, the user exists, and then the container is started with the weights mounted from somewhere else:

docker run --rm -p 8000:8000 -v /data/models/llama-3-8b:/models my-inference:1

PermissionError: [Errno 13] Permission denied: '/models/.cache'

Nothing about the mount is translated. The kernel compares the numeric uid of the process against the numeric owner of the inode on the host filesystem, and there is no mapping layer between them unless you asked for one. If /data/models on the host is owned by uid 1000 with mode 0755, then uid 10001 inside the container can read it and cannot write to it. The user account named app inside the image is irrelevant to that comparison; it may not exist on the host at all.

There are three honest ways out and one that is not. Make the host directory group-owned by a gid the container process holds and mode 0775; or run the container with an explicit --user 1000:1000 so the process id matches what already owns the files; or mount the weights read-only with :ro and point every writable path somewhere else. The way that is not honest is going back to uid 0, which is where most of these afternoons end.

Read-only is usually the right answer for model weights specifically. They are immutable artefacts. Nothing at serving time should be able to write to them, and mounting them :ro turns a permission question into a property of the deployment.

The three paths a model server writes to

Once the process is unprivileged, every write it makes has to land somewhere it owns. In a Python inference image there are reliably three, and each of them defaults to a location the new user cannot use.

The Hugging Face cache. The Hub client stores downloads and tokens under HF_HOME, which Hugging Face documents as defaulting to ~/.cache/huggingface, with HF_HUB_CACHE defaulting to $HF_HOME/hub. When the process has no home directory, or has one it cannot write to, the failure surfaces as a permission error deep inside a library call rather than at startup. Set HF_HOME explicitly and create the directory with the right owner in the build. Hugging Face documents these variables in the Hub library reference.
The compile and kernel caches. Torch inductor, Triton and CUDA all cache compiled artefacts on first use. They follow XDG_CACHE_HOME or the home directory, so they land in the same trap and cost a recompile on every cold start if they end up somewhere ephemeral.
Anything that writes .pyc files. Setting PYTHONDONTWRITEBYTECODE=1 removes a whole class of write attempts against directories the process should not own, and costs nothing at serving time because the interpreter compiles to memory anyway.

Having done that, --read-only on the container root filesystem becomes achievable, with a tmpfs for /tmp and a named volume for the cache. That is the configuration worth aiming at: the non-root user is the step that makes it possible, not the goal itself.

The same thing in a pod spec

In a cluster the same identity is set on the pod, and one field behaves differently from how people expect.

securityContext:
  runAsUser: 10001
  runAsGroup: 10001
  runAsNonRoot: true
  fsGroup: 10001
  fsGroupChangePolicy: OnRootMismatch
  seccompProfile:
    type: RuntimeDefault
containers:
  - name: server
    securityContext:
      allowPrivilegeEscalation: false
      readOnlyRootFilesystem: true
      capabilities:
        drop: ["ALL"]

fsGroup is the field that quietly does not solve the problem. The kubelet applies it by taking ownership of the volume contents for the supported volume types, and the Kubernetes documentation is explicit that this does not happen for every kind of volume: for multi-writer types such as NFS the cluster does not perform the recursive permission change, and a hostPath mount is the host’s own directory with the host’s own ownership. If your weights arrive over NFS or from a host path, fsGroup will not fix them and the ownership has to be right on the source.

fsGroupChangePolicy: OnRootMismatch is worth setting whenever the volume does support the recursive change, because the default policy walks and chowns every file on every mount. On a volume holding a multi-gigabyte checkpoint split across many shards, that walk is a startup cost paid on every pod, and it shows up as a slow readiness probe rather than as anything that names itself. One more reason to prefer read-only weights: a volume mounted read-only is not a candidate for the walk at all.