DEV Community

Suresh Kumar Pallapothu
Suresh Kumar Pallapothu

Posted on Originally published at sureshpallapothu.in

Day 3: Hardware Acceleration in Docker — GPU Passthrough and Limits for Constrained Hardware

#ai #docker #gpu #aiintermediateseries

A leading reason AI workloads fail in containers is the boundary between the container's isolated namespace and the host's hardware. Running inference on raw CPUs creates big latency bottlenecks, and exposing a GPU or accelerator takes deliberate host setup.

The acceleration architecture

A container cannot talk to a PCIe device without driver support. The NVIDIA Container Toolkit supplies it by wrapping the standard runtime with a hardware-aware one (nvidia-container-runtime).

Diagram: FIG 1 · The NVIDIA Container Toolkit injects the host's driver libraries and device nodes into the container at launch. Your image does not bundle the kernel driver. See the animated version.

Three ideas make it work.

Diagram: FIG 2 · Decoupling the driver from the image is what makes accelerated containers portable. Compatibility between the host driver and your image's CUDA version still matters. See the animated version.

  1. Driver decoupling. The host runs the kernel driver (nvidia.ko). The image never contains it.
  2. Runtime injection. At launch the toolkit injects the matching user-space driver library (libcuda.so) and the device nodes (/dev/nvidia*) into the container.
  3. Application agility. The image needs only the high-level CUDA runtime libraries, so it does not depend on the host's driver build. The host driver must still be recent enough for the CUDA version inside the image.

Launching an accelerated container

With the toolkit installed, register it with Docker and restart the daemon.

sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker
Enter fullscreen mode Exit fullscreen mode

That edits /etc/docker/daemon.json. You can also set it by hand, and making it the default runtime is optional, since --gpus selects it per container.

{
  "default-runtime": "nvidia",
  "runtimes": {
    "nvidia": { "path": "nvidia-container-runtime", "runtimeArgs": [] }
  }
}
Enter fullscreen mode Exit fullscreen mode 
docker run -d --name local-inference \
  --gpus all \
  --shm-size="2g" \
  -p 8000:8000 \
  enterprise-model-server:latest
Enter fullscreen mode Exit fullscreen mode

Verify it: docker run --rm --gpus all ubuntu nvidia-smi should print the same GPU table as the host.

Why --shm-size matters

PyTorch passes tensors between worker processes through shared memory (/dev/shm). Docker's default is only 64 MB, so parallel data loading can crash suddenly with bus errors. Raising it, for example to 2g, avoids that.

Diagram: FIG 4 · PyTorch uses shared memory to pass tensors between worker processes. Docker's 64 MB default is too small for that, so raise it with --shm-size. See the animated version.

Hard limits for a 16 GB machine

An unconstrained container is a danger to the host. A multi-step prompt can grow the KV-cache and demand several extra gigabytes of RAM at once, so enforce limits with Linux cgroups.

docker run -d --name constrained-model \
  --cpus="4.0" \
  --memory="10g" \
  --memory-swap="10g" \
  -p 11434:11434 \
  ollama/ollama:latest
Enter fullscreen mode Exit fullscreen mode

Diagram: FIG 3 · An unconstrained model container can take the whole machine down. Kernel cgroup limits keep the damage inside the container. The 10 GB cap is an example for a 16 GB host. See the animated version.

  • --memory="10g" caps the container at 10 GB and leaves about 6 GB for the host OS, networking daemons and file cache.
  • --memory-swap="10g" (equal to --memory) disables swap for the container. Swapping model weights to disk slows inference by orders of magnitude, so it is better to fail fast.
  • When the container exceeds its cap, the kernel's OOM killer stops that container, not the host. This is the default behaviour (--oom-kill-disable=false), so you do not need to set it. Never use --oom-kill-disable=true without a memory limit, because then a spike can hang the whole machine.
  • Add --gpus only when a GPU is present. On a CPU-only laptop, the memory and CPU caps are what protect you.

Treat these flags as part of your deployment definition, and in Kubernetes you will express the same limits as resource requests and limits.

Run it yourself

This is the exact example from the video. These are real docker run commands that read the limits from inside the container. There is no GPU on my laptop, so only the limits half of the lesson is shown.

# no GPU on this machine, so limits only
docker run --rm --cpus=2 --memory=256m --memory-swap=256m \
  --shm-size=2g alpine:3.20 sh -c '
  cd /sys/fs/cgroup
  for f in memory.max memory.swap.max cpu.max; do
    echo "$f: $(cat $f)"
  done
  echo shm: $(df -h /dev/shm | tail -1 | tr -s " " | cut -d" " -f2)'

docker run --rm --memory=64m --memory-swap=64m alpine:3.20 \
  sh -c 'x=$(dd if=/dev/zero bs=1M count=200 | tr "\0" x)'
echo "over the cap: exit code $?"
Enter fullscreen mode Exit fullscreen mode

Run it:

bash limits.sh
Enter fullscreen mode Exit fullscreen mode

Output I got while recording the video:

memory.max: 268435456
memory.swap.max: 0
cpu.max: 200000 100000
shm: 2.0G
over the cap: exit code 137
Enter fullscreen mode Exit fullscreen mode

Coming Up Next

Day 4: Optimising model weights: quantization basics with GGUF and AWQ.

HardwareAcceleration #Docker #GPUs #DevOps #Linux #TechInfrastructure #CUDA #SystemAdministration

Originally published at https://sureshpallapothu.in/blog/ai-int-day-3-gpu-passthrough, where this post includes animated diagrams.

Top comments (0)