Installing the NVIDIA Device Plugin for Kubernetes

#ai #machinelearning #llm #tutorial

The device plugin is the component that makes a GPU visible to the Kubernetes scheduler. It is also the component people blame when the real problem is one layer below it, so this page installs it in the order that isolates each layer.

Three layers, three failures

A GPU pod works only when all three of these are true, and each one fails differently:

The driver is loaded on the host. Fails as nvidia-smi not existing or erroring on the node itself.
The NVIDIA container runtime is configured. Fails as containers that start normally and see no /dev/nvidia*.
The device plugin DaemonSet is running. Fails as nvidia.com/gpu being absent from node allocatable, so pods never schedule at all.

The confusing case is a plugin pod that is Running and Ready while advertising zero devices, which happens when layer two is missing. The plugin is healthy; it simply found nothing to advertise. Checking the pod status is therefore not a check.

The prerequisite that is not optional

NVIDIA’s README for the plugin lists its prerequisites explicitly: NVIDIA drivers, nvidia-container-toolkit version 1.7.0 or newer, and — the one that gets skipped — the NVIDIA container runtime configured as the default low-level runtime on the node.

Default matters. The toolkit can be installed and registered as an available runtime while runc remains the default, in which case ordinary containers get no device injection. The plugin’s own pods are ordinary containers, so they see no GPUs and advertise none. On containerd this is the default_runtime_name setting in /etc/containerd/config.toml; on a managed node group with a GPU-optimised AMI it is set for you, which is the main reason a managed image is worth using. The container toolkit page covers the Docker-side equivalent.

Version numbers on this page reflect the plugin’s published README at the time of writing. Check the current release before pinning anything — the plugin, the toolkit and the driver each have their own compatibility window.

Installing the DaemonSet

There are two supported paths. The static manifest is one command and no chart repository, which makes it the right choice for a first install on a cluster you are still proving out:

kubectl create -f https://raw.githubusercontent.com/NVIDIA/\
k8s-device-plugin/v0.17.1/deployments/static/nvidia-device-plugin.yml

The Helm chart is what NVIDIA recommends for anything ongoing, because every configuration option on this cluster — time-slicing, MIG strategy, per-node config selection — is a chart value rather than a manifest edit:

helm repo add nvdp https://nvidia.github.io/k8s-device-plugin
helm repo update

helm upgrade -i nvdp nvdp/nvidia-device-plugin \
  --namespace nvidia-device-plugin \
  --create-namespace \
  --version 0.17.1

One thing to check in the rendered DaemonSet before applying it on a cluster with tainted GPU nodes: the plugin must tolerate whatever taint those nodes carry, or it will not be scheduled onto the very nodes it exists to inspect. That circularity — tainted GPU nodes, plugin cannot land, no GPUs advertised, no GPU pods anywhere — is the single most common self-inflicted outage in this area, and it is covered from the taint side on taints and tolerations for a GPU node pool.

Confirming allocatable

Wait for the DaemonSet to be ready: kubectl -n nvidia-device-plugin rollout status ds/nvdp-nvidia-device-plugin. One pod per node in the DaemonSet’s scope.
Read the number that matters, not the pod status:
```
kubectl get nodes -o custom-columns=\
NAME:.metadata.name,\
GPU:.status.allocatable.nvidia\.com/gpu
```
A GPU node should now show its physical GPU count. This is the success criterion for the whole page.
If it shows <none> or 0, read the plugin logs on that node: kubectl -n nvidia-device-plugin logs ds/nvdp-nvidia-device-plugin. A plugin that cannot see devices says so, and the message will point at the driver or the runtime rather than at Kubernetes.
Prove the end-to-end path with a one-shot pod that runs nvidia-smi -L, as on scheduling GPU pods. Allocatable being non-zero proves the accounting; only a container that lists a device proves the injection.

What the plugin tells the kubelet

Knowing the protocol makes the failure modes readable rather than mysterious. The device plugin API is documented by Kubernetes and is small. The plugin starts a gRPC service on a socket under /var/lib/kubelet/device-plugins/ — a hardcoded path, which is why the DaemonSet mounts that directory from the host — and then registers itself with the kubelet over /var/lib/kubelet/device-plugins/kubelet.sock. It must be serving before it registers, or registration fails.

After that, two calls do the work. ListAndWatch is a stream: the plugin sends the current list of devices and their health, and sends a new list whenever anything changes. That stream is where the allocatable count comes from. Allocate is called at container creation, and its response tells the kubelet what to inject — for NVIDIA, the device nodes and the environment that makes specific GPU UUIDs visible inside the container. So the scheduler’s decision and the device injection are two separate events, and confirming a GPU pod needs to test both.

Three behaviours fall out of this and each explains a real symptom:

Capacity and allocatable can disagree. Kubernetes documents that when a device is reported unhealthy the kubelet decreases the node’s allocatable count while capacity stays the same. A node showing capacity 8 and allocatable 7 is telling you a GPU has failed, and kubectl describe node shows both numbers side by side.
A pod already holding a failed device keeps it. The documentation is explicit that pods assigned to failed devices stay assigned; the workload starts erroring and the pod either fails or enters a crash loop depending on its restart policy. Nothing reschedules it for you.
A kubelet restart deletes every plugin socket. A new kubelet removes the sockets in that directory, and plugins are expected to notice and re-register. A plugin that does not is a node whose GPUs vanish from allocatable after an unrelated kubelet upgrade — restarting the plugin pod fixes it, which is why that advice appears so often without explanation.

When to use the GPU Operator instead

The device plugin alone assumes the driver and the runtime are already on the node, which is true of the GPU-optimised images the managed Kubernetes services publish. NVIDIA’s GPU Operator is the other shape: it installs and manages the driver, the toolkit, the plugin, GPU Feature Discovery, DCGM exporter and the MIG manager as one bundle reconciled by a controller.

The trade is specific rather than a matter of taste. The operator owns the driver, so kernel upgrades and driver upgrades become its problem instead of the node image’s — valuable on self-managed nodes, redundant and occasionally conflicting on a cloud GPU AMI that already ships a driver. If you are on managed node groups with a vendor GPU image, the plugin alone is the smaller thing to run. If you are building nodes yourself, the operator is doing work you would otherwise be doing in a machine image, and it is also the surface through which MIG configuration is applied.

Install one or the other. Running the standalone plugin alongside the operator’s plugin puts two DaemonSets in charge of the same devices, and the symptom is an allocatable count that flaps. NVIDIA publishes the plugin and its README on GitHub, and the GPU Operator has its own documentation set.