Oleksandr Kuryzhev

Posted on Nov 7 Originally published at kuryzhev.cloud

Kubernetes DRA Extended Resources: GPUs Without Device Plugins

#kubernetes #devops

Originally published on kuryzhev.cloud

A platform team has 200 training manifests that all say nvidia.com/gpu: 1. They also want per-model GPU selection, MIG slices and shared devices, which the device plugin model cannot express. Rewriting every manifest to use ResourceClaims would be a large, slow and risky project. DRA extended resources exist to remove that trade-off.

Kubernetes v1.37 graduated DRA extended resources to GA, as covered in the v1.37 release announcement. It matters now because the migration path is finally stable: you can move GPU nodes from a device plugin to a Dynamic Resource Allocation (DRA) driver without touching workload YAML.

Short answer: install a DRA driver for your GPUs, create a DeviceClass that declares the extended resource name it should satisfy, and keep requesting nvidia.com/gpu in resources.limits. Roll it out node pool by node pool. It applies if you run Kubernetes 1.37+ and want DRA features, but skip it if a device plugin already does everything you need.

What DRA extended resources actually do

Classic extended resources work like this. A device plugin on each node tells the kubelet "I have 8 of nvidia.com/gpu", and the count shows up in node allocatable. The scheduler treats it as an opaque integer. It cannot tell an H100 from a T4, and it cannot see topology or partitions.

DRA replaces that integer with structured data. A driver publishes ResourceSlice objects describing real devices and their attributes. The scheduler allocates specific devices through ResourceClaim objects, using CEL selectors in a DeviceClass or claim.

The extended resources bridge connects the two worlds. A DeviceClass can name an extended resource, for example nvidia.com/gpu. When a pod asks for that name in its limits, the scheduler creates a claim on the pod's behalf, allocates a device from the matching class, and records the result in the pod status. Per the KEP and docs, the pod author sees no difference, while the platform team gets DRA's allocation model underneath.

The details (field names, status fields, driver support) are documented in the DRA documentation. Confirm them against your exact patch version with kubectl explain deviceclass.spec.

How people use it wrong

Running both stacks on the same GPUs. A typical failure is installing the DRA driver while the device plugin DaemonSet is still running. Both then advertise the same physical devices through different mechanisms, and the system can over-promise. Pods land on a node, one path says the GPU is free, and the container fails at start or two pods fight over a device. The documented guidance is that a device is managed by one mechanism at a time.

Rewriting everything to ResourceClaims on day one. Teams read "DRA" and convert all manifests to resourceClaims plus ResourceClaimTemplate. That is only needed for workloads that need selection or sharing. The bridge exists so most pods can stay unchanged.

Treating it as a drop-in for any cluster. Mixed-version node pools, older kubelets, or a driver that does not support the extended resource mapping will produce pods stuck in Pending. Check your driver's release notes, not just the Kubernetes version.

Forgetting the accounting side. Quotas, cost dashboards and alerts often key off node allocatable nvidia.com/gpu. Once a pool moves to DRA, that number may no longer be the source of truth. Verify how your quota and metrics tooling treat DRA-allocated devices before you cut over.

The correct approach

Migrate one labeled node pool at a time. Follow these steps:

Cordon and drain the pool.
Remove the device plugin from it. For example, add node affinity to the device plugin DaemonSet that excludes the pool's label.
Install the vendor DRA driver.
Create the DeviceClass.
Uncordon the nodes.

The NVIDIA driver is distributed as the nvidia-dra-driver-gpu Helm chart. Follow its README for current values, because chart options change between releases.

This DeviceClass maps the legacy resource name onto devices from the driver. The driver name gpu.nvidia.com is NVIDIA's; substitute your vendor's. The CEL expression is the part to check against your driver's published attributes.

apiVersion: resource.k8s.io/v1
kind: DeviceClass
metadata:
  name: gpu.example.com-default
spec:
  selectors:
  - cel:
      expression: device.driver == "gpu.nvidia.com"   # match only this driver's devices
  extendedResourceName: nvidia.com/gpu                # pods requesting this name get a DRA-backed claim

🧪 Lab — try this in your browser

Kubernetes Manifest Linter — Lint multi-document Kubernetes YAML for production readiness and security, with line:col diagnostics, severity grouping, a 0-100 score and fix hints.

The workload stays exactly as it was. Nothing in it mentions DRA:

apiVersion: v1
kind: Pod
metadata:
  name: gpu-smoke-test
spec:
  restartPolicy: Never
  containers:
  - name: cuda
    image: nvcr.io/nvidia/cuda:<tag>   # pick a CUDA base image matching your driver
    command: ["nvidia-smi", "-L"]
    resources:
      limits:
        nvidia.com/gpu: 1               # unchanged legacy request

To verify, check three things:

The driver is publishing devices. kubectl get resourceslices should list one or more slices per GPU node.
A claim was generated for the pod. kubectl get resourceclaims should show a claim in the pod's namespace in an allocated state.
The pod status records the claim. Run kubectl get pod gpu-smoke-test -o yaml and look for the extended resource claim status section.

Then kubectl logs gpu-smoke-test should print one GPU line from nvidia-smi -L.

Watch out: if the pod stays Pending, run kubectl describe pod and read the scheduler events first. The usual causes are a missing extendedResourceName on the DeviceClass or a CEL expression that matches nothing.

Advanced patterns

Once a pool runs on DRA, you can mix styles. Most pods keep the legacy request. The few that need a specific GPU model, a MIG profile or a shared device use an explicit ResourceClaimTemplate with a CEL selector on device attributes. Run kubectl get resourceslices -o yaml to see the attribute names your driver really publishes before writing selectors.

You can also define several DeviceClasses for different tiers, for example one that excludes older cards. Teams then get access through namespaces and RBAC on claim templates. That moves policy from node labels and taints into declarative objects the scheduler understands.

Multi-pod jobs are another fit. v1.37 also ships Workload/PodGroup gang scheduling in beta, so the pods of a distributed training job can be scheduled together instead of a partial set starting and holding devices. Because it is beta, check the release notes for its feature gates, API shape and how it interacts with DRA allocation before relying on it.

Watch out: the bridge only covers the simple "N devices of a kind" case. If you need a pod to combine a legacy request and rich selection on the same device, test that exact combination first. The docs list the supported interactions.

When not to bother: a small fleet of identical GPUs with whole-device allocation and no sharing needs gets little from DRA. You would add a driver, new API objects to debug and a new failure surface in exchange for features you do not use. The device plugin is still a fine choice there.

Performance notes

The costs are different from the device plugin's, not necessarily bigger. The scheduler now runs a DynamicResources plugin that evaluates CEL selectors against ResourceSlices, and the control plane stores slices and claims. Large GPU fleets with many short-lived pods generate a claim object per pod, so watch API server object counts and etcd size.

I won't quote numbers, because they depend on cluster size and driver. Measure instead:

In Prometheus, compare scheduler_pending_pods before and after migrating a pool.
Compare the scheduler plugin duration histogram scheduler_plugin_execution_duration_seconds, filtered to the DynamicResources plugin.
Track pod startup time from scheduled to running, since the kubelet must call the driver to prepare devices before the container starts. Your driver documents its own metrics for that step.

For rollback, keep the old device plugin manifest ready and migrate pools one at a time. Reverting a pool then means draining it, removing the driver and restoring the plugin. For related platform patterns, browse more posts on kuryzhev.cloud. The authoritative reference for allocation semantics, status fields and supported drivers is the Kubernetes DRA documentation.

DEV Community