DEV Community

Richard Okonicha
Richard Okonicha

Posted on

How To Set Up Kubernetes on AMD GPUs with ROCm

#kubernetes #amd #gpu #rocm

How To Set Up Kubernetes on AMD GPUs with ROCm

Introduction

Running Kubernetes on AMD GPU hardware is possible, but the path is not as well documented as the NVIDIA equivalent. The ROCm stack, the device plugin, the metrics exporter, and the container runtime each have their own quirks — and on AMD DevCloud, Docker and k3s use separate containerd instances, which breaks the normal image-pull flow.

This tutorial walks you through provisioning an AMD MI300X instance with ROCm preinstalled, installing a single-node k3s cluster, importing the ROCm image into k3s containerd, deploying the AMD device plugin and metrics exporter, and verifying that Kubernetes can schedule pods onto the GPU. You will also see how to validate the gRPC health path between the device plugin and the metrics exporter, patch a known rocm-smi --rasinject CLI bug, and recover from a stale kubelet checkpoint that blocks GPU scheduling.

When you are finished, you will have a working ROCm-on-Kubernetes cluster that you can use as a test substrate for GPU workloads.

Prerequisites

To complete this tutorial, you will need:

  • Access to an AMD DevCloud account with the ability to create GPU instances. You can request access through the AMD Developer website.
  • A local macOS or Linux machine with kubectl and docker installed and SSH access configured. You can install kubectl by following the Kubernetes documentation and Docker by following the Docker documentation.
  • Familiarity with Kubernetes concepts such as DaemonSets, device plugins, and feature gates. If you are new to Kubernetes, the Kubernetes Basics guide provides a helpful starting point.

Step 1 — Provisioning the AMD MI300X Infrastructure

Start by creating a droplet on AMD DevCloud that provides a single MI300X GPU with ROCm preinstalled. The gpu-mi300x1-192gb-devcloud plan gives you 192GB of host memory and the ROCm software stack under /opt/rocm.

If you create the droplet through the DevCloud web UI, select the rocm-7-14 image and attach your SSH key. The provisioning process takes roughly 90 seconds. Once the droplet is ready, SSH into it and verify the baseline ROCm installation:

rocm-smi --showid --showproduct
Enter fullscreen mode Exit fullscreen mode

The rocm-smi tool reports the GPU model, GFX architecture, and ROCm version. On a fresh MI300X droplet you should see output like this:

Output
GPU[0]  : Device Name: AMD Instinct MI300X VF
GPU[0]  : GFX Version: gfx942
ROCm: /opt/rocm -> /opt/rocm-7.0.2
rocm-smi: ROCM-SMI version 4.0.0+2b22ab01
Enter fullscreen mode Exit fullscreen mode

If these values match your droplet, the hardware and driver stack are ready.

Next, confirm that RAS (Reliability, Availability, and Serviceability) blocks are enabled with zero errors at baseline:

rocm-smi --showrasinfo
Enter fullscreen mode Exit fullscreen mode

You should see output like this:

Output
GPU[0]  : RAS UMC: Enabled, 0 correctable, 0 uncorrectable
GPU[0]  : RAS SDMA: Enabled, 0 correctable, 0 uncorrectable
GPU[0]  : RAS GFX: Enabled, 0 correctable, 0 uncorrectable
Enter fullscreen mode Exit fullscreen mode

UMC, SDMA, and GFX must show Enabled with 0 correctable and uncorrectable errors. A clean baseline matters because these blocks are what you will use to measure hardware health during testing.

You have now verified the GPU hardware and baseline ROCm installation. Next, you will install Kubernetes and configure it for GPU workloads.

Step 2 — Installing and Configuring k3s

DevCloud droplets ship with both Docker and k3s installed, but they use separate containerd instances. Because k3s cannot see images pulled by Docker, you must transfer the ROCm image from Docker into k3s containerd after disabling Docker. For this reason, stop and disable Docker before installing Kubernetes:

Execute the following commands to stop Docker and prevent it from restarting on boot. The systemctl disable command ensures Docker does not interfere with k3s containerd:

systemctl stop docker
systemctl disable docker
Enter fullscreen mode Exit fullscreen mode

Install k3s, a lightweight Kubernetes distribution that is well-suited for single-node clusters:

The curl -sfL flags follow redirects silently, fail on HTTP errors, and show errors. The INSTALL_K3S_SKIP_START=true environment variable tells the installer to skip starting k3s so you can configure it first:

curl -sfL https://get.k3s.io | INSTALL_K3S_SKIP_START=true sh -
Enter fullscreen mode Exit fullscreen mode

After the installation script completes, import the ROCm Docker image into k3s containerd so that the k3s runtime can use it:

docker save rocm:latest -o /tmp/rocm-latest.tar
ctr -n k8s.io --address /run/k3s/containerd/containerd.sock images import /tmp/rocm-latest.tar
Enter fullscreen mode Exit fullscreen mode

The first command saves the image from the Docker runtime, and the second command imports it into the k8s.io namespace of k3s containerd. Without this step, pods that request rocm:latest fail with ImagePullBackOff because k3s cannot access Docker's image store. This is not a theoretical problem — it is the first thing that breaks on a fresh DevCloud droplet.

Next, configure /etc/rancher/k3s/config.yaml to enable the feature gates and privileges that the AMD device plugin requires:

kubelet-arg:
  - "feature-gates=ResourceHealthStatus=true"
allow-privileged: true
Enter fullscreen mode Exit fullscreen mode

ResourceHealthStatus=true enables kubelet to surface per-device health in pod status. allow-privileged=true is required because the AMD device plugin needs access to /dev/kfd, /dev/dri, and /sys, along with the SYS_ADMIN capability for the health-check variant.

Start the k3s services and wait for the node to reach the Ready state:

systemctl start k3s
kubectl get nodes
Enter fullscreen mode Exit fullscreen mode

You should see a single control-plane node in the Ready state running k3s v1.36.3 or later. Your node name, age, and exact IP will differ:

Output
NAME                                                 STATUS   ROLES           AGE   VERSION
rocm-7-14-software-gpu-mi300x1-192gb-devcloud-atl1   Ready    control-plane   6h49m   v1.36.3+k3s1
Enter fullscreen mode Exit fullscreen mode

With the cluster running, you will now deploy the metrics exporter that exposes health data to Prometheus and the device plugin.

Step 3 — Configuring the Metrics Exporter

The AMD Device Metrics Exporter (rocm/device-metrics-exporter:v1.5.0) exposes Prometheus metrics and a gRPC health service on each node. Open a file named metrics-exporter.yaml in your text editor and add the following DaemonSet and ConfigMap:

apiVersion: v1
kind: ConfigMap
metadata:
  name: metrics-exporter-config
  namespace: kube-system
data:
  exporter-config.yaml: |
    CommonConfig:
      HealthService:
        Enable: true
        PollingRate: "10s"
      Debug:
        EnableAPI: true
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: amd-metrics-exporter
  namespace: kube-system
spec:
  selector:
    matchLabels:
      name: amd-metrics-exporter
  template:
    metadata:
      labels:
        name: amd-metrics-exporter
    spec:
      containers:
      - name: exporter
        image: rocm/device-metrics-exporter:v1.5.0
        command: ["./device-metrics-exporter"]
        args: ["--config", "/etc/metrics/exporter-config.yaml"]
        env:
        - name: AMD_ENABLE_HEALTH_GRPC
          value: "true"
        - name: AMD_DEBUG_ENABLE_API
          value: "true"
        securityContext:
          privileged: true
        volumeMounts:
        - name: socket-dir
          mountPath: /var/lib/amd-metrics-exporter
        - name: config
          mountPath: /etc/metrics
          readOnly: true
      volumes:
      - name: socket-dir
        hostPath:
          path: /var/lib/amd-metrics-exporter
          type: DirectoryOrCreate
      - name: config
        configMap:
          name: metrics-exporter-config
Enter fullscreen mode Exit fullscreen mode

The ConfigMap enables the gpu_health metric at a 10-second polling interval and turns on the debug gRPC API. The DaemonSet runs the exporter in privileged mode, mounts the host directory where the gRPC socket will be created, and passes the config via mounted volume. We use rocm/device-metrics-exporter:v1.5.0 because the latest tag on Docker Hub does not exist for this image.

Apply the manifest:

kubectl apply -f metrics-exporter.yaml
Enter fullscreen mode Exit fullscreen mode

Verify that the exporter is reachable on the node:

curl http://localhost:5000/metrics | grep gpu_health
Enter fullscreen mode Exit fullscreen mode

You should see a metric line similar to the following:

Output
gpu_health{card_model="",container="",gpu_compute_partition_type="spx",gpu_id="0",gpu_memory_partition_type="nps1",gpu_partition_id="NA",hostname="your_node_name",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number=""} 1
Enter fullscreen mode Exit fullscreen mode

A value of 1 indicates Healthy. The exporter also exposes a gRPC socket at /var/lib/amd-metrics-exporter/amdgpu_device_metrics_exporter_grpc.socket, which the device plugin uses to read health updates.

The metrics exporter is now exposing health data. Next, you will deploy the AMD device plugin to advertise the GPU to Kubernetes.

Step 4 — Deploying the AMD Device Plugin

The ROCm Kubernetes device plugin repository provides a health-check DaemonSet that exposes per-GPU health through the ListAndWatch stream. Apply the manifest directly from the upstream repository:

kubectl apply -f https://raw.githubusercontent.com/ROCm/k8s-device-plugin/master/k8s-ds-amdgpu-dp-health.yaml
Enter fullscreen mode Exit fullscreen mode

This manifest is the experimental health-check variant. It mounts /var/lib/amd-metrics-exporter/ into the device plugin container so the plugin can read the gRPC socket exposed by the metrics exporter. It enables -pulse=10, which sends a heartbeat every 10 seconds. It requests SYS_ADMIN capability and mounts /dev/kfd, /dev/dri, and /sys from the host.

Verify that the DaemonSet pod is running:

kubectl get pods -n kube-system -l name=amdgpu-dp-ds
Enter fullscreen mode Exit fullscreen mode

You should see one pod per node in the Running state with no restarts. If the pod restarts, inspect the logs:

kubectl logs -n kube-system -l name=amdgpu-dp-ds
Enter fullscreen mode Exit fullscreen mode

Look for lines such as Found 1 AMDGPUs and Watching GPU with bus ID: 0000:83:00.0. These confirm that the plugin discovered the GPU and is actively watching it.

The device plugin is now running and advertising the GPU. Next, you will verify that Kubernetes can schedule pods onto the GPU and that containers inside those pods can access the hardware.

Step 5 — Verifying GPU Scheduling and Compute Access

To confirm that the device plugin is advertising the GPU correctly, check the node allocatable resources:

kubectl get node -o jsonpath='{.items[0].status.allocatable}'
Enter fullscreen mode Exit fullscreen mode

You should see amd.com/gpu: "1" alongside CPU, memory, and pod capacity. The full allocatable on a fresh MI300X droplet looks like this:

{
    "amd.com/gpu": "1",
    "cpu": "20",
    "ephemeral-storage": "710372463469",
    "hugepages-1Gi": "0",
    "hugepages-2Mi": "0",
    "memory": "247409204Ki",
    "pods": "110"
}
Enter fullscreen mode Exit fullscreen mode

If the GPU is missing from allocatable, the device plugin has not yet registered; wait a few moments and inspect the DaemonSet logs.

Next, open a file named rocm-health-test.yaml in your text editor and add the following pod manifest:

apiVersion: v1
kind: Pod
metadata:
  name: rocm-health-test
spec:
  containers:
  - name: rocm
    image: rocm:latest
    command: ['sleep', '600']
    resources:
      limits:
        amd.com/gpu: 1
    imagePullPolicy: Never
  restartPolicy: Never
Enter fullscreen mode Exit fullscreen mode

Apply the manifest and watch the pod schedule:

kubectl apply -f rocm-health-test.yaml
kubectl get pod rocm-health-test -w
Enter fullscreen mode Exit fullscreen mode

When the pod reaches the Running state, execute a command inside it to verify GPU access:

kubectl exec -it rocm-health-test -- rocm-smi --showid --showproduct
Enter fullscreen mode Exit fullscreen mode

The output should show the same GPU model and GFX version that you observed on the host.

Output
GPU[0]: Device Name: AMD Instinct MI300X VF
GPU[0]: GFX Version: gfx942
Enter fullscreen mode Exit fullscreen mode

If rocm-smi reports No AMD GPUs were detected, the container is missing access to /dev/kfd or /dev/dri.

GPU scheduling and compute access are working correctly. Next, you will validate the gRPC health path to confirm that the device plugin and metrics exporter can communicate across namespace boundaries.

Step 6 — Validating the gRPC Health Path

To prove that the device plugin can reach the metrics exporter across namespace boundaries, verify the gRPC socket permissions and test a cross-namespace connection.

Inspect the socket on the host:

ls -l /var/lib/amd-metrics-exporter/amdgpu_device_metrics_exporter_grpc.socket
Enter fullscreen mode Exit fullscreen mode

You should see permissions 0777. On our cluster, the socket inode is 253,1:

Output
srwxrwxrwx 1 root root 0 Aug  7 15:24 /var/lib/amd-metrics-exporter/amdgpu_device_metrics_exporter_grpc.socket
Enter fullscreen mode Exit fullscreen mode

The inode must match the bind-mounted path inside the device plugin container. Replace <device-plugin-pod> with the actual pod name from Step 4:

kubectl exec -it -n kube-system <device-plugin-pod> -- ls -li /var/lib/amd-metrics-exporter/
Enter fullscreen mode Exit fullscreen mode

You should see output like this:

Output
total 0
253,1 7120801 srwxrwxrwx 1 root root 0 Aug  7 15:24 amdgpu_device_metrics_exporter_grpc.socket
Enter fullscreen mode Exit fullscreen mode

Compare the inode numbers shown on the host and inside the container. Identical inodes confirm that the mount is the same filesystem object.

If you want to test a live gRPC connection, you can build a small Go program that connects to the socket and calls the GetGPUState or List RPC. Copy the binary into the device plugin container with kubectl cp and run it. A successful connection proves that network namespace isolation does not block the health path.

The gRPC path is verified. Next, you will learn how to recover from a stale kubelet checkpoint that blocks GPU scheduling.

Step 7 — Recovering from Stale Kubelet State

During testing, a stale kubelet checkpoint can hold a GPU allocation for a deleted pod, causing Insufficient amd.com/gpu scheduling failures. If you see this error after deleting a pod, the kubelet has not yet released the device back to the pool.

To recover, force-delete the stale pod directory under /var/lib/kubelet/pods/<uid>, then restart the device plugin DaemonSet:

kubectl rollout restart daemonset amdgpu-device-plugin-daemonset -n kube-system
Enter fullscreen mode Exit fullscreen mode

If the scheduler is still stuck, restart k3s to clear kubelet state entirely:

systemctl restart k3s
Enter fullscreen mode Exit fullscreen mode

After the restart, the device plugin re-registers and allocatable returns to 1. This is a known operational issue with the device plugin lifecycle on k3s, and it is the first thing to check when GPU scheduling fails for no obvious reason.

Step 8 — Patching the rocm-smi --rasinject CLI

The shipped ROCm 7.14 rocm-smi binary has a bug in its --rasinject argument parser. The argument is defined with nargs=1 and metavar="BLOCK", but the code reads two arguments (block and error type). This means:

rocm-smi --rasinject umc ce
# Error: unrecognized arguments: ce
rocm-smi --rasinject umc
# IndexError: list index out of range
Enter fullscreen mode Exit fullscreen mode

The fix is to patch /opt/rocm/core-7.14/libexec/rocm_smi/rocm_smi.py:

# Before:
"--rasinject",
help="Inject RAS poison for specified block (ONLY WORKS ON UNSECURED BOARDS)",
type=str,
metavar="BLOCK",
nargs=1,

# After:
"--rasinject",
help="Inject RAS poison for specified block and error type (ONLY WORKS ON UNSECURED BOARDS)",
type=str,
metavar=("BLOCK", "ERRTYPE"),
nargs=2,
Enter fullscreen mode Exit fullscreen mode

After the patch, the CLI accepts the correct syntax:

rocm-smi --rasinject umc ce
# This is experimental feature, use 'amdgpuras' tool for ras error manipulations for newer vbios
# WARNING: GPU[%s]  : RAS control is not available
Enter fullscreen mode Exit fullscreen mode

The --help output now correctly shows:

[--rasinject BLOCK ERRTYPE]
Enter fullscreen mode Exit fullscreen mode

Note that the MI300X VF firmware on DevCloud droplets does not expose RAS injection (RAS control is not available), so the CLI accepts the args correctly but the hardware rejects the operation. The patch is still necessary for future hardware or local bare-metal deployments that do expose RAS injection.

Conclusion

In this tutorial, you provisioned an AMD MI300X instance with ROCm 7.14, installed a single-node k3s cluster, imported the ROCm image into k3s containerd, deployed the metrics exporter and AMD device plugin with health-check support, and verified GPU scheduling, compute access, and the gRPC socket end to end. You also learned how to recover from a stale kubelet checkpoint that blocks GPU scheduling and how to patch the rocm-smi --rasinject CLI parser for future RAS error injection testing.

The metrics exporter exposes 192 Prometheus metrics, including gpu_health. The gRPC socket at /var/lib/amd-metrics-exporter/amdgpu_device_metrics_exporter_grpc.socket is reachable from the device plugin container, and the device plugin heartbeat runs every 10 seconds via -pulse=10.

If you want to extend this setup, you can deploy a real ML workload by swapping the rocm:latest image for a framework-specific image such as rocm/tensorflow-installer:latest or rocm/pytorch:latest. You can also experiment with DRA device taints (KEP-5055) on a cluster that has the DRA feature gate enabled, or use the patched rocm-smi --rasinject command on hardware that exposes RAS injection to simulate degraded GPU states.

Top comments (0)