DEV Community: Richard Okonicha

Deterministic Agentic Governance: Bringing Order to Autonomous Software Engineering

Richard Okonicha — Mon, 17 Aug 2026 09:14:04 +0000

Deterministic Agentic Governance: Bringing Order to Autonomous Software Engineering

How we built a framework that makes AI agents reliable enough for production

The Problem: Agents Don't Play Nice at Scale

You've seen the demo. An AI agent writes a feature, runs tests, commits to git. Magic.

Then you try it on a real codebase. Ten tasks in, the agent hallucinates an API. Fifteen tasks in, it forgets the architectural constraints you gave it in the first prompt. Twenty tasks in, you're manually reverting broken commits and wondering why you didn't just write the code yourself.

The math is brutal. If each step has a 95% success rate (optimistic), a 20-step task chain succeeds only 36% of the time:

P(success) = 0.95^n

n=5   → 77%
n=10  → 60%
n=20  → 36%
n=50  → 8%

This is compositional decay—the exponential failure rate of chaining probabilistic operations.

Why Current Approaches Fail

1. Context Rot

Every conversation turn adds to the LLM's context window. After 20 turns:

Initial instructions get diluted (attention thinning)
Error logs pollute the context (cascade pollution)
The agent "forgets" constraints it was given initially

2. No Objective Verification

Asking an LLM to review its own code is like asking a student to grade their own exam. Hallucinated "looks good to me" responses are the norm.

3. No Safety Boundaries

Agents with shell access can rm -rf /, commit secrets, or open PRs to production branches.

4. Human-in-the-Loop Doesn't Scale

Manual approval for every step kills velocity. But no approval means broken code in main.

The Insight: Treat the LLM as a CPU, Not a Brain

What if we stop treating the LLM as an autonomous decision-maker and start treating it as a stateless processing unit—like a CPU?

The CPU doesn't manage memory, enforce permissions, or decide which program runs next. The operating system kernel does that.

So we built a governance kernel around the LLM.

The Three Pillars of Deterministic Governance

Pillar 1: Persistent State Ledger (PROGRESS.json)

Instead of stuffing task state into the LLM's context, we externalize it:

{
  "version": "1.0.0",
  "projectName": "electron-to-tauri-migration",
  "tasks": [
    {
      "id": "TASK-003",
      "category": "DATA_STORAGE",
      "referencePath": "electron/store/index.ts",
      "targetPath": "src-tauri/src/storage/mod.rs",
      "status": "PENDING",
      "retryCount": 0,
      "maxRetries": 5,
      "verificationLogs": []
    }
  ]
}

Every agent turn gets a fresh, clean context containing only:

The single task specification
The reference code (read-only)
Acceptance criteria

No history. No error logs from previous tasks. No instruction dilution.

Pillar 2: Declarative Permission Sandboxing

Before any code runs, the governance kernel enforces boundaries via policy-as-code:

{
  "readOnlyPaths": ["src/**", "package.json", "Cargo.toml"],
  "writeOnlyPaths": ["src/**", "tests/**", "dist/**"],
  "forbiddenPaths": [".git/**", ".env*", "*.key", "secrets/**"],
  "allowedCommands": ["npm", "cargo", "tsc", "eslint", "git"],
  "forbiddenCommands": ["rm -rf", "sudo", "curl", "wget", "ssh"],
  "networkAccess": false,
  "maxExecutionTimeMs": 300000,
  "maxMemoryMb": 1024
}

This isn't a suggestion. It's enforced at the OS level via environment variables and command validation.

Pillar 3: Compiler Verification Gates

No LLM self-evaluation. Only binary, deterministic gates:

Gate	Tools	Purpose
`TYPE_CHECK`	`tsc --noEmit`, `cargo check`	Type safety & compilation
`SYNTAX_CHECK`	`prettier --check`, `cargo fmt --check`	Formatting & syntax
`AST_LINT`	`eslint`, `cargo clippy`	Static analysis
`ANTI_STUB`	Regex/AST patterns	Detect `TODO`, `unimplemented!()`, empty blocks
`TEST_SUITE`	`npm test`, `cargo test`	Functional correctness

If any gate fails, the task is retried with the compiler error fed back to a fresh agent context. After maxRetries, the workspace is git-reset to the last clean commit.

The Governance Loop

while pending_tasks_exist():
    task = ledger.get_next_pending()

    base_commit = git.current_commit()

    for attempt in range(task.maxRetries + 1):
        if attempt > 0:
            git.reset_hard(base_commit)  # Clean slate
            ledger.increment_retry(task.id)

        # Fresh context, single task
        context = build_clean_context(task, reference_code)
        result = agent.execute(context)

        if not result.success:
            continue

        # Objective verification
        verification = run_all_gates(task.category)

        if verification.all_passed():
            git.commit(task.id, task.description)
            ledger.mark_completed(task.id)
            break
        else:
            ledger.log_verification_failures(task.id, verification)

    if not verification.all_passed():
        ledger.mark_failed(task.id)
        git.reset_hard(base_commit)

Result: Zero context rot. Deterministic verification. Atomic commits. Full audit trail.

Real-World Example: Electron → Tauri 2.0 Migration

We migrated a production desktop app (100k+ lines) from Electron/Node to Tauri 2.0 (Rust + React + SQLite).

10 coordinated tasks:

Task	Category	Verification Gates
Tauri main.rs entry point	INFRASTRUCTURE	TYPE, SYNTAX, LINT, ANTI_STUB
IPC layer replacement	INFRASTRUCTURE	TYPE, SYNTAX, LINT, ANTI_STUB
Hybrid SQLite + Markdown storage	DATA_STORAGE	TYPE, SYNTAX, LINT, ANTI_STUB, TEST
Non-blocking token streaming (10Hz)	LLM_INFERENCE	TYPE, SYNTAX, LINT, ANTI_STUB
React virtualization (@tanstack/react-virtual)	UI_COMPONENT	TYPE, SYNTAX, LINT, ANTI_STUB
Integration tests	TESTING	TYPE, SYNTAX, LINT, TEST
Build config & documentation	DOCUMENTATION	SYNTAX

Results:

3 tasks failed TYPE_CHECK on first attempt → auto-retried with compiler errors → passed
1 task failed ANTI_STUB (empty function body) → auto-retried → passed
All 10 committed atomically with structured messages
Full audit trail in PROGRESS.json + git history

Enterprise Governance: Three-Tier Authorization

Not all actions are equal. The framework enforces a tiered model:

Tier	Actions	Approval
Autonomous	Read code, create branches, run type checks, format files	Pre-approved
Semi-Autonomous	Open PRs, modify schemas, update dependencies	Human sign-off
Forbidden	Commit to protected branches, export secrets, unsandboxed network	Hard-blocked

Compliance alignment:

ISO/IEC 42001 (AI management systems)
NIST AI RMF (Risk management)
IEEE 7000 (Ethical system design)
SFIA 9 (Competency framework)

Supported Agents

The framework is agent-agnostic. Built-in adapters:

// Claude Code (Anthropic)
{ type: 'claude-code', command: 'claude', args: ['-p', '--bare', '--dangerously-skip-permissions'] }

// Aider
{ type: 'aider', command: 'aider', args: ['--no-git', '--yes', '--message'] }

// OpenCode
{ type: 'opencode', command: 'opencode', args: ['run', '--headless'] }

// Kilo Code
{ type: 'kilo', command: 'kilo', args: ['run', '--headless'] }

// Custom - implement AgentAdapter interface

Getting Started

# Install
npm install deterministic-agentic-governance

# Initialize in your workspace
npx govern init --workspace ./my-project --project my-migration

# Define tasks (JSON)
cat > tasks.json << 'EOF'
[{
  "id": "TASK-001",
  "category": "UI_COMPONENT",
  "referencePath": "src/components/OldButton.tsx",
  "targetPath": "src/components/NewButton.tsx",
  "description": "Migrate button to new design system",
  "acceptanceCriteria": ["TypeScript compiles", "No stubs", "Passes ESLint"],
  "dependencies": [],
  "maxRetries": 3,
  "timeoutMs": 180000,
  "metadata": {}
}]
EOF

# Add and run
npx govern add --file tasks.json
npx govern run
npx govern status

When to Use This (And When Not To)

✅ Ideal For

Large-scale migrations (framework ports, language rewrites)
Automated refactoring (design systems, strict TypeScript, dead code)
Cross-platform code generation (OpenAPI → clients, schema → ORM)
Compliance-required environments (fintech, healthcare, defense)
CI/CD automation (dependency updates, security patches)

❌ Not Ideal For

Quick prototyping → Use Cursor/Claude Code directly
Single-file edits → IDE copilot
Exploratory coding → Unconstrained agent
Learning → Notebook/REPL

The Key Differentiator

Every task either passes all compiler/linter gates and commits atomically, or rolls back cleanly with full diagnostic capture. No context rot. No hallucinated verifications. No partial broken states.

Architecture Diagram

┌─────────────────────────────────────────────────────────────────┐
│                     GOVERNANCE ORCHESTRATOR                      │
│  ┌─────────────┐  ┌─────────────┐  ┌─────────────┐              │
│  │   LEDGER    │  │  SANDBOX    │  │ VERIFICATION│              │
│  │  MANAGER    │  │  ENFORCER   │  │ ORCHESTRATOR│              │
│  └─────────────┘  └─────────────┘  └─────────────┘              │
│         │               │                │                       │
│         ▼               ▼                ▼                       │
│  ┌─────────────────────────────────────────────────────────┐    │
│  │                    AGENT ADAPTERS                        │    │
│  │  ┌──────────┐ ┌────────┐ ┌──────────┐ ┌──────────────┐  │    │
│  │  │Claude Code│ │ Aider  │ │ OpenCode │ │   Kilo Code  │  │    │
│  │  └──────────┘ └────────┘ └──────────┘ └──────────────┘  │    │
│  └─────────────────────────────────────────────────────────┘    │
│         │                                                       │
│         ▼                                                       │
│  ┌─────────────────────────────────────────────────────────┐    │
│  │                     GIT MANAGER                          │    │
│  │  Atomic commits • Auto-rollback • Branch management      │    │
│  └─────────────────────────────────────────────────────────┘    │
└─────────────────────────────────────────────────────────────────┘

Resources

npm: npm install deterministic-agentic-governance
GitHub: https://github.com/richardokonicha/deterministic-agentic-governance
Documentation: See README.md for full API reference
Example: examples/electron-to-tauri/ — complete 10-task migration

Conclusion

The industry is shifting from writing code to governing agents. But governance without determinism is just chaos with better marketing.

This framework proves that you can have autonomous execution with deterministic guarantees. The LLM does what it's good at (code generation from context). The governance kernel does what it's good at (state, permissions, verification, rollback).

The result: AI agents you can actually trust in production.

Published as part of the Deterministic Agentic Governance Framework v1.0.0. MIT License.

How To Set Up Kubernetes on AMD GPUs with ROCm

Richard Okonicha — Sat, 08 Aug 2026 02:59:11 +0000

How To Set Up Kubernetes on AMD GPUs with ROCm

Introduction

Running Kubernetes on AMD GPU hardware is possible, but the path is not as well documented as the NVIDIA equivalent. The ROCm stack, the device plugin, the metrics exporter, and the container runtime each have their own quirks — and on AMD DevCloud, Docker and k3s use separate containerd instances, which breaks the normal image-pull flow.

This tutorial walks you through provisioning an AMD MI300X instance with ROCm preinstalled, installing a single-node k3s cluster, importing the ROCm image into k3s containerd, deploying the AMD device plugin and metrics exporter, and verifying that Kubernetes can schedule pods onto the GPU. You will also see how to validate the gRPC health path between the device plugin and the metrics exporter, patch a known rocm-smi --rasinject CLI bug, and recover from a stale kubelet checkpoint that blocks GPU scheduling.

When you are finished, you will have a working ROCm-on-Kubernetes cluster that you can use as a test substrate for GPU workloads.

Prerequisites

To complete this tutorial, you will need:

Access to an AMD DevCloud account with the ability to create GPU instances. You can request access through the AMD Developer website.
A local macOS or Linux machine with kubectl and docker installed and SSH access configured. You can install kubectl by following the Kubernetes documentation and Docker by following the Docker documentation.
Familiarity with Kubernetes concepts such as DaemonSets, device plugins, and feature gates. If you are new to Kubernetes, the Kubernetes Basics guide provides a helpful starting point.

Step 1 — Provisioning the AMD MI300X Infrastructure

Start by creating a droplet on AMD DevCloud that provides a single MI300X GPU with ROCm preinstalled. The gpu-mi300x1-192gb-devcloud plan gives you 192GB of host memory and the ROCm software stack under /opt/rocm.

If you create the droplet through the DevCloud web UI, select the rocm-7-14 image and attach your SSH key. The provisioning process takes roughly 90 seconds. Once the droplet is ready, SSH into it and verify the baseline ROCm installation:

rocm-smi --showid --showproduct

The rocm-smi tool reports the GPU model, GFX architecture, and ROCm version. On a fresh MI300X droplet you should see output like this:

Output
GPU[0]  : Device Name: AMD Instinct MI300X VF
GPU[0]  : GFX Version: gfx942
ROCm: /opt/rocm -> /opt/rocm-7.0.2
rocm-smi: ROCM-SMI version 4.0.0+2b22ab01

If these values match your droplet, the hardware and driver stack are ready.

Next, confirm that RAS (Reliability, Availability, and Serviceability) blocks are enabled with zero errors at baseline:

rocm-smi --showrasinfo

You should see output like this:

Output
GPU[0]  : RAS UMC: Enabled, 0 correctable, 0 uncorrectable
GPU[0]  : RAS SDMA: Enabled, 0 correctable, 0 uncorrectable
GPU[0]  : RAS GFX: Enabled, 0 correctable, 0 uncorrectable

UMC, SDMA, and GFX must show Enabled with 0 correctable and uncorrectable errors. A clean baseline matters because these blocks are what you will use to measure hardware health during testing.

You have now verified the GPU hardware and baseline ROCm installation. Next, you will install Kubernetes and configure it for GPU workloads.

Step 2 — Installing and Configuring k3s

DevCloud droplets ship with both Docker and k3s installed, but they use separate containerd instances. Because k3s cannot see images pulled by Docker, you must transfer the ROCm image from Docker into k3s containerd after disabling Docker. For this reason, stop and disable Docker before installing Kubernetes:

Execute the following commands to stop Docker and prevent it from restarting on boot. The systemctl disable command ensures Docker does not interfere with k3s containerd:

systemctl stop docker
systemctl disable docker

Install k3s, a lightweight Kubernetes distribution that is well-suited for single-node clusters:

The curl -sfL flags follow redirects silently, fail on HTTP errors, and show errors. The INSTALL_K3S_SKIP_START=true environment variable tells the installer to skip starting k3s so you can configure it first:

curl -sfL https://get.k3s.io | INSTALL_K3S_SKIP_START=true sh -

After the installation script completes, import the ROCm Docker image into k3s containerd so that the k3s runtime can use it:

docker save rocm:latest -o /tmp/rocm-latest.tar
ctr -n k8s.io --address /run/k3s/containerd/containerd.sock images import /tmp/rocm-latest.tar

The first command saves the image from the Docker runtime, and the second command imports it into the k8s.io namespace of k3s containerd. Without this step, pods that request rocm:latest fail with ImagePullBackOff because k3s cannot access Docker's image store. This is not a theoretical problem — it is the first thing that breaks on a fresh DevCloud droplet.

Next, configure /etc/rancher/k3s/config.yaml to enable the feature gates and privileges that the AMD device plugin requires:

kubelet-arg:
  - "feature-gates=ResourceHealthStatus=true"
allow-privileged: true

ResourceHealthStatus=true enables kubelet to surface per-device health in pod status. allow-privileged=true is required because the AMD device plugin needs access to /dev/kfd, /dev/dri, and /sys, along with the SYS_ADMIN capability for the health-check variant.

Start the k3s services and wait for the node to reach the Ready state:

systemctl start k3s
kubectl get nodes

You should see a single control-plane node in the Ready state running k3s v1.36.3 or later. Your node name, age, and exact IP will differ:

Output
NAME                                                 STATUS   ROLES           AGE   VERSION
rocm-7-14-software-gpu-mi300x1-192gb-devcloud-atl1   Ready    control-plane   6h49m   v1.36.3+k3s1

With the cluster running, you will now deploy the metrics exporter that exposes health data to Prometheus and the device plugin.

Step 3 — Configuring the Metrics Exporter

The AMD Device Metrics Exporter (rocm/device-metrics-exporter:v1.5.0) exposes Prometheus metrics and a gRPC health service on each node. Open a file named metrics-exporter.yaml in your text editor and add the following DaemonSet and ConfigMap:

apiVersion: v1
kind: ConfigMap
metadata:
  name: metrics-exporter-config
  namespace: kube-system
data:
  exporter-config.yaml: |
    CommonConfig:
      HealthService:
        Enable: true
        PollingRate: "10s"
      Debug:
        EnableAPI: true
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: amd-metrics-exporter
  namespace: kube-system
spec:
  selector:
    matchLabels:
      name: amd-metrics-exporter
  template:
    metadata:
      labels:
        name: amd-metrics-exporter
    spec:
      containers:
      - name: exporter
        image: rocm/device-metrics-exporter:v1.5.0
        command: ["./device-metrics-exporter"]
        args: ["--config", "/etc/metrics/exporter-config.yaml"]
        env:
        - name: AMD_ENABLE_HEALTH_GRPC
          value: "true"
        - name: AMD_DEBUG_ENABLE_API
          value: "true"
        securityContext:
          privileged: true
        volumeMounts:
        - name: socket-dir
          mountPath: /var/lib/amd-metrics-exporter
        - name: config
          mountPath: /etc/metrics
          readOnly: true
      volumes:
      - name: socket-dir
        hostPath:
          path: /var/lib/amd-metrics-exporter
          type: DirectoryOrCreate
      - name: config
        configMap:
          name: metrics-exporter-config

The ConfigMap enables the gpu_health metric at a 10-second polling interval and turns on the debug gRPC API. The DaemonSet runs the exporter in privileged mode, mounts the host directory where the gRPC socket will be created, and passes the config via mounted volume. We use rocm/device-metrics-exporter:v1.5.0 because the latest tag on Docker Hub does not exist for this image.

Apply the manifest:

kubectl apply -f metrics-exporter.yaml

Verify that the exporter is reachable on the node:

curl http://localhost:5000/metrics | grep gpu_health

You should see a metric line similar to the following:

Output
gpu_health{card_model="",container="",gpu_compute_partition_type="spx",gpu_id="0",gpu_memory_partition_type="nps1",gpu_partition_id="NA",hostname="your_node_name",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number=""} 1

A value of 1 indicates Healthy. The exporter also exposes a gRPC socket at /var/lib/amd-metrics-exporter/amdgpu_device_metrics_exporter_grpc.socket, which the device plugin uses to read health updates.

The metrics exporter is now exposing health data. Next, you will deploy the AMD device plugin to advertise the GPU to Kubernetes.

Step 4 — Deploying the AMD Device Plugin

The ROCm Kubernetes device plugin repository provides a health-check DaemonSet that exposes per-GPU health through the ListAndWatch stream. Apply the manifest directly from the upstream repository:

kubectl apply -f https://raw.githubusercontent.com/ROCm/k8s-device-plugin/master/k8s-ds-amdgpu-dp-health.yaml

This manifest is the experimental health-check variant. It mounts /var/lib/amd-metrics-exporter/ into the device plugin container so the plugin can read the gRPC socket exposed by the metrics exporter. It enables -pulse=10, which sends a heartbeat every 10 seconds. It requests SYS_ADMIN capability and mounts /dev/kfd, /dev/dri, and /sys from the host.

Verify that the DaemonSet pod is running:

kubectl get pods -n kube-system -l name=amdgpu-dp-ds

You should see one pod per node in the Running state with no restarts. If the pod restarts, inspect the logs:

kubectl logs -n kube-system -l name=amdgpu-dp-ds

Look for lines such as Found 1 AMDGPUs and Watching GPU with bus ID: 0000:83:00.0. These confirm that the plugin discovered the GPU and is actively watching it.

The device plugin is now running and advertising the GPU. Next, you will verify that Kubernetes can schedule pods onto the GPU and that containers inside those pods can access the hardware.

Step 5 — Verifying GPU Scheduling and Compute Access

To confirm that the device plugin is advertising the GPU correctly, check the node allocatable resources:

kubectl get node -o jsonpath='{.items[0].status.allocatable}'

You should see amd.com/gpu: "1" alongside CPU, memory, and pod capacity. The full allocatable on a fresh MI300X droplet looks like this:

{
    "amd.com/gpu": "1",
    "cpu": "20",
    "ephemeral-storage": "710372463469",
    "hugepages-1Gi": "0",
    "hugepages-2Mi": "0",
    "memory": "247409204Ki",
    "pods": "110"
}

If the GPU is missing from allocatable, the device plugin has not yet registered; wait a few moments and inspect the DaemonSet logs.

Next, open a file named rocm-health-test.yaml in your text editor and add the following pod manifest:

apiVersion: v1
kind: Pod
metadata:
  name: rocm-health-test
spec:
  containers:
  - name: rocm
    image: rocm:latest
    command: ['sleep', '600']
    resources:
      limits:
        amd.com/gpu: 1
    imagePullPolicy: Never
  restartPolicy: Never

Apply the manifest and watch the pod schedule:

kubectl apply -f rocm-health-test.yaml
kubectl get pod rocm-health-test -w

When the pod reaches the Running state, execute a command inside it to verify GPU access:

kubectl exec -it rocm-health-test -- rocm-smi --showid --showproduct

The output should show the same GPU model and GFX version that you observed on the host.

Output
GPU[0]: Device Name: AMD Instinct MI300X VF
GPU[0]: GFX Version: gfx942

If rocm-smi reports No AMD GPUs were detected, the container is missing access to /dev/kfd or /dev/dri.

GPU scheduling and compute access are working correctly. Next, you will validate the gRPC health path to confirm that the device plugin and metrics exporter can communicate across namespace boundaries.

Step 6 — Validating the gRPC Health Path

To prove that the device plugin can reach the metrics exporter across namespace boundaries, verify the gRPC socket permissions and test a cross-namespace connection.

Inspect the socket on the host:

ls -l /var/lib/amd-metrics-exporter/amdgpu_device_metrics_exporter_grpc.socket

You should see permissions 0777. On our cluster, the socket inode is 253,1:

Output
srwxrwxrwx 1 root root 0 Aug  7 15:24 /var/lib/amd-metrics-exporter/amdgpu_device_metrics_exporter_grpc.socket

The inode must match the bind-mounted path inside the device plugin container. Replace <device-plugin-pod> with the actual pod name from Step 4:

kubectl exec -it -n kube-system <device-plugin-pod> -- ls -li /var/lib/amd-metrics-exporter/

You should see output like this:

Output
total 0
253,1 7120801 srwxrwxrwx 1 root root 0 Aug  7 15:24 amdgpu_device_metrics_exporter_grpc.socket

Compare the inode numbers shown on the host and inside the container. Identical inodes confirm that the mount is the same filesystem object.

If you want to test a live gRPC connection, you can build a small Go program that connects to the socket and calls the GetGPUState or List RPC. Copy the binary into the device plugin container with kubectl cp and run it. A successful connection proves that network namespace isolation does not block the health path.

The gRPC path is verified. Next, you will learn how to recover from a stale kubelet checkpoint that blocks GPU scheduling.

Step 7 — Recovering from Stale Kubelet State

During testing, a stale kubelet checkpoint can hold a GPU allocation for a deleted pod, causing Insufficient amd.com/gpu scheduling failures. If you see this error after deleting a pod, the kubelet has not yet released the device back to the pool.

To recover, force-delete the stale pod directory under /var/lib/kubelet/pods/<uid>, then restart the device plugin DaemonSet:

kubectl rollout restart daemonset amdgpu-device-plugin-daemonset -n kube-system

If the scheduler is still stuck, restart k3s to clear kubelet state entirely:

systemctl restart k3s

After the restart, the device plugin re-registers and allocatable returns to 1. This is a known operational issue with the device plugin lifecycle on k3s, and it is the first thing to check when GPU scheduling fails for no obvious reason.

Step 8 — Patching the rocm-smi --rasinject CLI

The shipped ROCm 7.14 rocm-smi binary has a bug in its --rasinject argument parser. The argument is defined with nargs=1 and metavar="BLOCK", but the code reads two arguments (block and error type). This means:

rocm-smi --rasinject umc ce
# Error: unrecognized arguments: ce
rocm-smi --rasinject umc
# IndexError: list index out of range

The fix is to patch /opt/rocm/core-7.14/libexec/rocm_smi/rocm_smi.py:

# Before:
"--rasinject",
help="Inject RAS poison for specified block (ONLY WORKS ON UNSECURED BOARDS)",
type=str,
metavar="BLOCK",
nargs=1,

# After:
"--rasinject",
help="Inject RAS poison for specified block and error type (ONLY WORKS ON UNSECURED BOARDS)",
type=str,
metavar=("BLOCK", "ERRTYPE"),
nargs=2,

After the patch, the CLI accepts the correct syntax:

rocm-smi --rasinject umc ce
# This is experimental feature, use 'amdgpuras' tool for ras error manipulations for newer vbios
# WARNING: GPU[%s]  : RAS control is not available

The --help output now correctly shows:

[--rasinject BLOCK ERRTYPE]

Note that the MI300X VF firmware on DevCloud droplets does not expose RAS injection (RAS control is not available), so the CLI accepts the args correctly but the hardware rejects the operation. The patch is still necessary for future hardware or local bare-metal deployments that do expose RAS injection.

Conclusion

In this tutorial, you provisioned an AMD MI300X instance with ROCm 7.14, installed a single-node k3s cluster, imported the ROCm image into k3s containerd, deployed the metrics exporter and AMD device plugin with health-check support, and verified GPU scheduling, compute access, and the gRPC socket end to end. You also learned how to recover from a stale kubelet checkpoint that blocks GPU scheduling and how to patch the rocm-smi --rasinject CLI parser for future RAS error injection testing.

The metrics exporter exposes 192 Prometheus metrics, including gpu_health. The gRPC socket at /var/lib/amd-metrics-exporter/amdgpu_device_metrics_exporter_grpc.socket is reachable from the device plugin container, and the device plugin heartbeat runs every 10 seconds via -pulse=10.

If you want to extend this setup, you can deploy a real ML workload by swapping the rocm:latest image for a framework-specific image such as rocm/tensorflow-installer:latest or rocm/pytorch:latest. You can also experiment with DRA device taints (KEP-5055) on a cluster that has the DRA feature gate enabled, or use the patched rocm-smi --rasinject command on hardware that exposes RAS injection to simulate degraded GPU states.