Getting Started with HAMi: GPU Virtualization on Kubernetes

Running multiple AI workloads on the same GPU without MIG? I needed this for a mixed fleet of A100s and AMD MI210s where MIG wasn't an option. That's when I discovered HAMi (Heterogeneous AI Computing Virtualization), now a CNCF Sandbox project.

The Problem I Had

My inference cluster had 8 A100s. Each model needed ~4-8GB. Running whole-GPU allocation meant 87% waste. MIG wasn't available on my older A100 revisions, and I had AMD cards coming online that needed the same sharing strategy.

What HAMi Actually Does

HAMi sits between your pods and the GPU driver. It intercepts CUDA calls via libvgpu.so and enforces per-pod memory and compute limits that Kubernetes can't express natively.

No hardware partitioning. No time-slicing risk. Just software-enforced isolation you configure per pod.

Installation

# Add the repo
helm repo add hami-charts https://project-hami.github.io/HAMi/

# Install with GPU support
helm install hami hami-charts/hami \
  --namespace hami-system \
  --create-namespace \
  --set devicePlugin.enabled=true \
  --set scheduler.enabled=true

First Workload

apiVersion: v1
kind: Pod
metadata:
  name: inference
spec:
  containers:
  - name: model
    image: my-inference:latest
    resources:
      limits:
        nvidia.com/gpu: 1
        nvidia.com/gpumem: 4096    # 4GB
        nvidia.com/gpucores: 25    # 25% compute

HAMi schedules this on a node with available resources and enforces the limits at runtime.

Verifying It's Working

# Check HAMi pods
kubectl get pods -n hami-system

# Watch resource allocation
kubectl describe node <node-name> | grep hami

You'll see hami.io/vgpu-memory and hami.io/vgpu-cores alongside the standard NVIDIA resources.

Gotchas I Hit

1. Scheduler extender required: HAMi won't work with default scheduler. The extender handles the custom resource math.

2. Library injection: Your containers need LD_PRELOAD=/usr/local/vgpu/libvgpu.so. The device plugin handles this automatically.

3. Memory overhead: HAMi itself uses ~200MB per node. Plan accordingly.

Multi-Vendor Reality Check

I tested the same pod spec on AMD MI210s. Changed nothing except the driver. HAMi handled the ROCm translation layer transparently. That's the real value — one scheduler, multiple GPU architectures.

Production Metrics

After 3 months running 200+ inference pods across 12 GPUs:

• GPU utilization: 23% → 78%
• Cost per inference: down 67%
• No OOM kills (HAMi memory limits actually work)

When NOT to Use HAMi

If you need hardware isolation for compliance, use MIG. If you're in a single-tenant dev environment, time-slicing is simpler. HAMi sits in the middle — multi-tenant software isolation without MIG's rigidity.

Next Steps

The HAMi GitHub has examples for training workloads and batch jobs. The scheduler extender is pluggable if you need custom placement logic.

For Kubernetes GPU sharing without vendor lock-in, this is currently the only CNCF option.

GitHub: keda-gpu-scaler | gpu-mcp-server

---

Pavan Madduri is a Senior Cloud Platform Engineer at W.W. Grainger, a CNCF Golden Kubestronaut, and CNCF TAG Workloads Foundation Tech Lead. He maintains keda-gpu-scaler and gpu-mcp-server and contributes to CNCF projects including KEDA, Volcano, and Dragonfly. Find him on GitHub.