Your GPUs Are Lying to You: The Brutal Economics of AI on Kubernetes

Two numbers, same card, same second:

100% and 0.76%

One of them is what you pay for. The other one is what you should pay for.

The first is GPU utilization, the number on your dashboard. The second is SM Active, how much of the chip was actually working. Both came off an H100 on 9 September 2026, in the same second.

Which one do you think the bill follows? Hold that question. I come back to it at the end.

I rented an H100 to find out if I was right

In July I wrote a talk description that said: the bill lands, triple what you expected, and nobody can explain the gap. In September I rented a card to check whether I could defend that sentence.

Three rentals, about four hours, $18.71 plus German VAT. Cheapest thing in the whole project. The invoice knows exactly how long I had the card. It has no idea what I did with it. Remember that receipt.

Before the rented H100 I tried the smallest possible thing on the gaming GPU in my own PC: one thread, running on one of 36 cores. The gauge said 100% busy. That was the moment I knew I had a talk.

Meet Nadia. She runs the GPUs somebody else pays for, and she did everything right: benchmarked the model, sized the fleet from that number, and her dashboard hasn't dropped below 90% since. She is not careless. She is us.

A GPU is a truck you rent by the hour. "Busy" means the truck is on the road. It never means the truck is full. Benchmarks run full trucks. Production doesn't. So the fleet triples, and the person who signs the checks asks: you told me they're 90% busy, so why did we just buy three times more?

Three lies stand between that dashboard and that question.

Lie #1 — the metric

What does GPU utilization actually measure? NVIDIA wrote the answer down for us:

Percent of time over the past sample period during which one or more kernels was executing on the GPU.

That's the nvidia-smi manual, describing the same NVML field (nvmlUtilization_t.gpu) that DCGM exports as DCGM_FI_DEV_GPU_UTIL. The sample period is somewhere between one second and a sixth of a second depending on the product.

One. Or more.

An H100 has 132 SMs, 80 GB of memory and one very wide bus. One thread busy on one SM gives you 100%. The entire chip saturated gives you 100%. Same number. The metric never asks how much. It's a yes/no question NVIDIA asks a few times a second.

So I lied to it. One thread, spinning, on 1 of 132 SMs — under one percent of the chip:

	GPU_UTIL	SM Active	Tensor	MFU	tok/s
spin loop	100%	0.76%	~0	~0	0

0.76% is exactly 1/132. The dashboard measures on the road. Everybody reads it as full.

Fine, run something serious

Fair objection. A spinning thread is a toy. So: a 7B model on vLLM, one request at a time, on the same rented card. I ask it "Hi".

	GPU_UTIL	SM Active	Tensor	DRAM	MFU	tok/s
batch 1	90%	54.59%	1.62%	62.3%	0.23%	148

The GPU is 90% busy and 0.23% of it is doing work.

Why? To write one word, the chip reads all 15 GB of the model. Every word. Every time. One envelope, full trip to the loading dock.

Two operations per weight across 7.62 billion weights is 15.2 GFLOP per token, roughly 15 microseconds of math. Reading 15.2 GB of weights at the measured 2.2 TB/s takes 6.8 milliseconds. The math units are busy for 0.015 ms out of 6.8. That's your 0.23%. An H100 needs about 295 operations per byte read before it becomes compute-bound, and batch-1 decode uses each byte exactly once.

That's physics, not sloppy code. The fix isn't a bigger truck. It's more envelopes per trip.

Batch 64

	GPU_UTIL	SM Active	Tensor	DRAM	MFU	tok/s
batch 1	90%	54.59%	1.62%	62.3%	0.23%	148
batch 64	85%	62.07%	11.18%	57.4%	11.6%	7,485

148 to 7,485 tokens per second. Fifty times the work.

And the lie-number went down. 89% to 85%.

Same trip, same fuel, sixty-four envelopes. MFU moved from 0.23% to 11.6%, and for inference that's a good number — the literature puts a healthy inference workload at 5–15%. That number describes a full truck. Most dashboards are describing an empty one.

SM Active only crawls from 54.59% to 62.07% while the work goes up 50×. Even the second-honest metric is nearly blind to batching. DCGM's own documentation says why:

Note that "active" does not necessarily mean a warp is actively computing. For instance, warps waiting on memory requests are considered active.

SM Active catches the spin loop. For inference you need Tensor Core activity and MFU.

The fields that don't lie

They ship with DCGM. They're just not on the default dashboard. Think of a restaurant kitchen:

• DCGM_FI_PROF_SM_ACTIVE — are the cooks at their stations?
• DCGM_FI_PROF_SM_OCCUPANCY — how many orders is each cook juggling at once?
• DCGM_FI_PROF_PIPE_TENSOR_ACTIVE — is the special oven running?
• DCGM_FI_PROF_DRAM_ACTIVE — are the cooks running back and forth to the fridge?
• MFU — meals served, divided by the meals this kitchen could theoretically serve in an hour.

Wire them up next to the lie:

# dcgm-counters.csv
# --- The lie ---
DCGM_FI_DEV_GPU_UTIL,      gauge, GPU utilization (in %) — fraction of time at least one kernel was running.
# --- The truth (Datacenter Profiling fields; need --cap-add SYS_ADMIN and a datacenter GPU) ---
DCGM_FI_PROF_SM_ACTIVE,          gauge, Ratio of cycles an SM has at least 1 warp assigned.
DCGM_FI_PROF_SM_OCCUPANCY,       gauge, Ratio of number of warps resident on an SM.
DCGM_FI_PROF_PIPE_TENSOR_ACTIVE, gauge, Ratio of cycles the tensor (HMMA) pipe is active.
DCGM_FI_PROF_DRAM_ACTIVE,        gauge, Ratio of cycles the device memory interface is active.

MFU is the one that translates to money: the work actually served, divided by the work the GPU could theoretically serve in the same hour.

The rule. When GPU utilization is above 80%:

• SM Active under 10% → nothing is running.
• SM Active high but MFU under 5% → something is running, but it isn't computing.

In business terms: you are bleeding money.

Lie #2 — the scheduler

NVIDIA isn't alone here. Kubernetes wrote its answer down too, on the "Schedule GPUs" page:

Containers (and Pods) do not share GPUs. There's no overcommitting of GPUs. Each container can request one or more GPUs. It is not possible to request a fraction of a GPU.

Those two sharing sentences were later dropped from the page. The behaviour wasn't.

resources:
  limits:
    nvidia.com/gpu: 1     # integer. request == limit. no fractions. no overcommit.

An integer. A pod that needs 1% of a card takes the card.

Watch it happen. pod-a runs one spinning thread and gets the GPU. pod-b is byte-for-byte identical and asks for one:

pod-a   Running
pod-b   Pending    0/1 nodes are available: Insufficient nvidia.com/gpu

The node says nvidia.com/gpu 1/1 allocated. The card says 1% busy.

99% empty. 100% unavailable. One envelope booked the whole truck.

Kubernetes didn't waste the card. It fenced it off. And the fix is one config flag:

# device-plugin time-slicing config
version: v1
flags:
  migStrategy: none
sharing:
  timeSlicing:
    renameByDefault: false
    failRequestsGreaterThanOne: false
    resources:
      - name: nvidia.com/gpu
        replicas: 4

The device plugin README describes exactly what that does:

The NVIDIA device plugin allows oversubscription of GPUs through a set of extended options in its configuration file. […] If this configuration were applied to a node with 8 GPUs on it, the plugin would now advertise 80 nvidia.com/gpu resources to Kubernetes instead of 8.

The node now advertises nvidia.com/gpu: 4. pod-b goes Pending → Running in about ten seconds. Same card, same pod, nothing else changed.

Three honest ways to share a card

Time-slicing. They take turns. No isolation at all — one workload can stall the others and nothing stops it. Fine for dev and bursty work.

MPS. They ride together in one context. No walls, so one bad tenant hurts everyone. Don't oversell this one.

MIG. Real walls, in hardware. Rigid: you drain the node to re-partition, and it's A100/H100-class only. On my rented card it split into seven slices, and the node advertised them:

nvidia.com/mig-1g.10gb: 7

All three are opt-in. The default is one pod, one card.

Where this is going: DRA went GA in Kubernetes 1.34, partitionable devices and consumable capacity are still beta, and NVIDIA donated its DRA driver to CNCF at KubeCon EU 2026. Direction of travel, not this quarter.

In business terms: by default, you are buying it twice. Kubernetes counts trucks. It never looks inside them.

Lie #3 — the bill

This one tells the truth. That's the problem.

Remember the receipt? My longest test: 2 hours 30 minutes, one H100, $11.01. That's $4.41 an hour.

Cloud billing is duration times dollars. There is no column for how much of it you used.

Take the batch-1 hour. Dashboard: 90% GPU utilization. Honest metric, same hour: 0.23% MFU.

$4.41 per hour
= $0.01 that did work
+ $4.40 that bought idle silicon     (99.8%)

One H100 running around the clock is roughly $3,200 a month. At batch 1, that's $7 of real work and $3,193 of idle silicon. That is what the extra GPUs would buy you. Now multiply by however many cards you have.

Here is the honest full-hour version, from the actual billed window I captured (15:45–16:45 UTC, half batch 1, half batch 64):

"the_lie":   { "gpu_util_pct_avg": 86.2 },
"the_truth": { "sm_active_pct_avg": 57.9, "tensor_active_pct_avg": 6.8, "mfu_pct": 5.89 },
"the_bill":  { "cost_usd": 4.42, "cost_that_did_work_usd": 0.26,
               "cost_of_idle_silicon_usd": 4.16, "idle_pct": 94.1,
               "tokens_generated": 13791321, "usd_per_1m_tokens": 0.32 }

86% on the dashboard. 5.89% MFU. $0.26 of that hour did work and $4.16 bought idle silicon. I ran that measurement twice, on two separate droplets, and got 5.9% both times.

The reasonable objection: you just don't know how to configure them, or our workload is different, we know better.

Maybe. So consider Cast AI's report covering January 2025 to April 2026, across about 23,000 Kubernetes clusters. Average GPU utilization: 5%.

And that's the dashboard number. The honest ones are lower.

So, which one does the bill follow?

Neither.

It bills hours. It does not bill work. The bill is honest and useless: it tells you what you spent, and never what you got.

In business terms: you cannot prove any of it was worth it.

Why does this exist in 2026?

The metric isn't broken. It's answering a 2007 question.

Back then a GPU ran one thing at a time, and "is it busy?" was the whole answer. Then the chip grew to 132 cores and the question never got updated.

Kubernetes counts whole cards for a related reason: a GPU never grew an operating system. CPUs got sixty years of schedulers, preemption and isolation. GPUs got a driver. Time-sharing shipped in 1961; hardware partitioning on a GPU shipped in 2020. Time-slicing, MPS and MIG are that missing OS, arriving late.

While the CPU spent fifty years evolving, the GPU was, quite literally, playing games.

What to do on Monday

Graph MFU next to GPU utilization. Same dashboard, same panel if you can. The moment those two lines separate, you have a number you can act on.

Scale on queue depth, never on utilization. Utilization reads 90% whether the truck is full or empty, so autoscaling on it either scales forever or never. KEDA on vllm:num_requests_waiting adds a truck only when envelopes are actually waiting, which means the trucks you already pay for fill up first. If you can't autoscale, cap and batch instead. With continuous batching a queue is the truck filling up, not a failure. The latency cost is small per token — 6.8 ms to 8.6 ms going from batch 1 to batch 64.

Stop using the scheduler defaults. Pick time-slicing, MPS or MIG based on the isolation you actually need.

Price inference per million tokens, not per GPU-hour, and put that number in front of whoever signs the check. Mine, for the same silicon in the same hour:

	$ per million tokens
batch 1	$8.28
batch 64	$0.16

Fifty times the price. Same silicon. Gauge unchanged.

Or, as it goes in German: Zeit ist Geld, aber das Finanzamt will trotzdem beides.

The last thing

The dashboard never lied to hurt you. Neither did the benchmark. Both answered a question honestly — is the GPU busy? and what can this chip do when it's full? Neither of them was the question I was paying for: is the GPU doing my work?

Utilization is not work.

GPUs are evolving toward what we're now asking of them, but they've only just started. The silicon isn't the baby. The software around it is. And these cards are very power hungry, so you are burning more than money.

Get it right and you get a 98% discount. Get it wrong and you pay 50× more, and destroy a bit of the environment as a bonus.

Measure the question you're paying for.