Why is GPU utilization low during PyTorch training?

Low GPU utilization in PyTorch training usually means the GPU is waiting on the CPU: for the next batch, for a sync like .item(), or for Python to launch the next small kernel. On Nodus, as on any platform that bills by the hour, an NVIDIA H100 SXM listed at $2.60/hr on the Nodus pricing page costs the same whether its kernels run 40% or 95% of the time. Measure first with nvidia-smi dmon and torch.profiler, then fix the data loader, the syncs and the batch size, in that order.

What does GPU utilization in nvidia-smi actually measure?

It measures time, not work. NVIDIA's nvidia-smi documentation defines GPU utilization as the "percent of time over the past sample period during which one or more kernels was executing on the GPU", with a sample period between 1/6 of a second and 1 second.

That definition has two consequences. A low number is a clear signal: the GPU had nothing to run for much of each second. A high number is weaker evidence: one tiny kernel keeping the GPU busy counts the same as a full matrix multiply. For how much of the GPU's arithmetic you actually use, the measure is model FLOPs utilization, which we cover in how long it takes to train an AI model.

Watch it while training runs, one line per second, with a timestamp:

nvidia-smi dmon -s ut -o T

-s u prints SM and memory utilization and -s t adds PCIe receive and transmit throughput, per the same NVIDIA page. A sawtooth that drops to near zero between steps is the classic starved-loader pattern. A steady 30% to 50% points to syncs or small kernels instead.

How do you tell whether the data loader is the bottleneck?

Time how long the training loop waits for each batch. This is the first check we run on any slow job, because it splits the problem in two with about ten lines of code:

import time, torch

data_wait, compute = 0.0, 0.0
t0 = time.perf_counter()
for i, (x, y) in enumerate(loader):
    t1 = time.perf_counter()
    data_wait += t1 - t0
    x, y = x.cuda(non_blocking=True), y.cuda(non_blocking=True)
    loss = loss_fn(model(x), y)
    loss.backward()
    opt.step()
    opt.zero_grad(set_to_none=True)
    torch.cuda.synchronize()  # diagnosis only: makes the step time honest
    t0 = time.perf_counter()
    compute += t0 - t1
    if i == 200:
        break
print(f"share of time waiting for data: {data_wait / (data_wait + compute):.0%}")

Our rule of thumb: if more than 10% of the loop is spent waiting for data, fix the loader before touching the model. If the share is near zero and utilization is still low, the GPU is idle inside the step, and the profiler is the next tool.

Remove the torch.cuda.synchronize() line once you are done. The PyTorch performance tuning guide lists syncs like it as a cause of low utilization in their own right.

How do you fix a starved DataLoader?

Load batches in parallel worker processes and copy them asynchronously. The DataLoader docs set num_workers=0 by default, which loads data in the main process. The tuning guide explains the cost: the training process "has to wait for the data to be available to continue the execution."

from torch.utils.data import DataLoader

loader = DataLoader(
    dataset,
    batch_size=256,
    shuffle=True,
    num_workers=8,            # tune to your CPU cores and storage
    pin_memory=True,          # page-locked host memory for faster async copies
    persistent_workers=True,  # keep workers alive between epochs
    prefetch_factor=4,        # batches loaded ahead by each worker
)

for x, y in loader:
    x = x.cuda(non_blocking=True)
    y = y.cuda(non_blocking=True)
    ...

Each argument does one job. pin_memory=True enables "faster and asynchronous memory copy from the host to the GPU", per the tuning guide, and non_blocking=True lets the copy overlap with compute. prefetch_factor defaults to 2 when workers are on, meaning 2 x num_workers batches are queued across all workers. persistent_workers=True stops the loader from shutting workers down after each epoch.

There is no universal right num_workers. The tuning guide says it "should be tuned depending on the workload, CPU, GPU, and location of training data." Try 4, 8 and 16 and keep the smallest value that drives the data-wait share near zero.

If the workers are all running and the CPU cores sit at 100%, more workers will not help. Your augmentation or decoding is the real bottleneck. Preprocess the dataset once and save the result, or move heavy transforms onto the GPU. If the CPU is idle but batches are still late, you are waiting on storage: copy the dataset to local disk before training instead of reading every file from network storage.

Which code patterns make the GPU wait inside a training step?

Anything that forces the CPU to wait for the GPU. PyTorch queues GPU work asynchronously, and the tuning guide says to "let the CPU run ahead of the accelerator as much as possible." It lists the usual offenders: print(cuda_tensor), cuda_tensor.item(), copies like .cpu() and .to(device), nonzero(), and Python if statements that depend on a CUDA tensor's value.

The most common one is logging the loss every step:

# Stalls every step: .item() waits for the GPU to finish
running_loss += loss.item()

# Better: accumulate on the GPU and read it every 100 steps
running_loss += loss.detach()
if step % 100 == 0:
    print(f"loss {running_loss.item() / 100:.4f}")
    running_loss.zero_()

Small batches are the second pattern. Each step then launches many short kernels, and the GPU finishes each one before Python has queued the next.

Raise the batch size as far as memory and your optimizer settings allow. Where you cannot, torch.compile fuses pointwise operations into fewer kernels. The tuning guide adds that torch.compile(model, mode="reduce-overhead") uses CUDA graphs to cut launch and sync overhead, at some cost in device memory.

Why can a busy GPU still train slowly?

Because full FP32 math leaves the Tensor Cores unused. The tuning guide says mixed precision "offers up to 3x overall speedup on Volta and newer GPU architectures", and asks for matrix dimensions in multiples of 8 so Tensor Core kernels apply. That does not raise the utilization number. It does finish the same work in less GPU time.

for x, y in loader:
    x, y = x.cuda(non_blocking=True), y.cuda(non_blocking=True)
    with torch.autocast(device_type="cuda", dtype=torch.bfloat16):
        loss = loss_fn(model(x), y)
    loss.backward()
    opt.step()
    opt.zero_grad(set_to_none=True)

The PyTorch AMP docs say autocast should wrap only the forward pass and the loss, as above. If you choose float16 instead of bfloat16, read their torch.amp.GradScaler section first.

How do you read torch.profiler output for a slow step?

Profile a handful of steps after warm-up and look for gaps on the GPU timeline. The torch.profiler docs provide a schedule that skips wait steps, runs warmup steps, then records active steps:

from torch.profiler import profile, schedule, ProfilerActivity, tensorboard_trace_handler

with profile(
    activities=[ProfilerActivity.CPU, ProfilerActivity.CUDA],
    schedule=schedule(wait=5, warmup=2, active=5),
    on_trace_ready=tensorboard_trace_handler("./prof"),
) as prof:
    for step, (x, y) in enumerate(loader):
        train_step(x, y)
        prof.step()
        if step >= 12:
            break

print(prof.key_averages().table(sort_by="self_cuda_time_total", row_limit=15))

Open the trace in TensorBoard or a Chrome trace viewer. Blank stretches on the GPU row between steps mean data waiting. Blank stretches inside a step, lined up with cudaStreamSynchronize or aten::item on the CPU row, mean a sync. A dense row of tiny kernels means launch overhead.

Which symptom points to which fix?

This is the decision table we use, built from the PyTorch and NVIDIA docs above and the measurements in the two sections before it.

What you see	Likely cause	First fix
Sawtooth in dmon, data-wait share above 10%	Loader in the main process or too few workers	num_workers, pin_memory, persistent_workers, prefetch_factor
Workers running, CPU cores at 100%	CPU-bound decoding or augmentation	Preprocess once, or move transforms to the GPU
CPU idle, batches still late	Reading from network storage	Copy the dataset to local disk first
Gaps inside the step, aten::item in the trace	Host-device sync from logging or Python control flow	Accumulate on device, log every N steps
Steady low utilization, many tiny kernels	Small batch, kernel launch overhead	Larger batch, torch.compile
High utilization, slow steps	FP32 math, no Tensor Cores	torch.autocast with BF16, sizes in multiples of 8

To compare fixes on real hardware, a Nodus Sweep runs the same Job once per parameter value and reports cost, time and throughput per cell. That turns "try 4, 8 and 16 workers" into one submission instead of three.

What does low GPU utilization cost?

You pay for the hour, not for the kernels. Take an 8 x H100 SXM job that runs 24 hours at $2.60 per GPU-hour, the rate on the Nodus pricing page:

8 GPUs x $2.60 = $20.80 per hour, and $20.80 x 24 h = $499.20 for the run.

At 40% utilization, 60% of that bill buys waiting: $499.20 x 0.60 = $299.52. At 60% it is $199.68, at 80% it is $99.84, and at 95% it is $24.96.

At 40% utilization, more than half of a $499.20 run pays for a GPU that has nothing to do.

The better question is what the same work costs once fixed. That 24-hour run at 40% has 9.6 hours of GPU busy time (24 x 0.40). If a fix keeps that busy time constant and only removes waiting, the wall time becomes 9.6 h divided by the new utilization:

Utilization	Wall time	Cost on 8 x H100
40%	24 h	$499.20
60%	16 h	$332.80
80%	12 h	$249.60
95%	10.1 h	$210.19

Prices checked November 6, 2026, at the Nodus H100 SXM rate of $2.60/hr. Our arithmetic: 9.6 h / utilization x $20.80/hr.

Going from 40% to 80% utilization halves the cost of the same work, from $499.20 to $249.60.

This is a model, not a benchmark. Real fixes rarely keep busy time exactly constant, and mixed precision shrinks it too. The direction holds, though: on this run, doubling utilization saves as much as halving the hourly rate would. For sizing the run itself, see how many GPUs you need to train a model.

Does Nodus charge for idle GPU time?

Yes, while your Job is running. Nodus bills a running Job by time, so it costs its hourly rate whether the GPU is busy or waiting. The Nodus Jobs guide says a Job "charges only for the time it used". Once it finishes, "nothing is running or billed", and --max-cost stops a Job that would spend more than its cap.

nodus run --gpu L4 --image nodus/pytorch --max-cost 5 --timeout 2h -- python train.py

That ends the waste after the job. Raising utilization is still the only way to cut the waste during it.

FAQ

Does high GPU memory use mean the GPU is busy?

No. Memory that PyTorch has allocated says nothing about activity. The memory column in nvidia-smi dmon is different: NVIDIA defines it as the percent of time device memory was being read or written, so it does track activity.

Does increasing num_workers always help?

No. Past the point where data waiting reaches zero, extra workers only use more host memory, and if the CPU is already saturated they can slow the loop. Each worker also holds prefetch_factor batches in memory.

Should I set torch.backends.cudnn.benchmark = True?

For convolutional networks with fixed input sizes, yes: the tuning guide says the autotuner picks the fastest convolution algorithm for your hardware. With highly variable input sizes it can cost more than it saves, and the guide says other layer types are not supported.

How do I check utilization on every GPU of a multi-GPU job?

nvidia-smi dmon reports up to 16 devices at once, one row per GPU. Look for one GPU that lags: in data-parallel training, the others wait for it at every gradient sync.

Originally published on the Nodus blog.