Why Local Inference Never Saturates a GPU at Batch Size One

#ai #discuss #llm #machinelearning

Your GPU is not underused. It is doing the only thing it can do at batch size one, which is wait for memory — and the meter you are reading was never measuring the thing you think it is.

The claim

Generating one token requires reading every resident weight exactly once and performing roughly two floating-point operations on each. That ratio — arithmetic per byte moved — is fixed by the algorithm and is about two orders of magnitude below what a modern GPU is built for. So during single-stream generation the arithmetic units are idle almost all the time, and no amount of tuning changes it, because the bottleneck is not in the part you can tune.

This is not a defect of local inference. It is the same property that makes hosted inference cheap: providers recover the wasted arithmetic by batching many users’ tokens into one weight read. You cannot, because you are one user.

The reason this is worth arguing rather than stating is that it contradicts an intuition people act on. If the GPU looks idle, the obvious move is to find the setting that makes it busy — more threads, a different backend, a bigger batch, a core overclock. Almost all of that effort goes into the half of the machine that is not the constraint, and the disappointment that follows gets attributed to the runtime being immature rather than to the workload being the shape it is. Knowing which resource is saturated tells you which experiments are worth running, and the answer here is unusually clear-cut.

One FLOP per byte against two hundred

Arithmetic intensity is operations divided by bytes moved. For a matrix multiply against a single vector — which is what decode is, at every layer — each weight is loaded, multiplied and accumulated:

decode, batch size 1:
  operations per parameter   2      (one multiply, one add)
  bytes per parameter        2      (fp16 weights)
  arithmetic intensity       1.0 FLOP per byte

  at 4.9 bits per weight (Q4_K_M) the weight is 0.61 bytes,
  but it is dequantized before use, so the ratio improves only
  to roughly 3 FLOP per byte of memory traffic.

what the hardware wants:
  peak fp16 throughput / memory bandwidth
  a card offering ~165 TFLOP/s against ~1 TB/s wants ~165 FLOP per byte

A workload supplying 1 to 3 FLOPs per byte on hardware that needs 165 to be busy is using on the order of 1% of the arithmetic. That is the whole argument. Everything below is evidence for it and consequences of it.

The same arithmetic gives the roofline. Time per token cannot be less than weight bytes over bandwidth:

tok/s ceiling = bandwidth / weight_bytes

Llama 3.1 8B at Q4_K_M, 4.58 GiB = 4.92e9 bytes
  at 1008 GB/s ->  205 tok/s
  at  360 GB/s ->   73 tok/s

Llama 3.1 70B at Q4_K_M, 40.2 GiB = 43.2e9 bytes
  at 1008 GB/s ->   23 tok/s

Substitute your card’s published bandwidth. These are upper bounds that assume perfect streaming and zero overhead; nothing reaches them. But they predict the ordering and the magnitude, which a compute-bound model of the same workload does not.

The published evidence

The prediction is testable against a source neither written for this argument nor produced by us. llama.cpp’s quantize README publishes, for Llama-3.1-8B on one machine, both prompt-processing and text-generation rates at every quantization level. Prefill processes many tokens in one pass and is compute-bound; decode produces one token per pass and, by this argument, should be bandwidth-bound. If the argument holds, the two should behave completely differently as the file size changes.

quant     size (GiB)   prefill t/s   decode t/s   size x decode
F16          14.96         923.49        29.17         436
Q8_0          7.95         865.09        50.93         405
Q6_K          6.14         812.01        58.67         360
Q5_K_M        5.33         758.69        67.23         358
Q4_K_M        4.58         821.81        71.93         329
Q2_K          2.95         784.45        79.85         236

Prefill is flat: it varies by about 20% across a range where the file size varies by 5x, and not monotonically. Decode moves by 2.7x over the same range, in the direction size predicts. And the last column — file size multiplied by decode rate, which is bytes read per second — is roughly constant at the large end and falls off at the small end, which is exactly what a bandwidth-bound process with a fixed per-step overhead looks like: once the weights are small enough, the constant costs stop being negligible.

Those numbers are llama.cpp’s, measured on hardware the README does not name, so the absolute values mean nothing here. The structure means everything. A compute-bound decode would not track file size at all, because Q4 and F16 do the same number of multiply-accumulates.

The prefill column is the control, and it is what makes the argument more than a story that fits. If both halves were bandwidth-bound, prefill would track file size too and it does not. If both were compute-bound, decode would be flat and it is not. Two measurements of the same model on the same machine, differing only in how many tokens go through per pass, land on opposite sides — which is exactly the prediction that follows from arithmetic intensity, since prefill amortises one weight read over hundreds of tokens and decode over one.

The residual detail — that bytes read per second falls as the file gets smaller — is worth reading as a second confirmation rather than a discrepancy. Fixed per-step costs do not shrink with the weights: launching kernels, reading and writing the cache, sampling, and, for the lower quants, more work per byte to unpack the block structure. As the weights shrink, those costs become a larger share of each step, so the effective bandwidth achieved drops. A process bounded by something other than weight traffic would not show that pattern either.

What the utilisation number measures

The meter people check is utilization.gpu from nvidia-smi, and it does not report the fraction of arithmetic units doing work. It reports the fraction of a sampling period during which at least one kernel was executing. A kernel that is stalled on memory for 99% of its cycles is still executing.

So the reading can be misleading in both directions. It shows a high number while the arithmetic units are nearly idle — the card is busy waiting, and busy is what it counts. And it shows a low number on a model with many small kernels, where the gaps between launches fall inside the sampling window, without anything being wrong. Neither reading tells you whether you are near the roofline. The bandwidth calculation above does, in one division. There is more on reading these counters under GPU utilisation.

What follows from it

Quantization buys speed, not just space. Fewer bytes per weight is less to read per token. This is why the decode column above tracks file size, and it is a second reason to quantize beyond fitting on the card.
Overclocking the core does approximately nothing. Memory clock is the lever that matters for generation. Core clock matters for prefill, which is the compute-bound half.
Concurrency is nearly free until it is not. Serving a second request reads the same weights for both tokens, so throughput scales close to linearly at small batch sizes without hurting per-request latency much. This is exactly what a hosted provider does, and why per-token prices are far below what the same model costs to run alone — the argument is in local versus API cost.
A sparse model breaks the size-to-speed link. A mixture-of-experts model reads only its active parameters per token while holding all of them, so it decodes like a small model and sizes like a large one. That asymmetry is derived on the MoE memory page.
Spilling to system RAM is catastrophic for the same reason. If bandwidth sets the pace, moving 10% of the weights to a bus ten times slower roughly doubles the time per token. The derivation is on the overflow page.