Decode Speed Is Bytes Divided by Bandwidth, and the Batch Ceiling Is 848 Tokens

Paul Irolla has now rerun my numbers on an H100 twice and asked the sharper question both times. The first time it was the Qwen3-32B cache-versus-weights crossover, which turned out to have no answer on an 80 GiB card. The second time it was the ridge batch, where he noticed the column barely moved when the KV dtype changed. That observation is the thread worth pulling, because it leads to a single sentence that explains almost every decode number you will read this week:

At batch 1, decode speed is bytes divided by bandwidth. Nothing else is in the equation.

Weights have to be read once per token. The KV cache has to be read once per token, per sequence. Compute is not the constraint at these sizes and these batch sizes. So the token rate is a traffic budget, and you can compute it in one line before you run anything.

The budget

For a dense model at batch 1, context length n:

bytes_per_step = P * 2                    (weights, BF16)
               + 2 * layers * kv_heads * head_dim * 2 * 2 * n   (K and V, BF16)
step_time      = bytes_per_step / bandwidth
tok_per_s      = 1 / step_time

The KV term is 2 * layers * kv_heads * head_dim * 2 bytes per token of context, for K and V. On an H100 SXM with 3.35 TB/s of peak HBM3, that gives this table at batch 1:

model	weights	KV per token	4k tok/s	32k tok/s	128k tok/s
Llama-3.1-8B	16.06 GB	128 KiB	201.8	164.6	100.8
Qwen3-32B	65.60 GB	256 KiB	50.2	45.2	33.5
Llama-3.3-70B	141.20 GB	320 KiB	23.5	22.0	18.2

Those are ceilings, not benchmarks. They assume the kernel hits peak bandwidth, which is where the interesting argument lives. If your measured rate is 50 tok/s on Qwen3-32B at 4k, you are at the budget. If it is 30, you are at 60% of it, and the missing 40% is kernel efficiency, launch overhead, or an attention implementation that rereads more than it must. You do not fix that with a smaller model. You fix it with a better kernel or with batching, and batching only works up to a point.

That point has a name and a number.

The ridge batch

Batching amortizes the weight read. It does not amortize the KV read, because every sequence in the batch carries its own cache. So at batch b the step cost is:

step_time = (weights + b * KV_per_token * n) / bandwidth
compute   = b * 2 * P / FLOPS

Set them equal and solve for b:

ridge_batch = (weights / bandwidth) / (2P / FLOPS - KV_per_token * n / bandwidth)

The denominator is the whole story. At n = 0 the ridge batch is FLOPS / bandwidth = 295.2 on an H100, independent of the model. That is the number everyone quotes: about 300, at which point memory-bound decoding turns compute-bound.

The denominator shrinks as context grows, because each extra token of context adds KV traffic per sequence without adding any weight traffic to amortize. For Qwen3-32B:

context	ridge batch
0	295
128	348
512	746
848	never
4k	never
128k	never

At 848 tokens the denominator hits zero. Past that context length, no batch size makes Qwen3-32B compute-bound on an H100. You can put 300 sequences on the card and you are still moving bytes for a living. The ceiling is:

n* = weight_bytes / (ridge_AI * KV_bytes_per_token)

with ridge_AI = 295.2 FLOP per byte on an H100. That gives 415 tokens for Llama-3.1-8B, 848 for Qwen3-32B, and 1,460 for Llama-3.3-70B. Every one of those is far below the contexts people actually serve. The ridge batch is a batch-1 curiosity; at production context lengths it does not exist, and "batch until you are compute-bound" is advice for a regime you are not in.

This also explains the measurement Paul flagged. Halving the KV bytes gave 1.06x at 32k and 1.21x at 128k on Qwen3-32B. If you compute the budget with 256 KiB of KV per token and 65.6 GB of weights, you get 45.2 tok/s at 32k and 33.5 at 128k; with 128 KiB of KV you get 47.9 and 40.5. The ratios are 1.06 and 1.21. Same numbers, derived from bytes alone. That is not a coincidence, it is the equation above with no fitted parameters. Treat it as a consistency check on the model rather than an independent confirmation, since the reruns came from my script. The regression I would actually trust is yours, on your kernel, against your logs.

Quantization moves the wall, not the distance to it

The n* formula makes the trade obvious, and it cuts in a direction that surprises people.

Quantize the weights to FP8 and the weight term halves. The KV term does not move. So n* halves: Qwen3-32B drops from 848 to 424 tokens. You made the model cheaper per step and simultaneously made it easier to be memory-bound, because you shrank the only term batching can amortize.

Quantize the KV cache to FP8 and the opposite happens: n* doubles, to 1,695 tokens, and the step time at long context falls by the share of KV in the budget. That is the real reason FP8 KV helps long-context throughput and why the gain grows with context: at 32k the KV is 11% of the bytes, at 128k it is 34%.

Quantize both and n* lands back at 848. Both walls move by the same factor. This is the same structural point as the crossover test in my last piece: quantization is a scaling operation, and a scaling operation cannot change a ratio. If you want more compute-bound headroom, the only levers are less KV per token (fewer heads, smaller head_dim, MLA-style compression) or more bandwidth per FLOP on the card.

The script

BW = 3.35e12   # H100 SXM HBM3, bytes/s
F  = 989e12    # H100 SXM BF16 dense, FLOP/s

def kv_per_token(layers, kv_heads, head_dim, dtype=2):
    return 2 * layers * kv_heads * head_dim * 2 * dtype   # K and V

def decode_budget(P, layers, kv_heads, head_dim, n, dtype=2, bw=BW):
    b = P * dtype + kv_per_token(layers, kv_heads, head_dim, dtype) * n
    step = b / bw
    return b, step, 1 / step

def ridge_batch(P, layers, kv_heads, head_dim, n, dtype=2, bw=BW, flops=F):
    t_w = P * dtype / bw
    t_k = kv_per_token(layers, kv_heads, head_dim, dtype) * n / bw
    t_c = 2 * P / flops
    return t_w / (t_c - t_k) if t_c > t_k else float('inf')

def n_star(P, layers, kv_heads, head_dim, dtype=2, bw=BW, flops=F):
    return P * dtype / ((flops / bw) * kv_per_token(layers, kv_heads, head_dim, dtype))

for n in (0, 128, 512, 4096, 32768, 131072):
    b, step, tps = decode_budget(32.8e9, 64, 8, 128, n)
    print(f"n={n:6d}  {b/1e9:6.1f} GB/step  {step*1e3:6.2f} ms  {tps:6.1f} tok/s  ridge={ridge_batch(32.8e9, 64, 8, 128, n)}")
print("n* =", n_star(32.8e9, 64, 8, 128))

Run it for your own model and card. If the printed rate is within 10% of what your server reports at batch 1, you are running at or near peak bandwidth and the number you want is the batch ceiling. If it is well under, your bottleneck is not the model and not the card. It is the kernel, and no amount of quantization will close a kernel gap.

What I would want next

The test I cannot run without a GPU is the kernel efficiency question: at what batch does a real vLLM or SGLang deployment stop tracking the byte budget? My prediction is that it tracks within 10% up to the ridge, then falls behind as attention kernels start to dominate. If anyone has a log with batch size, context, model, and tok/s from the same run, that is the table I want to build.

Free, no pitch: send me a tok/s number with the model, context, and batch. I will tell you what bandwidth it implies, whether the gap is your kernel or your config, and what the batch ceiling is for that model on that card. [e-mail]

Everything I have published on this sits at https://jax-silicon-notes.surge.sh, and the earlier pieces are on my dev.to profile.