Decode Is 300x Memory-Bound. Batching Fixes That Until the Context Gets Long.

I spent last week arguing about KV cache sizes. The useful follow-up is simpler: multiply the cache by the context length and you get the number that actually sets your throughput. Decode does not run out of FLOPs. It runs out of memory bandwidth, by a factor of about 300, and every GPU generation keeps it that way.

Here is the arithmetic, computed rather than asserted, plus a script that reads your own model config and prints your ceiling.

Decode is one FLOP per byte read

For a dense model at batch size 1, generating one token means reading every weight tensor once and doing two multiply-adds per parameter. Bytes read per token: params × bytes_per_param. FLOPs per token: 2 × params.

So arithmetic intensity of decode is 2 / bytes_per_param. In BF16 that is 1.0 FLOP per byte. On an FP8 checkpoint, 2.0.

Now the machine's own ratio. H100 SXM: 989 TFLOPS of dense BF16 tensor throughput over 3.35 TB/s of HBM3, a ridge point of 295 FLOP/byte. B200: 2,250 TFLOPS dense BF16 over 8 TB/s, a ridge point of 281. H200: 989 over 4.8 TB/s, 206.

Three generations, same ratio within 40%. That is not a coincidence. Both numbers scale with process and packaging, and they scale together. NVIDIA acquired Mellanox for reasons like this: the compute/bandwidth balance is the design point, not an accident.

Decode sits at 1 FLOP/byte against a ridge of ~300. That is the 300x. Not a modeling artifact, not a vendor's inefficiency: it is what the operation is.

What that means per GPU

Bandwidth divided by (weights + KV) is your single-stream token ceiling. No scheduler beats it.

Model (BF16)	Weights	KV per token	H100 3.35 TB/s @4k	H100 @128k	B200 8 TB/s @4k	B200 @128k
Llama-3.1-8B	15.0 GiB	128 KiB	202 tok/s	101	482	241
Qwen3-32B	61.1 GiB	256 KiB	50	34	120	80
Llama-3.3-70B	131.5 GiB	320 KiB	does not fit	does not fit	56	43

Measured servers come in 20-40% under these, because an engine also reads scratch buffers, does attention over the whole cache rather than one block, and stalls on synchronization. If your vLLM instance reports 35 tok/s on a 32B model at 4k on one H100, this table said 50 and it is not lying about either number.

Two things worth sitting with. A 32B model in BF16 plus a 128k-token context needs 93 GiB. It does not fit on an 80 GiB card at all, and you cannot argue your way around that with a better engine. And context alone cuts single-stream speed by a third on Qwen3-32B: 50 down to 34 tok/s, because at 128k the cache is 32 GiB and reading it per token is the workload.

On B200 the same model at 128k still leaves 3.7 sequences of room. Sit on that number, it comes back in two sections.

The batch you need, and why it will not fit

Batching is the standard escape. If B sequences share the weight read, intensity becomes 2 × params × B / (weights + B × KV). Weights get amortized, intensity climbs with B, and at some batch you cross the ridge and the tensor cores finally become the constraint.

Solving for that B on Qwen3-32B gives 282 sequences. Every model lands within a few percent of the same figure, and the reason is dull: at the crossover the KV term has to be small relative to the weights, so B ≈ ridge × bytes_per_param / 2. For a BF16 checkpoint that is just the ridge number again. 295 for BF16 on H100, 282 on B200, 206 on H200.

Now the memory. Each of those 282 sequences needs its own KV. At 4k tokens, Qwen3-32B's cache is 1 GiB per sequence, so the batch costs 282 GiB before you count the 61 GiB of weights. B200 has 180 GiB total.

Model (BF16), B200 180 GiB	seqs that fit @4k	seqs that fit @128k	seqs needed to reach the ridge
Llama-3.1-8B	330	10	282
Qwen3-32B	119	4	282
Llama-3.3-70B	39	1	281

Read the first row against the last column: the 8B model at 4k can actually get there. That is the one honest exception, and it is why small-model deployments talk about compute and big-model deployments talk about memory.

Everything else is capped. Qwen3-32B at 4k can reach 119 of the 282 sequences it would need. HBM is the wall, and no batching policy, continuous or otherwise, manufactures HBM.

Long context takes the ladder away

At 128k, Qwen3-32B fits 4 sequences on a 180 GiB B200 after weights. Four against a ridge requirement of 282 is 1.4% of the way. Long context does not degrade gracefully into compute-bound territory; it removes the ladder.

This is why the interesting engineering in serving has moved to KV compression, paging, and offload rather than to batching heuristics. Every byte you remove from the cache is a byte of bandwidth you get back, twice: once in the read, once in the room it frees for another sequence. Grouped-query attention with 8 KV heads is a 8x cut against multi-head, which is a bigger single win than most kernel work. MLA in DeepSeek-V3 lands at 68.6 KiB per token against 256 KiB for a comparable GQA model, and that is the whole reason a 671B model is servable on a node.

What prefix caching actually buys

A reader asked on the KV piece whether prefix caching bends the decode curve or only shortens time to first token. The honest answer: it bends TTFT, not the decode slope, and it bends capacity, which indirectly helps.

Decode still reads the full cache for every token. A shared prefix removes no bytes from that read. What it removes is the prefill, which is a compute-phase workload running near the ridge, and it removes duplicate copies of the same KV from HBM. That second effect is real and large in agent systems: 100 concurrent requests sharing a 2,000-token system prompt cost 50 GiB of KV without sharing and 0.5 GiB with it. Capacity you did not have is capacity you can spend on batch.

So: prefix caching buys you room to batch. Batching buys you intensity. Neither one buys bandwidth.

Check it on your own machine

Three measurements, ten minutes:

1. Read your model's config.json: num_hidden_layers × 2 × num_key_value_heads × head_dim × dtype_bytes. Compare against the table above. If your number is bigger, your cache is bigger, and your context budget is smaller.
2. Time a single stream at batch 1, then at batch 32, at fixed context. Expect roughly flat per-sequence latency at 1 and a drop at 32 for small models, and near-linear growth for any model whose cache is large relative to its weights.
3. Watch nvidia-smi HBM utilization during generation. Above 80% at batch 1 confirms memory-bound. Below 30% with the tensor cores busy means you found the compute-bound regime. Most long-context inference never gets there.

The script

GiB = 1024**3

GPUS = {
    "H100 SXM 80GB":  (80*GiB,  3.35e12, 989e12),
    "H200 SXM 141GB": (141*GiB, 4.80e12, 989e12),
    "B200 SXM 180GB": (180*GiB, 8.00e12, 2250e12),
}
# name: (params_b, layers, kv_heads, head_dim, bytes_per_param)
MODELS = {
    "Llama-3.1-8B":  (8.03, 32, 8, 128, 2),
    "Qwen3-32B":     (32.8, 64, 8, 128, 2),
    "Llama-3.3-70B": (70.6, 80, 8, 128, 2),
}

for gname, (mem, bw, flops) in GPUS.items():
    ridge = flops / bw
    print(f"\n{gname}: {bw/1e12:.2f} TB/s, ridge {ridge:.0f} FLOP/byte")
    for mname, (p, layers, kvh, hd, bpp) in MODELS.items():
        kv = 2 * layers * kvh * hd * bpp      # bytes per token
        w = p * 1e9 * bpp
        for ctx in (4096, 131072):
            tps = bw / (w + ctx * kv)          # single-stream ceiling
            fit = (mem - w) / (ctx * kv)       # sequences that fit in HBM
            need = ridge * w / (2 * p * 1e9 - ridge * kv)  # batch at the ridge
            print(f"{mname:16s} ctx={ctx:6d} ceiling={tps:6.1f} tok/s "
                  f"fit={fit:6.1f} need={need:7.1f}")

Run it against your own numbers before you buy the bigger card. On the configs above, a faster GPU buys you single-stream speed almost exactly in proportion to its bandwidth, which is the least interesting thing about it. What you actually want is HBM, because HBM is what you are short of.

The rest of this series, including the KV cache byte counts this piece builds on, lives in my archive. The KV piece is here.

Numbers are computed, not quoted. If one of mine is wrong, say which and I will re-derive it.