A reader, @paul_irolla, asked me a fair question under my last piece: I run the cache-versus-weights crossover for Llama, so run the same test for Qwen3-32B before buying either number.
I ran it. On a single 80 GiB H100, the test has no answer. Not because of Qwen, not because of FP8, not because of anything about how you quantize. Because of the card.
Here is the arithmetic, and then the part I got wrong the first time I framed it.
There are two gates, not one
The crossover I use is the context length where the KV cache holds as many bytes as the weights:
L_cross = W / k W = weight bytes, k = KV bytes per token
The second gate is the one that actually kills deployments. It is the context length that fits in HBM next to the weights:
L_fit = (HBM - W) / k
The crossover is only reachable if L_fit >= L_cross, which simplifies to:
HBM >= 2 * W
That is it. No dtype, no attention variant, no model family. If your card cannot hold twice the weight bytes at zero context, the cache will never pass the weights, and the "which is bigger" test is not a question your box can answer.
I had been treating the crossover as a model property. It is a card property applied to a model size.
Qwen3-32B on three cards
Qwen3-32B: 64 layers, 8 KV heads, head_dim 128, so 256 KiB per token of BF16 KV cache (131,072 bytes in FP8). BF16 weights are 65.6 GB, which is 76% of an 80 GiB H100 before a single token arrives.
| card | KV dtype | crossover | fits in HBM | reachable | seqs @128k | weights share |
|---|---|---|---|---|---|---|
| H100 80GB | BF16 | 250.2k | 77.4k | no | 0.59 | 76% |
| H100 80GB | FP8 | 500.5k | 154.9k | no | 1.18 | 76% |
| H200 141GB | BF16 | 250.2k | 327.3k | yes | 2.50 | 43% |
| H200 141GB | FP8 | 500.5k | 654.6k | yes | 4.99 | 43% |
| B200 180GB | BF16 | 250.2k | 487.0k | yes | 3.72 | 34% |
| B200 180GB | FP8 | 500.5k | 974.1k | yes | 7.43 | 34% |
On the H100 the crossover sits at 3.23x the fit limit. Quantizing the cache to FP8 does not narrow that: it doubles both numbers, because both are computed by dividing by k. It moves the wall you hit from 77.4k tokens to 154.9k tokens, which is a real and useful thing, and it does not change the shape of the problem at all.
The 3.23 is W / (HBM - W). It is fixed by how much of the card the weights already take. At 76% of the card, three quarters of your memory is spent before the cache earns a single byte.
Move to a card that holds twice the weight bytes and the same model flips:
- H200 141 GB: weights are 43% of the card, fit 327.3k beats crossover 250.2k, and at 128k the box holds 2.50 sequences at BF16 KV.
- B200 180 GB: weights are 34%, fit 487.0k, 3.72 sequences at 128k.
Same weights, same KV format, opposite answer. The card decides.
Which models can reach the crossover at all
The rule HBM >= 2 * W sorts models fast:
| model | BF16 weights | share of H100 | crossover (BF16 KV) | fit on H100 | reachable |
|---|---|---|---|---|---|
| Llama-3.1-8B | 16.1 GB | 19% | 122.5k | 532.8k | yes |
| Qwen3-32B | 65.6 GB | 76% | 250.2k | 77.4k | no |
| Llama-3.3-70B | 141.2 GB | 164% | 430.9k | does not fit | no |
Llama-3.1-8B reaches its crossover at 23% of what the card can hold. Qwen3-32B never gets there on 80 GiB, needs an H200 to get there, and Llama-3.3-70B needs a card holding 282 GB before the question even applies, which is four H100s in tensor parallel or nothing.
The part that actually bites: cache share of the read
The per-token read at context L is W + L * k, so the cache share of that read is L * k / W. Set the cache share next to the crossover and the same numbers tell you whether FP8 KV buys you speed or capacity.
Single-stream ceiling on one H100 at 3.35 TB/s:
| model | cache share @128k, BF16 | tok/s @4k | tok/s @128k | tok/s @128k, FP8 |
|---|---|---|---|---|
| Llama-3.1-8B | 51.7% | 201.8 | 100.8 | 135.9 |
| Qwen3-32B | 34.4% | 50.2 | 33.5 | 40.5 |
| Llama-3.3-70B | 23.3% | 23.5 | 18.2 | 20.6 |
Read those rows as an inversion. The model everyone worries about, 70B, has the smallest cache share of any of them at 128k, because its weights already saturate the read. The model nobody worries about, 8B, loses more than half its per-token bandwidth to the cache at the same context, and FP8 KV hands it 1.35x.
So "long context is expensive because of the KV cache" is true in the high-throughput regime, which is the small-model-on-a-big-card regime. On a 32B at 76% of an H100, the cache is never the bind. The weights are. FP8 KV there is a capacity purchase, one extra sequence at 128k, and the speed you collect is a side effect.
The operational version:
- Weights under 25% of the card: size on the cache. The crossover is real, contextual, and FP8 KV buys tokens per second.
- Weights over 50% of the card: size on the card. The cache never passes the weights, FP8 KV buys sequences, and no amount of KV tuning will move the wall.
The script
GIB = 1024**3
HBM = {"H100 80GB": 80*GIB, "H200 141GB": 141*GIB, "B200 180GB": 180*GIB}
BW = {"H100 80GB": 3.35e12, "H200 141GB": 4.80e12, "B200 180GB": 8.00e12}
# name: (params_b, layers, kv_heads, head_dim)
MODELS = {
"Llama-3.1-8B": (8.03, 32, 8, 128),
"Qwen3-32B": (32.8, 64, 8, 128),
"Llama-3.3-70B": (70.6, 80, 8, 128),
}
def kv_bytes(layers, kvh, hd, bpp):
return 2 * layers * kvh * hd * bpp # K and V, per token
for card, mem in HBM.items():
for m, (p, l, h, d) in MODELS.items():
w = p * 1e9 * 2 # BF16 weights
for kvb, tag in ((2, "BF16"), (1, "FP8")):
k = kv_bytes(l, h, d, kvb)
crossover = w / k
fit = (mem - w) / k
seqs = (mem - w) / (131072 * k)
print(f"{card:12s} {m:14s} {tag:5s} "
f"cross={crossover/1e3:7.1f}k fit={fit/1e3:8.1f}k "
f"{'reachable' if fit >= crossover else 'never':10s} "
f"seqs@128k={seqs:5.2f}")
# speed ceiling and cache share of the per-token read
for m, (p, l, h, d) in MODELS.items():
w = p * 1e9 * 2
for kvb, tag in ((2, "BF16"), (1, "FP8")):
k = kv_bytes(l, h, d, kvb)
for ctx in (4096, 131072):
print(f"{m:14s} {tag:5s} ctx={ctx:6d} "
f"{3.35e12/(w+ctx*k):6.1f} tok/s "
f"cache share {100*ctx*k/(w+ctx*k):5.1f}%")
What this model leaves out
Three things, so you can price the answer correctly.
These are bandwidth ceilings, not measured throughput. They assume the read is the only term and that attention FLOPs are free; past some context length, attention itself becomes the cost, and these numbers overstate the gain from cheapening the cache.
They assume one sequence and no paged attention. With PagedAttention or chunked prefill the arithmetic per token changes, though the two gates do not: L_cross = W/k and L_fit = (HBM - W)/k are just byte accounting.
And they ignore tensor parallelism, which changes HBM and W together and mostly leaves the ratio alone. Eight H100s give you eight times the memory and one eighth of the weights per card, so a 70B on 8x80GB lands back at 21% of each card. That is the same regime as the 8B on one card, and the same conclusion follows.
Run it on your own card and model. If your measurement disagrees with my ceiling, the measurement is right.
One free thing, no invoice: send me your card, your model's layer and KV-head count, and the context length you actually serve, and I will reply with the crossover, the fit limit, and which of the two is binding for you. One number each, no call required.
Top comments (0)