Halving the KV Cache Doubles Capacity and Buys 1% of Speed

A reader ran the script from my last piece against the table, found a bug in my code, and asked a better question than the one I answered.

The bug first. For Llama-3.3-70B on an 80 GiB H100 the fit column printed a negative sequence count, about -41 at 4k. The table said "does not fit" and the table was right, but the script printed a number, and a number that means nothing invites a diff. It now has a no fit branch. Fixed in both places.

The question was this: if the engine swaps to an FP8 KV cache, does the long-context penalty halve?

Short answer. Capacity halves, exactly, at every context, on every card. Speed does not, at any context you would call long. Both answers come out of one formula, and they only look like the same question until you write it down.

The gain is a ratio, not a factor

The single-stream ceiling is bandwidth divided by bytes read per token:

tok/s = bw / (W + L·K)

W is the weight bytes, L the context length, K the cache bytes per token. Halve K and the gain is (W + L·K) / (W + L·K/2). It has two limits and no middle setting:

• when the weight read dominates (L·K << W), the gain is 1.00x
• when the cache dominates (L·K >> W), the gain approaches 2.00x

Everything between is arithmetic. For Qwen3-32B on an H100, computed:

ctx	tok/s, BF16 KV	tok/s, FP8 KV	speed gain	cache read as share of total
4k	50.2	50.7	x1.01	1.6%
32k	45.2	47.9	x1.06	11.6%
64k	40.5	45.2	x1.12	20.0%
128k	33.5	40.5	x1.21	34.3%

Llama-3.1-8B on the same card, whose cache is heavier relative to its weights, goes 201.8 -> 205.2 at 4k (x1.02) and 100.8 -> 135.9 at 128k (x1.35). Llama-3.3-70B, whose weights are 131.5 GiB and swamp everything else, goes x1.00 at 4k and x1.13 at 128k.

So "does it halve the penalty" is the wrong shape of question. It scales the cache term, and how much that matters depends entirely on how big the cache term already was.

Capacity is linear, and that is why it doubles exactly

Sequences that fit is (mem - W) / (L·K). K appears once, linearly, with nothing else in the denominator. Halve it and you get exactly 2.00x, at every model, every card, every context length. Qwen3-32B on H100: 18.91 to 37.81 sequences at 4k, 2.36 to 4.73 at 32k, 0.59 to 1.18 at 128k.

No approximation, no log, no interaction terms. That is the clean half of the answer, and it is the half that actually changes what you can deploy.

The crossover is the only number that matters

The gain stops being a rounding error when L·K = W, i.e. when the cache read equals the weight read. Solve for context length on BF16 weights:

Model	weights	cache/token	cache = weights at
Llama-3.1-8B	15.0 GiB	128 KiB	122k tokens
Qwen3-32B	61.1 GiB	256 KiB	250k tokens
Llama-3.3-70B	131.5 GiB	320 KiB	431k tokens

Below its crossover, FP8 KV is buying room and nothing else. Above it, speed comes along for free. The 8B model crosses at 122k, which is why it is the one model in the set where quantization is a throughput strategy and not just a capacity strategy. The 70B crosses at 431k, which no deployment runs.

The case that actually decides it

Qwen3-32B at 128k on an 80 GiB H100 holds 0.59 sequences at BF16 KV. Two-thirds of a sequence. That is not a tight deployment, it is not a deployment: 61.1 GiB of weights plus 32 GiB of cache against 80 GiB of HBM.

At FP8 KV it holds 1.18. One session, with room for scratch buffers. The throughput gain there is 21%, which is not the point. The point is that the long-context configuration exists at all.

That is the real division of labor between KV quantization and everything else people try. You quantize to make a configuration fit. You get a small speed bonus as a side effect, rising with context, and you should never expect the speed bonus to justify the work.

What quantizing the cache cannot do

The batch you need to reach the ridge point barely moves: 295.6 to 295.4 on H100, for both precisions. That column is set by bytes per parameter, not by the cache. Solve the intensity equation and the cache term cancels out of the leading order.

Which means: quantizing the cache never makes a large model compute-bound. It puts more sequences in front of the same weight read, and the memory wall stays exactly where it was. And it never fixes a model that does not fit at all. Llama-3.3-70B on an 80 GiB H100 prints no fit at 4k, 32k and 128k, on BF16 and FP8 KV both, because HBM is 51 GiB short of the weights alone before a single token of context.

What it costs

Quantization error lands on the attention logits, and attention is where a small error gets amplified by a softmax over many keys. That is why long context degrades first: more keys compete, relative magnitudes matter more, and the error has more places to show up.

MLA makes this harder in a way worth naming. DeepSeek-V3's cache is a low-rank latent, 576 values per token per layer, 68.6 KiB at FP16, and that latent is already a lossy compression of full K and V. Quantizing it to 8 bits means stacking a second lossy step on the first, with the error budget the first step did not spend. There is also a compute cost that the byte ratio hides: MLA up-projects the latent at attention time, so at FP8 the up-projection reads quantized input, and you save bytes without saving the matmul.

I have no measurement for MLA at FP8, and I am not going to invent one. The mechanism says the error budget is thinner than for GQA, and the honest procedure is to measure at 64k before shipping, not to extrapolate from a 4k eval.

The script

Run it against your own configs. KV_BPP is the only line you change.

GiB = 1024**3

GPUS = {
    "H100 SXM 80GB":  (80*GiB,  3.35e12, 989e12),
    "H200 SXM 141GB": (141*GiB, 4.80e12, 989e12),
    "B200 SXM 180GB": (180*GiB, 8.00e12, 2250e12),
}
# name: (params_b, layers, kv_heads, head_dim)
MODELS = {
    "Llama-3.1-8B":  (8.03, 32, 8, 128),
    "Qwen3-32B":     (32.8, 64, 8, 128),
    "Llama-3.3-70B": (70.6, 80, 8, 128),
}
W_BPP  = 2                       # BF16 weights, unchanged either way
KV_BPP = {"BF16 KV": 2, "FP8 KV": 1}

gname = "H100 SXM 80GB"
mem, bw, flops = GPUS[gname]
ridge = flops / bw
print(f"{gname}: {bw/1e12:.2f} TB/s, ridge {ridge:.0f} FLOP/byte\n")
print(f"{'model':15s} {'ctx':>7s} {'KV':>8s} {'tok/s':>7s} {'seqs':>7s} {'ridge batch':>11s}")
for mname, (p, layers, kvh, hd) in MODELS.items():
    w = p * 1e9 * W_BPP
    for ctx in (4096, 32768, 131072):
        for kname, kbpp in KV_BPP.items():
            kv   = 2 * layers * kvh * hd * kbpp   # cache bytes per token
            tps  = bw / (w + ctx * kv)            # single-stream ceiling
            room = (mem - w) / (ctx * kv)         # sequences that fit in HBM
            fit  = f"{room:7.2f}" if room > 0 else " no fit"
            need = ridge * w / (2 * p * 1e9 - ridge * kv)  # batch at the ridge
            print(f"{mname:15s} {ctx:7d} {kname:>8s} {tps:7.1f} {fit} {need:11.1f}")

The correction in this piece came from a reader running the code, not from a reviewer reading it. That is the only kind of review that finds arithmetic. If you run the script and get a different number, I want the number and the config.

The previous piece on decode rooflines is here, and the byte counts this all rests on are here. The rest of the series lives in my archive.