FP8 KV Cache Quantization: Why Keys Break Before Values

You flipped on --kv-cache-dtype fp8_e4m3, doubled your max concurrency, watched perplexity move by 0.01, and shipped it. Two weeks later support tickets say the model "forgets the contract clause on page 40" — but only in long documents, and only sometimes.

Nothing is broken in your serving stack. You quantized the K cache and the V cache with the same scheme, and those two tensors have completely different error tolerances. KV cache quantization is asymmetric, and almost every default treats it as symmetric.

TL;DR

• Key cache errors get exponentiated by softmax; value cache errors get averaged. A quantization error that shifts an attention logit by 0.86 multiplies that token's attention weight by ~2.4×. The same relative error on V just perturbs one term of a weighted mean.
• K has channel-consistent outliers, V does not. A handful of head dimensions carry 10–100× the magnitude of the rest in every cached key, so a single per-tensor scale burns most of your bits on channels you don't care about.
• In a synthetic head (d=128, 4096 tokens, planted outlier channels), per-tensor INT8 on K+V gave 10.5% output error; per-channel K + per-token V gave 1.9%; quantizing V alone gave 0.7%.
• Perplexity will not catch this. The damage is concentrated in long-range retrieval, which contributes almost nothing to average next-token loss. Use needle/multi-hop retrieval evals and per-layer attention KL.
• Practical lever: quantize V harder than K. llama.cpp exposes this directly (--cache-type-k q8_0 --cache-type-v q4_0); vLLM currently gives you one dtype for both, so FP8-E4M3 with calibrated k_scale/v_scale is the safe setting.

Why does the K cache break before the V cache?

Because keys go through exp() and values do not. Attention is:

logits = (q · k_j) / sqrt(d)      →  softmax  →  out = Σ_j p_j · v_j

An error ε in one key's logit changes that token's unnormalized weight by a factor of e^ε. An error δ in one value vector changes the output by p_j · δ — scaled down by an attention weight that is usually far below 1, and partially cancelled against the other 4,000 value vectors you're averaging.

So the error paths are structurally different: K error is multiplicative and amplified, V error is additive and averaged. At 32k+ context with a flat attention distribution, thousands of small independent V errors average toward zero, while one distorted key can steal probability mass from the token that actually holds the answer.

Here's the measurement. Synthetic single head, d=128, 4096 cached tokens, three planted outlier channels — the structure real K caches show:

import torch

torch.manual_seed(0)
T, D = 4096, 128                      # cached tokens, head_dim

K = torch.randn(T, D)
K[:, [7, 23, 88]] *= 20.0             # channel-consistent outliers (real K caches look like this)
V = torch.randn(T, D)                 # values: no channel structure
q = torch.randn(D)

def qint8(x, axis=None):
    """symmetric absmax INT8. axis=None -> per-tensor, 0 -> per-channel, 1 -> per-token"""
    s = (x.abs().amax() if axis is None else x.abs().amax(dim=axis, keepdim=True)) / 127.0
    return torch.round(x / s).clamp(-127, 127) * s

def attn(Kq, Vq):
    logits = (Kq @ q) / D**0.5
    return torch.softmax(logits, dim=-1) @ Vq, logits

ref, ref_logits = attn(K, V)

for name, (Kq, Vq) in {
    "K+V per-tensor":             (qint8(K),         qint8(V)),
    "K per-channel, V per-token": (qint8(K, 0),      qint8(V, 1)),
    "V per-token only":           (K,                qint8(V, 1)),
    "K per-tensor only":          (qint8(K),         V),
}.items():
    out, lg = attn(Kq, Vq)
    print(f"{name:28s} out_err={(out-ref).norm()/ref.norm():.4f} "
          f"max_logit_err={(lg-ref_logits).abs().max():.3f}")

Measured output (numpy-equivalent run, same seed structure):

scheme	relative output error	max logit error
K+V per-tensor INT8	0.1049	0.862
K per-channel, V per-token	0.0192	0.099
V per-token only (K exact)	0.0070	0.000
K per-tensor only (V exact)	0.1044	0.862

Read the last two rows together: quantizing V alone costs 0.7%; quantizing K alone costs 10.4%. The combined error is essentially all K. And a max logit error of 0.86 means some token's attention weight is off by a factor of e^0.86 ≈ 2.4. If that token is the one holding your answer, retrieval fails — while the other 4,095 tokens absorb the difference invisibly.

Why do per-tensor scales waste bits on the K cache?

Because a symmetric per-tensor scale is set by the largest absolute value in the whole tensor, and in the K cache that value lives in a few fixed channels. In the simulation above, absmax / median|K| ≈ 130. With INT8 you have 127 levels spanning that range, so typical channels land on ~1–2 quantization levels — you paid for 8 bits and got fewer than 2 usable ones on the dimensions doing most of the work.

The outliers are channel-consistent: the same head dimensions are large in every cached token. That's exactly what per-channel (per-head_dim) scaling fixes, and it's why the row-2 error drops 5×. V caches don't show that structure, which is why per-token grouping is enough there.

FP8 softens this but doesn't remove it. E4M3 is floating point, so its relative precision (~3 mantissa bits, ≤6.25% step error) holds across binades instead of collapsing on small values — that's why FP8 KV usually behaves better than INT8 KV at the same bit width. What FP8 gives you is range, not precision: E4M3 saturates at ±448, E5M2 at ±57344 with only 2 mantissa bits. Two failure modes follow:

1. Uncalibrated scales. If your checkpoint carries no k_scale/v_scale, the runtime falls back to a unit scale and ±448 becomes a hard clipping threshold on raw key magnitudes. Clipped outlier channels are worse than coarsely quantized ones — you don't blur the logit, you delete the channel's contribution.
2. Calibration sets without long inputs. Scales fit on 512-token samples don't see the magnitude regime of a 100k-token prefill. Attention-sink tokens in particular carry keys with outsized norms; if calibration misses them, you clip exactly the positions the model uses to park unneeded attention mass.

Should you quantize keys pre-RoPE or post-RoPE?

Pre-RoPE is measurably easier to quantize, and nearly every stack does post-RoPE anyway. Standard implementations apply rotary embeddings before writing to cache, so the cached tensor is RoPE(K) — and RoPE mixes each channel with its pair partner by a position-dependent angle. A clean outlier channel becomes a rotating mixture of two channels, smeared over positions, so a per-channel scale fits worse the longer the sequence gets.

Quantizing pre-RoPE keys keeps the channel structure crisp but forces you to re-apply rotation after dequantization inside the attention kernel — extra work in the hottest loop. That's a real research result (KVQuant), not a config flag; treat it as a reason to expect post-RoPE per-channel scaling to underperform its theoretical ceiling, not as something to hand-roll in production.

There's a second reason production stacks stay per-tensor: layout. Paged KV caches are append-only, and a per-channel absmax changes every time a new token arrives with a bigger value in that channel — you'd have to rescale the entire history. The practical workaround (KIVI-style) is grouping along the channel axis within fixed-size blocks plus keeping the most recent N tokens in full precision. That residual window matters more than it looks: recent tokens get the highest attention weights, so unquantized recency buys accuracy cheaply.

Why doesn't perplexity catch KV cache quantization damage?

Because perplexity averages next-token loss over every position, and the overwhelming majority of positions are predictable from local context. Retrieval-critical positions — where the model must pull one specific fact from 40k tokens back — are a rounding error in that average. Worse, perplexity harnesses usually evaluate in short windows, which is precisely the regime where cache quantization is harmless.

Evaluate the failure mode you actually care about:

• Multi-needle retrieval at your real context length, not 4k. Vary needle depth; damage concentrates at mid-depth positions, not the ends.
• Per-layer attention KL between fp16 and quantized runs on the same prompt. Take the top-k attention indices per head and measure rank agreement. Middle layers with induction-style heads degrade first.
• Long-input tool calling / structured extraction, where one dropped token flips an argument value. This surfaces breakage well before a chat eval does.

How do you configure asymmetric KV cache quantization?

llama.cpp is the one common runtime that lets you set K and V independently — use it. Since V tolerates roughly an order of magnitude more error, spend your bits there:

# Good: keys stay at 8-bit, values go to 4-bit
./llama-server -m model.gguf \
  --flash-attn \                 # required for quantized V cache
  --cache-type-k q8_0 \
  --cache-type-v q4_0 \
  --ctx-size 131072

# Bad: symmetric, and K is the tensor that can't take it
#  --cache-type-k q4_0 --cache-type-v q4_0

vLLM exposes a single kv_cache_dtype for both tensors, so the lever there is calibration quality, not asymmetry:

# E4M3 + scales baked into the checkpoint by llm-compressor / AutoFP8
vllm serve meta-llama/Llama-3.1-70B-Instruct \
  --kv-cache-dtype fp8_e4m3 \
  --max-model-len 131072

Calibrate on samples that match your production length distribution. If you can't calibrate at all, fp8_e5m2 needs no scales but spends 3 of its 8 bits on range you'll never use — acceptable for a smoke test, not for a long-context product.

The memory case is genuinely strong, which is why this keeps getting enabled without review. Llama 3.1 70B: 80 layers × 8 KV heads × 128 head_dim × 2 (K and V) × 2 bytes = 320 KiB per token, or 40 GiB at 128k context for a single sequence. FP8 takes that to 20 GiB. That's the difference between two concurrent long-context requests and eight. Just don't pay for it in retrieval accuracy you never measured.

The short answer

FP8 and INT KV cache quantization break the key cache before the value cache because attention exponentiates key error and averages value error: a quantization error worth 0.86 in an attention logit shifts that token's attention weight by ~2.4×, while the same relative error on a value vector is diluted across thousands of positions. K caches also carry channel-consistent outliers 10–100× the median, so a per-tensor scale leaves typical channels with under 2 effective bits — which is why per-channel K scaling and per-token V scaling cut error roughly 5× in the simulation above. Perplexity won't show any of it, since the damage lands on long-range retrieval. Configure K conservatively and V aggressively (--cache-type-k q8_0 --cache-type-v q4_0 in llama.cpp), use calibrated k_scale/v_scale with fp8_e4m3 in vLLM, and validate with multi-needle retrieval at your production context length rather than a perplexity delta.