Researched October 2026. Sizing math from public model cards and the vLLM docs; not my own benchmarks.
In my last post I showed that a 128K context window can push a 7B model from 17 GB to 33 GB of VRAM — and that you have three moves, in order of cost-effectiveness: quantize the cache, pick a GQA-heavy architecture, or rent the bigger card. This post is about move #1, the cheapest one. It halves the KV cache with a single flag.
The One Flag
In vLLM:
vllm serve meta-llama/Llama-3.1-8B-Instruct \
--kv-cache-dtype fp8
That's it. The weights stay in FP16/BF16 — untouched. Only the KV cache is stored as 1 byte per value instead of 2. Per the vLLM docs, fp8 maps to E4M3, and there are three calibration options: no calibration (scales default to 1.0), random-token warmup, or dataset calibration via llm-compressor (recommended for max accuracy).
Why FP8 Works for KV Cache
Weight quantization gets all the attention, but KV values are arguably the easier target: within a single attention head, their values live in a narrow, well-behaved range. E4M3's tighter range and 3-bit mantissa match that distribution well; E5M2's wider range is mostly wasted on values that never appear, and its 2-bit mantissa visibly degrades attention scores that compound across layers. In other words, the format choice matters — fp8 (E4M3) is the right default, not fp8_e5m2.
Where the Math Flips
Llama 3.1 8B, FP16 weights (~16 GB), 2 GB overhead:
| Context | FP16 cache → total | FP8 cache → total |
|---|---|---|
| 32K | 4 GB → ~22 GB ✅ | 2 GB → ~20 GB ✅ |
| 64K | 8 GB → ~26 GB ❌ | 4 GB → ~22 GB ✅ fits 4090 |
| 128K | 16 GB → ~34 GB ❌ | 8 GB → ~26 GB ❌ still over |
Honest note: at 128K, FP8 cache alone doesn't save Llama 3.1 8B on a 24 GB card — 26 GB is still 26 GB. Your options there: pair it with FP8 weights (8 + 8 + 2 ≈ 18 GB ✅), or pick the GQA-heavy architecture from last post — Qwen2.5 7B at 128K needs ~7 GB of FP16 cache, ~3.5 GB in FP8, for a total around 19 GB ✅. The flag is powerful, not magic; the arithmetic still rules.
Three Caveats Before You Ship It
1. Hardware matters. The biggest wins are on Hopper GPUs with native FP8 support. On older cards, check your vLLM version's kernel support — the memory savings are real, but the compute path may not be as optimized.
2. Calibrate if quality matters. Default scales of 1.0 are fine to start, but for production, dataset calibration (llm-compressor) exists for a reason. Per-head quantization — finer-grained than per-tensor — needs the FlashAttention backend plus that calibration path.
3. Measure retrieval, not just perplexity. Standard benchmarks show small degradation, but long-context tasks with exact-recall demands (needle-in-a-haystack style document QA) are where you'd notice it first. Run your own eval before and after.
What This Costs in Practice
The whole point: quantization keeps you on the cheap card. A 64K-context 7B workload that needed an 80 GB card at ~$1.50+/hr in FP16 fits on a 24 GB 4090 at ~$0.35/hr with FP8 cache — same model, same context, roughly a quarter of the hourly cost, for one CLI flag.
I built a free GPU Advisor that walks through these tradeoffs (model size, context, quantization tolerance) and recommends the cheapest fitting setup. No signup.
Have you tried FP8 KV cache in production — and did the quality hold up on your workload? Curious where it breaks in practice.
Top comments (1)
Quantizing the KV cache is such a game changer for long-context work — the memory savings are way bigger than people expect.