When you type «nvfp4 vs» into Google it completes with «fp8», «gguf», «q4_k_m» and «bf16». The usual answer compares different models quantized by different people. On October 16 NVIDIA released Nemotron Labs Dory 1.0 4B in BF16, NVFP4, MXFP8 and GGUF at once, so I measured the same model in every format on an RTX 4070 Ti SUPER (Ada, 16 GB, no FP4 hardware), with the rules written down before the first run.
| Format | Engine | Generate (tok/s, batch 1) | Read 2048-token prompt (tok/s) | Weights |
|---|---|---|---|---|
| BF16 | vLLM | 51.0 | 7,003 | 7.69 GiB |
| FP8 (per tensor, W8A8) | vLLM | 78.5 | 10,118 | 4.67 GiB |
| MXFP8 | vLLM | 81.4 | 6,061 | 4.76 GiB |
| NVFP4 | vLLM | 116.8 | 6,978 | 3.28 GiB |
| GGUF Q5_K_M | llama.cpp | 58.8 | 1,710 | 3.42 GiB (file) |
- Among the vLLM formats, generation follows weight size. NVFP4 was 2.3 times faster than BF16. On Ada it runs through a weight-only Marlin kernel: the weights stay small, the math stays in 16 bits.
- Reading a long prompt, FP8 wins, because it is the only one of these formats whose multiplications Ada does in 8 bits.
- Quality, full GSM8K (1319 questions, 8192-token cap): BF16 1249, FP8 1241, MXFP8 1250, NVFP4 1238. All three tie with BF16 under the ±3-point rule; their 95 % intervals stay within ±2 points.
- Token by token it is another story. On fixed English and Spanish texts NVFP4 picks BF16's next token 89-90 % of the time, against 94-96 % for FP8 and MXFP8, while perplexity barely moves. MXFP8 is the most faithful to BF16.
The post also covers what broke before measuring (the llama.cpp converter rejects the public checkpoint, a prefix cache that inflated prefill, truncated reasoning being scored as answers) and a mistake the audit caught in my own first fidelity run.
Read the full measurements, the pre-registered sheet and the scripts → https://efraingaray.com/en/blog/nvfp4-vs-fp8-gguf-ada/
Top comments (2)
did the NVFP4 runs ever hit the 16 GB ceiling, and if so how did the scheduler handle swapping?
It never ran out of memory, by design: vLLM reserves memory up front (
gpu_memory_utilization0.85, 13.24 GiB) and gives whatever the weights don't use to the KV cache. With NVFP4 the weights took 3.28 GiB, which left 8.61 GiB of KV cache: 40,232 tokens, or 4.6 full-length requests at the 8,704-token limit. BF16 got 4.31 GiB (20,128 tokens, 2.3 requests). There's no swapping to CPU in this vLLM build: when KV blocks run out, the scheduler preempts a running request, frees its blocks and recomputes it later. I didn't log preemptions, so I can't tell you how often that happened in the 1,319-question run. The speed numbers in the post are at batch 1, where none of this comes into play.