vLLM FP8 vs bf16 on an H100: 1.4x the throughput, and what it did to GSM8K accuracy

vLLM can quantize a bf16 checkpoint to FP8 while it loads it: add --quantization fp8 and nothing else changes. On an H100 that should mean more tokens per second for the same money, but I wanted the actual numbers, and I wanted to know what it does to answer quality. So I rented one H100 on Runcrate today (2026-10-27), served the same 7B model in bf16 and then in FP8, ran the same load and the same 300 math questions against both, and terminated the machine as soon as the results were on disk.

Short version: FP8 gave 1.40x the output throughput at 64 concurrent requests (6,147 against 4,401 tokens/s) and 1.55x the speed for a single user. On GSM8K it answered 270 of 300 questions right against 272 for bf16.

Setup

• GPU: one H100 PCIe (80 GB), US Midwest, $3.86 per hour on demand at launch time.
• Model: Qwen/Qwen2.5-7B-Instruct, the regular bf16 checkpoint for both modes. FP8 here is vLLM's online quantization: weights are converted to FP8 at load time with a per-tensor scale, activations are scaled dynamically. The KV cache stays bf16 in both modes.
• Server: vLLM 0.11.0 (torch 2.8.0+cu128, transformers 4.57.6), --dtype bfloat16 --max-model-len 4096 --gpu-memory-utilization 0.90, plus --quantization fp8 for the FP8 run. Fresh server per mode.
• Throughput: vllm bench serve, random prompts, 512 tokens in and exactly 256 out (--ignore-eos), concurrency 1, 8, 32 and 64 (16, 64, 192 and 320 prompts), seed 42. Client on the same machine.
• Quality: the first 300 questions of the GSM8K test set, zero-shot, greedy (temperature=0), up to 768 output tokens, final number parsed from a #### <answer> line, 64 requests in flight.

The commands

Launch (the instance_type_id comes from GET /api/v1/instances/types):

curl -s https://www.runcrate.ai/api/v1/instances \
  -H "Authorization: Bearer $RUNCRATE_API_KEY" -H "Content-Type: application/json" \
  -d '{"name":"fp8-h100","instance_type_id":"<id from /instances/types>",
       "ssh_key_id":"<your key id>","max_runtime_minutes":60}'

On the machine:

curl -LsSf https://astral.sh/uv/install.sh | sh && export PATH=$HOME/.local/bin:$PATH
uv venv --python 3.12 /opt/env && . /opt/env/bin/activate
uv pip install vllm==0.11.0 'transformers<4.58' pandas pyarrow --torch-backend=cu128

M=Qwen/Qwen2.5-7B-Instruct
for MODE in bf16 fp8; do
  EXTRA=""; [ $MODE = fp8 ] && EXTRA="--quantization fp8"
  vllm serve $M --dtype bfloat16 $EXTRA --max-model-len 4096 \
    --gpu-memory-utilization 0.90 --port 8000 &
  until curl -sf localhost:8000/v1/models >/dev/null; do sleep 1; done
  for C in 1 8 32 64; do
    case $C in 1) N=16;; 8) N=64;; 32) N=192;; 64) N=320;; esac
    vllm bench serve --backend vllm --model $M --dataset-name random \
      --random-input-len 512 --random-output-len 256 --ignore-eos \
      --num-prompts $N --max-concurrency $C --seed 42 \
      --save-result --result-dir results --result-filename bench_${MODE}_c$C.json
  done
  python gsm8k.py $M > results/gsm8k_$MODE.json   # 300 questions, greedy, OpenAI client
  kill %1; wait
done

gsm8k.py is 40 lines: it loads openai/gsm8k from the Hugging Face Hub, sends each question with the instruction "Solve this math word problem step by step. On the last line write only the final answer as '#### '.", and compares the last number with the reference.

Results

Memory, as reported by vLLM at startup:

Mode	Weights in GPU memory	KV cache	KV cache size	Server ready after
bf16	14.25 GiB	51.27 GiB	959,936 tokens	66.5 s
FP8	8.14 GiB	57.38 GiB	1,074,320 tokens	67.5 s

Throughput and latency:

Concurrency	Mode	Output tok/s	Per-user tok/s	Median TTFT	p99 TTFT	$ per 1M output tokens
1	bf16	104	105	26 ms	28 ms	10.33
1	FP8	160	162	21 ms	25 ms	6.70
8	bf16	766	99	119 ms	141 ms	1.40
8	FP8	1,173	153	88 ms	109 ms	0.91
32	bf16	2,490	84	163 ms	502 ms	0.43
32	FP8	3,709	130	221 ms	375 ms	0.29
64	bf16	4,401	77	106 ms	939 ms	0.24
64	FP8	6,147	108	98 ms	755 ms	0.17

"Per-user tok/s" is 1000 / median time per output token. Cost per 1M output tokens is hourly price / (output tokens per second x 3600) x 1,000,000, counting generated tokens only.

GSM8K, first 300 test questions:

Mode	Correct	Accuracy	Wall time (64 in flight)
bf16	272 / 300	90.7%	15.3 s
FP8	270 / 300	90.0%	12.1 s

The two modes gave the same final answer on 272 of 300 questions. bf16 alone was right on 10 questions, FP8 alone on 8.

What surprised me

1. The biggest gain was for one user, not for the full batch. At concurrency 1, FP8 decoded at 162 tokens/s against 105 (1.55x). Single-stream decoding reads every weight once per token, so halving the bytes of the weights goes almost straight into speed. At 64 users the gain was 1.40x, still large, but the batch is more compute bound there.
2. Startup cost nothing extra. The online conversion added about a second to a 67 s server start. There is no separate FP8 checkpoint to find or trust.
3. The quality cost was within noise on this test. Two questions out of 300 is less than the disagreement between the modes in both directions (10 against 8). The final answer still differed on 28 questions, so FP8 does change outputs; it just did not make them worse on average here.
4. The freed memory mostly went to KV cache, and this test did not need it. About 6 GiB more cache (+12% tokens) matters with long contexts and many users; with 768-token requests and 64 users the cache was never full in either mode. The throughput gain above comes from faster matmuls and fewer bytes read, not from bigger batches.
5. The p99 TTFT at concurrency 32 was better with FP8 but the median was worse (221 ms against 163 ms). One run per setting; I would not read anything into that.

Caveats

• One model (7B), one GPU, one run per setting. Larger models, where weights take most of the memory, may gain more from FP8; smaller ones less.
• GSM8K with 300 questions has a standard error of about 1.7 points at 90% accuracy. It shows FP8 did not break math reasoning on this model; it cannot show a difference under about 3 points.
• This is online FP8 (per-tensor). Pre-quantized FP8 checkpoints with calibrated or per-channel scales, and FP8 KV cache (--kv-cache-dtype fp8), are different settings and were not tested.
• Ada and Hopper GPUs (L40S, RTX 6000 Ada, H100, H200) have FP8 tensor cores. On older cards vLLM falls back to weight-only FP8 kernels, and the speed math changes.
• Price is the on-demand price at launch time and moves daily.

Reproduce it

1. Create an API key and add an SSH key (POST /api/v1/ssh-keys).
2. Pick an H100 from GET /api/v1/instances/types and launch with the curl above, max_runtime_minutes set as a safety net.
3. Run the block above, copy results/ back, delete the instance.

The whole run, including install, model download, both server starts and both benchmarks, was billed $0.58.

If you want to run it yourself: runcrate.ai

Written by the Runcrate team, with AI assistance. The numbers come from the run described above; raw logs are kept with the post.