vLLM on three GPUs: Qwen2.5-7B tokens per second and cost per 1M tokens on RTX A6000, L40S and H100

#vllm #llm #gpu #benchmark

I wanted one number before picking a GPU for a small self-hosted model: what does a million generated tokens cost on each card, once the server is actually busy? Spec sheets don't answer that, so I rented three single-GPU machines on Runcrate this morning (2026-10-26), ran the same vLLM benchmark on each, and terminated them as soon as the numbers were on disk.

Setup

Model: Qwen/Qwen2.5-7B-Instruct, bf16, no quantization.
Server: vLLM 0.11.0 (torch 2.8.0+cu128, transformers 4.57.6), --max-model-len 4096 --gpu-memory-utilization 0.90, everything else default.
Load: vllm bench serve with random prompts, 512 tokens in and exactly 256 tokens out (--ignore-eos), at concurrency 1, 8, 32 and 64 (16, 64, 192 and 320 prompts), seed 42. The client ran on the same machine as the server, so there is no network in these numbers.
Machines (on-demand price at launch time):

Runcrate GPU	VRAM	Region	$/hour
RTX A6000	48 GB	Des Moines, US	0.74
L40S	48 GB	Kansas City, US	1.14
H100 PCIe	80 GB	Montreal, CA	3.55

Cost per 1M output tokens below is simply hourly price / (output tokens per second x 3600) x 1,000,000. It counts only generated tokens; the 512 prompt tokens per request are free riders in this math.

The commands

Launch (one call per GPU; the instance_type_id comes from GET /api/v1/instances/types):

curl -s https://www.runcrate.ai/api/v1/instances \
  -H "Authorization: Bearer $RUNCRATE_API_KEY" -H "Content-Type: application/json" \
  -d '{"name":"vllm-a6000","instance_type_id":"<id from /instances/types>",
       "ssh_key_id":"<your key id>","max_runtime_minutes":45}'

max_runtime_minutes is the safety net: the machine stops itself even if my script dies. Then, on the machine:

curl -LsSf https://astral.sh/uv/install.sh | sh && export PATH=$HOME/.local/bin:$PATH
uv venv --python 3.12 /opt/env && . /opt/env/bin/activate
uv pip install vllm==0.11.0 'transformers<4.58' --torch-backend=cu128

vllm serve Qwen/Qwen2.5-7B-Instruct --dtype bfloat16 --max-model-len 4096 \
  --gpu-memory-utilization 0.90 --port 8000 &

for C in 1 8 32 64; do
  case $C in 1) N=16;; 8) N=64;; 32) N=192;; 64) N=320;; esac
  vllm bench serve --backend vllm --model Qwen/Qwen2.5-7B-Instruct \
    --dataset-name random --random-input-len 512 --random-output-len 256 --ignore-eos \
    --num-prompts $N --max-concurrency $C --seed 42 \
    --save-result --result-dir results --result-filename bench_c$C.json
done

Then DELETE /api/v1/instances/<id> and a check that the list is empty.

Results

GPU	Concurrency	Output tok/s	Per-user tok/s	Median TTFT	p99 TTFT	$ per 1M output tokens
RTX A6000	1	42	43	82 ms	85 ms	4.86
RTX A6000	8	307	40	289 ms	529 ms	0.67
RTX A6000	32	1,012	32	479 ms	1,940 ms	0.20
RTX A6000	64	1,727	32	259 ms	3,347 ms	0.12
L40S	1	48	49	48 ms	49 ms	6.56
L40S	8	354	45	161 ms	299 ms	0.89
L40S	32	1,082	34	279 ms	1,050 ms	0.29
L40S	64	1,887	33	146 ms	2,169 ms	0.17
H100 PCIe	1	104	105	25 ms	26 ms	9.47
H100 PCIe	8	776	100	62 ms	125 ms	1.27
H100 PCIe	32	2,566	87	233 ms	410 ms	0.38
H100 PCIe	64	4,515	79	101 ms	960 ms	0.22

"Per-user tok/s" is 1000 / median time per output token, i.e. how fast one stream feels to the person reading it.

How long each machine took before it could serve anything (measured on the machine):

GPU	vLLM install	Model download (15 GB)	`vllm serve` until ready	First chat reply
RTX A6000	29 s	103 s	114 s	1.22 s
L40S	7 s	78 s	67 s	0.85 s
H100 PCIe	4 s	13 s	54 s	0.56 s

What surprised me

The L40S is barely faster than the A6000 at decoding. It has roughly twice the bf16 compute on paper, but single-stream decode on a 7B model is limited by memory bandwidth, and the two cards are close there. 48 vs 42 tokens/s at concurrency 1, 1,887 vs 1,727 at 64. Per dollar, the A6000 wins at every load level I tried.
The H100 is not the per-token deal, it is the latency deal. It produced 2.6x the A6000's throughput at concurrency 64 for 4.8x the hourly price. But every user still saw 79 tokens/s at that load, against 32 on the other two cards, and the p99 time to first token stayed under a second. If your product is a chat UI with real people waiting, that is what you are paying for.
At concurrency 1, every GPU is expensive. $4.86 to $9.47 per million tokens. Batching is where the money is: going from 1 to 64 concurrent requests cut the cost per token by about 40x on all three cards. If your traffic can't keep the batch full, a smaller, older card idles more cheaply.
p99 TTFT at concurrency 64 on the A6000 was 3.3 s, while the median was 259 ms. Some requests wait for a whole batch slot. Look at tail latency before you take the cost-per-token number at face value.
Setup time varied more than benchmark time. The 15 GB model download took 13 s on one machine and 103 s on another. For short experiments, that download is a real part of the bill; put the weights on a volume if you will start the server more than once.

Caveats

One run per GPU, on one morning. Treat differences under about 10% as noise.
Random prompts, fixed 512/256 shape. Real chat traffic has prefix reuse and varied lengths; prefix caching would help all three cards.
These are PCIe cards, one per machine. No tensor parallelism, no FP8, no speculative decoding. An FP8 checkpoint would likely change the H100 math more than the others.
Prices are the on-demand prices at launch time on 2026-10-26 and move daily.

Reproduce it

Create an API key and add an SSH key (POST /api/v1/ssh-keys).
Pick an instance type from GET /api/v1/instances/types and launch with the curl above, max_runtime_minutes set.
Run the setup and benchmark block, copy results/ back, delete the instance.

The three machines, including setup time, were billed $1.07 together ($0.15 for the A6000, $0.29 for the L40S, $0.64 for the H100).

If you want to run it yourself: runcrate.ai

Written by the Runcrate team, with AI assistance. The numbers come from the run described above; raw logs are kept with the post.

DEV Community