A GPU hour has one price. What it delivers does not. Here is one measurement that shows how big the gap can be.
The setup. One rented H100 SXM 80GB on vast.ai at $2.689 per hour, 20 vCPUs. vLLM 0.27.1 (official image). Qwen2.5-0.5B-Instruct in fp16, unique prompts so nothing was cached. 28 September 2026. Every number below comes from vLLM's own /metrics counters, and they matched what my client recorded exactly.
Two loads, three rounds each.
- One request at a time: each request decoded at 648-649 tokens/s. The whole server produced 589-593 tokens/s. That is about $1.26 per 1M output tokens.
- 8 requests at once: each request got a bit slower, 549-573 tokens/s. But the whole server produced 3,194-3,348 tokens/s. That is about $0.22-0.23 per 1M output tokens.
Same GPU, same hour, same price. About 5.5x apart.
The common mistake. Many cost estimates take one request's speed and divide the hourly price by it. At 8 at once, one request ran at about 549-573 tokens/s, which would price the hour at about $1.30-1.36 per 1M tokens. The real cost was $0.22-0.23. In each round the whole server made 5.65-5.84x what one request suggests.
So pricing from one request's speed can overstate cost by about 5.7x at this load. At other loads the gap is different. That is the point.
What decides your real cost.
- How many requests your server actually runs at once, not what a benchmark ran.
- The idle time inside the hour. A GPU that waits still bills, so cost per token is the hourly price over everything the server produced in that hour, idle included.
- For scale-to-zero, the boot. On this same H100, one cold wake was about 75 of 77.3 billed seconds before the server could answer.
Same ratio on a smaller card. On a T4 with the same model and test, at 8 at once each request decoded at 122-124 tokens/s while the whole server made 701-708 tokens/s. Again about 5.7x.
What I can and cannot say. This is a small model, one GPU, three rounds per load. A 7B or 70B model, another server, or longer prompts will give different numbers. I am not saying your cost is $0.22. I am saying the number depends on your load, and you can only know it by measuring your own server.
The method, the raw numbers and the open-source tool are here, if useful: github.com/qaisermehdi3-coder/qvunex
If you have measured cost per token on your own servers at different loads, I would like to compare.
Top comments (0)