The same H100 hour cost $1.26 or $0.22 per million tokens

Qaiser Mehdi — Tue, 13 Oct 2026 00:46:48 +0000

A GPU hour has one price. What it delivers does not. Here is one measurement that shows how big the gap can be.

The setup. One rented H100 SXM 80GB on vast.ai at $2.689 per hour, 20 vCPUs. vLLM 0.27.1 (official image). Qwen2.5-0.5B-Instruct in fp16, unique prompts so nothing was cached. 28 September 2026. Every number below comes from vLLM's own /metrics counters, and they matched what my client recorded exactly.

Two loads, three rounds each.

One request at a time: each request decoded at 648-649 tokens/s. The whole server produced 589-593 tokens/s. That is about $1.26 per 1M output tokens.
8 requests at once: each request got a bit slower, 549-573 tokens/s. But the whole server produced 3,194-3,348 tokens/s. That is about $0.22-0.23 per 1M output tokens.

Same GPU, same hour, same price. About 5.5x apart.

The common mistake. Many cost estimates take one request's speed and divide the hourly price by it. At 8 at once, one request ran at about 549-573 tokens/s, which would price the hour at about $1.30-1.36 per 1M tokens. The real cost was $0.22-0.23. In each round the whole server made 5.65-5.84x what one request suggests.

So pricing from one request's speed can overstate cost by about 5.7x at this load. At other loads the gap is different. That is the point.

What decides your real cost.

How many requests your server actually runs at once, not what a benchmark ran.
The idle time inside the hour. A GPU that waits still bills, so cost per token is the hourly price over everything the server produced in that hour, idle included.
For scale-to-zero, the boot. On this same H100, one cold wake was about 75 of 77.3 billed seconds before the server could answer.

Same ratio on a smaller card. On a T4 with the same model and test, at 8 at once each request decoded at 122-124 tokens/s while the whole server made 701-708 tokens/s. Again about 5.7x.

What I can and cannot say. This is a small model, one GPU, three rounds per load. A 7B or 70B model, another server, or longer prompts will give different numbers. I am not saying your cost is $0.22. I am saying the number depends on your load, and you can only know it by measuring your own server.

The method, the raw numbers and the open-source tool are here, if useful: github.com/qaisermehdi3-coder/qvunex

If you have measured cost per token on your own servers at different loads, I would like to compare.

Ready is not warm: one SGLang cold start on a T4

Qaiser Mehdi — Mon, 12 Oct 2026 22:51:26 +0000

I measure what GPU time actually buys. One number surprised me this month, so here it is with the raw timeline.

The setup. One T4 on Colab. SGLang 0.5.21. Qwen2.5-0.5B-Instruct in fp16. I started a clock just before launching the server, the way a scale-to-zero platform starts billing when a worker is created. Then 8 short chat requests, then 60 seconds of health checks, then stop. One run, 5 October 2026.

The timeline.

0 s: server launched (billing starts)
~371 s: prefill CUDA graph capture finished (it took 287.7 s)
386.7 s: /health returns 200, log says "ready to roll"
~563 s: first chat request finishes (it took about 176 s)
~579 s: second request finishes (about 16 s), the other six within the same second
639.6 s: billing stops

After that, 8 more requests on the same server took 0.08 to 0.22 s each.

What this means for the bill. If you split the bill at the health check, boot looks like 60% of it (386.7 of 639.6 s). But the first user got an answer about 563 s in. That is 88% of the bill gone before anyone was served.

So "ready" was not the end of the wait. It was about two thirds of it.

What I can and cannot say.

The server's own counters matched my client exactly: 8 requests, 264 prompt tokens, 80 output tokens.
The log does not say what the first 176 s were spent on, so I am not claiming a cause.
Each health check in the tail shows up in the SGLang log as a one-token prefill on the GPU, but the request counters do not move. So "the GPU did work" and "a request was served" are different things.
For comparison, on vLLM 0.27.1 on the same card, the 8 requests after ready took 5.8 s in total. Boot there was about 165 of 171 billed seconds.
This is one run on one small card. A bigger GPU or another model can look different. That is the point: measure your own.

If you run scale-to-zero, record three times, not one:

billing start to health check OK
billing start to the first finished request
the idle tail after the last request

Then price the whole billed time, not just the part where requests were running.

The method and the open-source tool I used are here, if useful: github.com/qaisermehdi3-coder/qvunex

If you have measured this on other cards or servers, I would like to see your numbers.

DEV Community: Qaiser Mehdi

The same H100 hour cost $1.26 or $0.22 per million tokens

Ready is not warm: one SGLang cold start on a T4