I measured four GPUs serving the same 7B model on vLLM. Which one is cheapest per token has no single answer. It flips with how fast your answers must be.
The setup
Same model (Qwen2.5-7B-Instruct), same vLLM (0.27.1, default flags), same load on every card: prompts of about 512 tokens, 256-token answers, 1 to 256 requests at once, 60 seconds per level. Four rented cards on vast.ai:
- RTX 3090 at $0.259/hour
- RTX 5090 at $0.522/hour
- A100 80GB PCIe at $1.017/hour
- H100 80GB SXM at $3.557/hour
The cheapest card at each answer time
Answer time here is p95: 95% of requests finished within it. For each card I took the cheapest load level that still met the time.
- Under 2 s: only the H100 can do it. $0.441 per 1M output tokens.
- Under 3 s: H100 $0.162, RTX 5090 $0.382, A100 $0.803. The 3090 can't. The H100 is about 2.4x cheaper than the 5090.
- Under 4 s: RTX 5090 $0.120, H100 $0.162, A100 $0.243. Now the 5090 wins, by about 1.4x.
- Under 8 s: RTX 5090 $0.050, H100 $0.099, A100 $0.106, RTX 3090 $0.490. The 5090 is about 2x cheaper than the H100.
The H100 is the most expensive card per hour, and still the cheapest per token when answers must be fast. It's the only card that keeps answers under 3 s with 64 requests in flight (p95 2.83 s), so it gets to run full while staying fast. The 5090 has to stay at 4 requests at once to answer that fast, so it uses only a small part of what it can do (379 tokens/s, against 2,901 at 64 at once).
Same load, different answer
At 64 requests at once on every card:
- RTX 5090: $0.050 per 1M, answers in 5.4 s (p50)
- RTX 3090: $0.082, answers in 17.9 s
- A100: $0.106, answers in 6.0 s
- H100: $0.162, answers in 2.6 s
Compare at the same load and the H100 looks like the worst deal. Compare at the same answer time and it can be the best one. Most GPU comparisons only show one of these.
The price you pay changes it again
All of the above uses one vast.ai price per card. Cost per token scales with the hourly price, so a different price can flip the result.
At another cloud's listed rates ($0.78/hour for the 5090, $2.65/hour for an H100 SXM), the same measurements give:
- Under 8 s: 5090 $0.075, H100 $0.074. About the same.
- Under 3 s: H100 $0.121, 5090 $0.571. The H100 is 4.7x cheaper.
So "the 5090 is cheaper per token" can be true at one price and false at another. The tool below lets you type in your own price per card.
How it was counted
Cost = GPU price x the measured window. The window runs until the last in-flight request finishes, because that time is paid for too. Requests and tokens came from vLLM's own /metrics before and after each level, and a separate client-side meter counted the same things. They matched exactly at every level on all four cards. 0 errors.
Limits
- These costs assume the GPU is busy the whole time. Idle hours in a real day raise them, and only real traffic shows by how much.
- One run per level, four different hosts on different days, synthetic prompts, one model.
- The 5090 and 3090 were run up to 64 requests at once, the A100 and H100 up to 256. So the 5090's cost at long answer times may go lower than shown.
- Nothing is interpolated between load levels. Only measured points are used.
Try it
Pick your answer time and type your own GPU prices: https://qaisermehdi3-coder.github.io/qvunex/
Every report, the raw /metrics and the server logs are here: https://github.com/qaisermehdi3-coder/qvunex/tree/main/benchmarks/corpus
If you have a GPU, one command runs the same test on it (about 10 minutes, synthetic prompts, nothing leaves your machine). Add your model and vLLM flags after the price to test your own setup:
curl -sL https://raw.githubusercontent.com/qaisermehdi3-coder/qvunex/main/planb.sh | bash -s -- YOUR_PRICE_PER_HOUR
Send me the result and your card can go in the public table with your name on it. If you'd rather not run anything, send your model, vLLM flags and typical prompt and answer length, and I'll run a short mini-report on a rented GPU for free. The full report on your own setup is paid work: qvunexaudit@gmail.com
Top comments (1)
The concurrency cliff on the 5090 matches what happens with agent traffic. Request arrival in agent workflows is naturally spiky because eight agents finish tool execution and hit the endpoint at the exact same moment.
On consumer memory buses, that arrival spike tanks time-to-first-token because full-prompt prefill starves ongoing decode iterations. Did you run default vLLM settings or test
--enable-chunked-prefill? Chunking incoming prompts prevents concurrent prompt arrivals from blowing past the three-second latency ceiling, which recovers some concurrency on smaller memory buses before the card drops off.