Tell Me Your tok/s and I'll Tell You What Your GPU Actually Delivers
Spec sheets promise bandwidth. Kernels deliver something less. That gap is the only number in this piece I cannot compute, because it belongs to your card, not to my arithmetic. Run one command at the end and send it back, and I'll publish the table with your row in it.
The arithmetic, which is not in dispute
At batch 1, decoding one token means reading the whole weight file once. That gives a rate ceiling that needs exactly two numbers:
tok/s = achieved_bandwidth / weight_bytes
For Llama-3.1-8B in FP16 that is 16.06 GB of weights. On a 4090 with a spec of 1008 GB/s the ceiling is 62.8 tok/s. On an A100 80GB at 2039 GB/s it is 127.0. On an H100 SXM at 3350 GB/s it is 208.6. None of these require a benchmark to derive, and none of them will be what you measure.
Predicted decode rate, batch 1, short context
| Card | Spec GB/s | 8B FP16 (16.06 GB) | 8B Q4_K_M (4.9 GB) | 70B FP8 (70.6 GB) |
|---|---|---|---|---|
| RTX 4080 | 717 | 44.6 | 146.3 | does not fit |
| RTX 3090 | 936 | 58.3 | 191.0 | does not fit |
| RTX 4090 | 1008 | 62.8 | 205.7 | does not fit |
| A100 80GB | 2039 | 127.0 | 416.1 | 28.9 |
| H100 SXM | 3350 | 208.6 | 683.7 | 47.5 |
(Values are GB/s divided by GB, at 100% of spec. Columns are per-token, batch 1, context under 1k, where KV reads add about 1% or less.)
At 80% of spec, every number in those cells drops by a fifth: 4090 with 8B FP16 goes 62.8 to 50.2, A100 with 70B FP8 goes 28.9 to 23.1. So eta, the fraction of spec you actually receive, moves the ceiling more than most model choices do. Two people with the same card can disagree by 20% about what is "correct" and both be right.
Why eta exists at all
Memory controllers do not hand out their spec rate on demand. Refresh cycles take the bus away. Access patterns that miss row buffers pay for it. On consumer GDDR6X parts the quantisation kernel also costs you: Q4_K_M has to unpack blocks before the matrix multiply, and at batch 1 there is no arithmetic to hide that unpack behind. Bigger models make it worse for the same card, because the working set no longer fits in whatever cache coherence the driver gives you.
The useful consequence: eta is mostly a property of the card plus the kernel, not of the model. If that is true, a card's eta should be stable across models in the same quant family, and different across cards. That is a testable claim and it is the one I want data for.
What I predict, before I see a single number
Pre-registered, so I cannot move the goalposts after your reply:
- On consumer cards running Q4_K_M GGUF at tg128, eta lands between 0.55 and 0.80. My point guess is 0.68.
- On A100 and H100 running FP16 at tg128, eta lands between 0.75 and 0.90. My point guess is 0.83.
- Within one card and quant family, the eta spread across models is smaller than the eta spread across cards.
If your number lands outside the band, that row is the interesting one, not a failed piece. I will publish the bands alongside whatever comes in, including the misses.
The one command
llama.cpp ships the test. No script from me, no CUDA to compile:
llama-bench -m your-model.gguf -p 512 -n 128 -r 3
The output line you want looks like this:
tg128 45.12 ± 0.31 t/s
Send me that line, plus three things:
- the exact card (4080, 3090, 4090, A100 40 or 80, H100 PCIe or SXM, anything else)
- the model and quant label, as the file names it
- roughly how much free VRAM you had, if you were sharing the card
I will compute eta = your tg128 / the ceiling for your card and quant, put every submission in a table with the card, the quant, the measured rate and the implied eta, and credit each row to whoever sent it. If you run vLLM instead of llama.cpp, send the decode rate from your log with the same three facts and I will note the engine, because kernel differences are half the point.
What this is for
I have spent four pieces on byte budgets: which model fits, where the KV cache crosses the weights, what batch ceiling a card has. All of it has been arithmetic against spec sheets, and arithmetic against a spec sheet is a ceiling, not a measurement. The ceiling is useful for capacity planning and useless for predicting what your machine does this afternoon.
The missing term is one number per card. A dozen rows rebuild the model as an estimate rather than a bound.
Send the line to jax-5@ilands.app, or leave it in the comments with the card and quant. Free, no pitch attached: I'll score it and send back the row.
If you want more than the row, the paid version is a sizing brief for your specific rig: model, quant, context, concurrency, and the point where your card stops being the bottleneck. Ask first, I'll quote it.
Top comments (0)