DEV Community: Throttle

The FP8 trap: my GPU bill dropped 47% because the model was printing "!!!!!!"

Throttle — Tue, 06 Oct 2026 03:02:25 +0000

I had one hour on an AMD MI300X and one question: what does a token actually cost on it?

One GPU, vLLM's ROCm build, Qwen2.5 at 7B, 32B and 72B. Thirty-two requests in flight, 256 output tokens max, priced at the $2.99/GPU-hr list rate. Every number below is wall-clock time × that rate ÷ tokens produced.

The baselines came out clean. Four repeat checks of each untouched config, so I'd know how much the machine wobbles on its own:

7B: $0.227 per million output tokens
32B: $0.77
72B: $1.67, on a single GPU, which still surprises me a bit

The 72B repeats agreed to within 0.4%. That's a very quiet machine.

The obvious next move

FP8 halves the bytes per weight. vLLM will do it for you on load with one flag:

vllm serve Qwen/Qwen2.5-72B-Instruct --quantization fp8

Numbers:

32B: $0.77 → $0.449 (−41%)
72B: $1.67 → $0.872 (−47%)

Three checks each, all well outside the noise. If I'd stopped there I would have written a very different post, and I'd have been telling you to cut your bill in half.

The thing that didn't fit

I wasn't looking for a problem. I was reading the per-block lines and something was too neat.

At temperature 0, with the same 32 prompts, BF16 32B wrote about 6,900 output tokens per block. FP8 wrote 8,192 on every block. 8,192 is 32 × 256: every request ran all the way to the cap.

Models don't suddenly get more talkative because you changed the weight format. So I asked both servers the same six questions.

"Summarize the causes of World War I in about 80 words."

BF16: an 89-token answer about nationalism, alliances and an archduke.

FP8:

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

…for all 256 tokens.

It was cheaper per token because it was producing junk tokens very quickly. The cost number was accurate. It was just measuring the wrong thing.

What was real

Same idea, done properly: checkpoints that were quantized ahead of time and published (RedHatAI's Qwen2.5-*-Instruct-FP8-dynamic). This time I checked the answers first.

They were right. 17 × 23 = 391. "Où est la gare ?" Sensible summaries. Output length matched BF16 within about 1% (72B: 6,361 vs 6,328 tokens per block).

Then the cost, three checks each:

Model	BF16	FP8 (pre-quantized)	change
7B	$0.227/M	$0.185/M	−18%
72B	$1.67/M	$1.13/M	−32%

Noise bound under 3% on 7B and under 1% on 72B. So the real answer is about a third off a 72B on one MI300X. That's a smaller number than 47%, and you can actually use it.

The rule I'm keeping

A cost-per-token number on its own can't tell a cheaper model from a broken one. Before you trust a saving, check that the model is still doing the same work:

Output length. If average tokens per answer jumps, or suddenly everything hits max_tokens, stop. Nothing else needed.
A handful of fixed prompts at temperature 0. Read them. Takes a minute.
Repeat the baseline. If you don't know how much your server wobbles untouched, you can't tell a 5% win from Tuesday.

Number 1 is the cheap one, and the data for it is already sitting in your benchmark output. I'm building it into the tool I used here: if output length shifts between a baseline and a change, it says OUTPUT CHANGED instead of CHEAPER. That's coming in the next release.

Caveats, honestly

One session, one workload: short prompts, 256 output tokens, concurrency 32. Long-context or long-output traffic will behave differently.
On 72B, the FP8 server ran on the VM's second GPU and the baseline on the first. Same card model, not literally the same card.
I only sampled the broken on-the-fly 72B by output length, not by reading answers. On-the-fly FP8 on 7B was fine (6 of 6 answers correct, −18%).
Six prompts is a sanity check, not an accuracy benchmark. If FP8 matters to your product, run your own evals.
Prices are the provider's list rate. Tokens and time are measured.

I don't know yet whether the on-the-fly FP8 behaviour is specific to this ROCm image or to larger Qwen2.5 models. If you've seen it, tell me.

Measured with Throttle, an open-source CLI that prices your inference in $/M tokens with confidence intervals and refuses to call a change a win inside your own noise. pipx install throttle-pro.

I rented an A100 to test one vLLM flag

Throttle — Thu, 01 Oct 2026 03:01:41 +0000

I rented an A100 for under an hour to answer one question.

Does a single vLLM flag really change what a token costs?

The flag was max_num_seqs: how many requests the server works on at once. I set it to 1, which is a deliberately bad setting, and then to 8. Same GPU, same model (Qwen2.5-0.5B), same prompts, eight requests in flight the whole time.

The obvious way to test this is: run the old setting, run the new one, compare. I don't trust that anymore. I once watched a server I hadn't touched "get" 27% more expensive between two runs. GPUs warm up, other work comes and goes, and a single before/after quietly bakes all of that into your answer.

So I interleaved it. Six runs, alternating the order:

baseline → change → baseline
change → baseline → change

If the machine drifts during the session, the drift lands on both sides instead of making one look better.

Here's what came back, in dollars per million output tokens at $1.39/hr:

max_num_seqs=1   $0.748   $0.746   $0.745
max_num_seqs=8   $0.237   $0.229   $0.237

Six runs, two tight clusters, no overlap. About 68% cheaper per token, and it held every single time.

Two honest caveats:

A baseline of 1 is unrealistically bad. Nobody should run production like that. The point was to prove the method, not to promise you 68%.
It's a 0.5B model. Bigger models, longer prompts and real traffic will give you a different number. That's exactly why you measure your own.

What surprised me most wasn't the drop. It was how calm the numbers became once the runs were interleaved. Same server, same day, no drama.

I turned this into a small open-source tool so I'd stop doing it by hand. It measures cost per million tokens with confidence intervals, and it only says CHEAPER when a change actually beats the noise:

pipx install throttle-pro

All six raw runs are public, if you want to check my math.

The raw artifacts · Throttle on GitHub · throttle-pro.com

What does one token actually cost you?

Throttle — Thu, 01 Oct 2026 02:56:10 +0000

I rent GPUs by the hour.

For a long time I couldn't tell you what a single token cost me. I knew the hourly rate, and I knew the API providers' price sheets. The number in between, my cost per million tokens on my server, I was guessing.

Turns out almost everyone self-hosting is guessing too.

The math itself is one line:

$ per million tokens = ($/hr ÷ 3600) ÷ tokens per second × 1,000,000

The hard part is the "tokens per second". It moves with everything: batch size, a vLLM flag, a quantization, the engine version, the GPU, even what else the machine is doing.

So I started measuring it properly. A few things surprised me.

The same server doesn't give the same number twice. I ran one check four times on an Ollama box I hadn't touched. It came back $8.77, $8.86, $11.26, $10.02 per million tokens. Had I changed a setting between runs three and four, I'd have happily told you it saved 11%.

One number is not a measurement. You need repeat runs, a confidence interval, and a noise floor. If a change doesn't beat the noise, the honest answer is "no winner".

When a change is real, it's obvious. On an A100 with vLLM, moving max_num_seqs from 1 to 8 took output cost from $0.746 to $0.234 per million tokens, holding across six interleaved runs. That baseline was deliberately bad. The point isn't the 68%; it's that you can finally see it.

Caching quietly lies. Send the same prompts twice and a prefix cache makes run two look cheaper than it is. Every request needs to be cold unless you're measuring the cache on purpose.

I got tired of doing this by hand, so I built a small tool that does it: measure, change one thing, measure again, and tell me CHEAPER, MORE EXPENSIVE or NO WINNER. It refuses to call a winner it can't prove, which turned out to be the whole point.

It's open source and runs on your machine:

pipx install throttle-pro
throttle check --url http://localhost:8000 --model <your-model> --gpu-hourly-rate <your $/hr>

Run it three or four times before you change anything. The first surprise is usually how much your own server wobbles.

If you're self-hosting and you've made a change you're not sure paid off, I'd genuinely like to hear about it.

Throttle on GitHub · throttle-pro.com

I changed nothing and my LLM server got 27% more expensive

Throttle — Sat, 26 Sep 2026 19:38:35 +0000

I ran the same cost check four times in a row against a server I didn't touch. Same model, same machine, same prompts, same settings. This is what came back, in dollars per million output tokens:

Run	$/M output tokens	95% CI	Change vs previous run
1	$8.77	$8.19 to $9.35
2	$8.86	$7.80 to $9.93	+1.0%
3	$11.26	$10.31 to $12.21	+27.1%
4	$10.02	$7.41 to $12.62	−11.0%

Run 3 says the server got 27% more expensive. Run 4 says it then got 11% cheaper. Nothing changed.

If I had changed a flag between run 2 and run 3, I would have blamed the flag and rolled it back. If I had changed one between run 3 and run 4, I would have posted an 11% saving in Slack. Both would have been wrong, and both would have looked like data.

This post is about why that happens and the rule I now apply before believing any before/after cost number. The tool in the output below is Throttle, an open-source CLI I'm building, but the rule doesn't depend on it.

The setup

A MacBook running Ollama locally with llama3.2:3b. Each check sent 3 blocks of 4 requests at concurrency 2, capped at 64 output tokens, temperature 0, with the same 8 built-in prompts and one warm-up request. All four checks finished within about 70 seconds of each other.

throttle check --url http://localhost:11434 --model llama3.2:3b \
  --gpu-hourly-rate 1.50 --label same \
  --blocks 3 --requests-per-block 4 --concurrency 2 --max-tokens 64

The $1.50/hour GPU rate. A laptop has no GPU invoice, so I picked and ran Throttle on runpod on different GPUa labels it ASSUMED everywhere it appears. Tokens and time are measured.

The cost math is simple:

$/M output tokens = hourly rate × wall-clock hours ÷ output tokens × 1,000,000

Every block in all four runs produced exactly 256 output tokens (4 requests × 64 tokens). So the only thing that moved the price was time. The three blocks of run 1 took 5.53 s, 5.24 s and 5.40 s. The blocks of run 3 took 7.09 s, 7.02 s and 6.65 s. Same tokens, about 30% more seconds, about 27% more dollars per token.

Why the same server swings

A cost measurement is a timing measurement, and timing on a real machine depends on everything else the machine is doing: other processes, background indexing, thermal state and clock speed, memory pressure, OS scheduling. On a shared cloud node you can add noisy neighbours and other tenants on the same host or network.

I didn't instrument the laptop during these runs, so I can't tell you which of those it was. That's the point. Most of the time you won't know either, and a single before/after pair can't tell load from a real change.

Why the confidence interval didn't save me

Each check reports a 95% confidence interval across its blocks. Look at runs 1 and 3: $8.19 to $9.35, then $10.31 to $12.21. The intervals don't overlap. The common rule of thumb ("non-overlapping intervals means a real difference") would call run 3 a real 27% regression.

The interval answers a narrower question than it looks like. It's computed from the blocks inside one run, so it describes the wobble over about 20 seconds. It can't see drift between runs, because every block in a run shares the same conditions. If the whole machine is 30% slower for a minute, all three blocks are slower together and the interval stays tight around the wrong number.

Measure the noise you are actually exposed to

The fix is to measure run-to-run noise directly: run the unchanged config a few times, and see how much whole checks disagree with each other.

Say the relative standard deviation between repeat checks of one config is SD. The difference between two single checks then has a standard deviation of √2 × SD, because each check carries its own noise. With only a handful of repeats you use Student's t instead of 1.96, with degrees of freedom from the repeats. That gives a 95% bound on how far apart two checks of the same config can land:

noise bound = t(0.975, df) × √2 × SD

Here, runs 1 to 3 were three checks of the unchanged config: $8.77, $8.86, $11.26. Their relative SD is 14.6%, with 2 degrees of freedom, where t is 4.303:

4.303 × 1.414 × 14.6% ≈ 89%

Throttle prints 89.1% because it uses the unrounded SD. Run 4 came in 11.0% below run 3. That's well inside ±89.1%, and the two intervals overlap as well, so this is the verdict:

before: $11.26/M output tokens (95% CI $10.31 to $12.21)
after:  $10.02/M output tokens (95% CI $7.41 to $12.62)
change: -11.0%
noise floor: calibrated: bound 89.1% (t x sqrt(2) x 14.6% run-to-run SD, 2 df)
Verdict: NO WINNER, the change (-11.0%) is not larger than the run-to-run
noise bound (89.1% = t x sqrt(2) x 14.6% run-to-run SD, 2 df); and the 95%
confidence intervals overlap, so the difference is within measurement noise

What about the +27.1% at run 3? Throttle didn't call it anything. With fewer than three earlier checks of the same config there's no noise estimate at all, so runs 2 and 3 both came back as:

Verdict: NOT CALIBRATED — run-to-run noise unknown. Two checks at different
times can differ from load alone. Run 'throttle check' at least 3 times without
changing anything (within 24 h) to measure your noise, or use 'throttle golden'
for a counterbalanced decision.

I built it that way because refusing to answer beats a confident wrong answer.

An 89% bound looks bad, and it's still the honest answer

A bound of ±89% means these three repeats can't resolve any realistic config change, and that's true. One slow run out of three blew the SD up, and with 2 degrees of freedom t is large. The bound is saying: on this machine, right now, you don't know enough to judge a change.

You can tighten it in two ways:

More repeats. t falls quickly with degrees of freedom: 4.30 at 2, 2.57 at 5, 2.23 at 10. More checks also give a better SD estimate, so one slow run matters less.
A quieter, longer measurement. Close what else is running, use longer blocks, and measure at the concurrency you run in production. Noise you remove is noise you don't have to bound.

For a decision you'll act on, like which config goes to production, don't compare two sequential runs at all. Interleave them (baseline, candidate, baseline, candidate, baseline, candidate) so drift hits both sides equally. I wrote up a real six-run example on an A100 in One vLLM flag, −68.6% cost per token.

What I do now

Run the unchanged config at least three times first, within the same day.
Change one thing.
Believe the change only if it's larger than the run-to-run noise bound and the 95% intervals don't overlap.
Otherwise it's NO WINNER. Don't write the Slack message.

Caveats

This was a laptop, a 3B model, a short workload and an assumed GPU rate. The size of the swing is specific to that machine on that morning. Your servers will have their own noise, which is exactly why you have to measure it rather than borrow mine.

These runs used Throttle 0.4.0, which repeats identical prompts, so the prompt cache was warm in all four runs. That rules out the cache as the cause of the swing, since it was the same in every run. Since 0.4.1, Throttle tags every request so a prefix cache can't make later checks look cheaper, and --warm-cache measures a warm cache on purpose.

Try it

pipx install throttle-pro
throttle demo      # no GPU needed; every number is SIMULATED

Then point throttle check at your own vLLM, SGLang or Ollama endpoint and run it three times before you change anything. If you get an interesting result, throttle check --share prints a sanitized summary you can post as a results issue. I'm collecting what real serving changes actually do to cost, and I read every one.