I changed nothing and my LLM server got 27% more expensive

Throttle — Sat, 26 Sep 2026 19:38:35 +0000

I ran the same cost check four times in a row against a server I didn't touch. Same model, same machine, same prompts, same settings. This is what came back, in dollars per million output tokens:

Run	$/M output tokens	95% CI	Change vs previous run
1	$8.77	$8.19 to $9.35
2	$8.86	$7.80 to $9.93	+1.0%
3	$11.26	$10.31 to $12.21	+27.1%
4	$10.02	$7.41 to $12.62	−11.0%

Run 3 says the server got 27% more expensive. Run 4 says it then got 11% cheaper. Nothing changed.

If I had changed a flag between run 2 and run 3, I would have blamed the flag and rolled it back. If I had changed one between run 3 and run 4, I would have posted an 11% saving in Slack. Both would have been wrong, and both would have looked like data.

This post is about why that happens and the rule I now apply before believing any before/after cost number. The tool in the output below is Throttle, an open-source CLI I'm building, but the rule doesn't depend on it.

The setup

A MacBook running Ollama locally with llama3.2:3b. Each check sent 3 blocks of 4 requests at concurrency 2, capped at 64 output tokens, temperature 0, with the same 8 built-in prompts and one warm-up request. All four checks finished within about 70 seconds of each other.

throttle check --url http://localhost:11434 --model llama3.2:3b \
  --gpu-hourly-rate 1.50 --label same \
  --blocks 3 --requests-per-block 4 --concurrency 2 --max-tokens 64

The $1.50/hour GPU rate. A laptop has no GPU invoice, so I picked and ran Throttle on runpod on different GPUa labels it ASSUMED everywhere it appears. Tokens and time are measured.

The cost math is simple:

$/M output tokens = hourly rate × wall-clock hours ÷ output tokens × 1,000,000

Every block in all four runs produced exactly 256 output tokens (4 requests × 64 tokens). So the only thing that moved the price was time. The three blocks of run 1 took 5.53 s, 5.24 s and 5.40 s. The blocks of run 3 took 7.09 s, 7.02 s and 6.65 s. Same tokens, about 30% more seconds, about 27% more dollars per token.

Why the same server swings

A cost measurement is a timing measurement, and timing on a real machine depends on everything else the machine is doing: other processes, background indexing, thermal state and clock speed, memory pressure, OS scheduling. On a shared cloud node you can add noisy neighbours and other tenants on the same host or network.

I didn't instrument the laptop during these runs, so I can't tell you which of those it was. That's the point. Most of the time you won't know either, and a single before/after pair can't tell load from a real change.

Why the confidence interval didn't save me

Each check reports a 95% confidence interval across its blocks. Look at runs 1 and 3: $8.19 to $9.35, then $10.31 to $12.21. The intervals don't overlap. The common rule of thumb ("non-overlapping intervals means a real difference") would call run 3 a real 27% regression.

The interval answers a narrower question than it looks like. It's computed from the blocks inside one run, so it describes the wobble over about 20 seconds. It can't see drift between runs, because every block in a run shares the same conditions. If the whole machine is 30% slower for a minute, all three blocks are slower together and the interval stays tight around the wrong number.

Measure the noise you are actually exposed to

The fix is to measure run-to-run noise directly: run the unchanged config a few times, and see how much whole checks disagree with each other.

Say the relative standard deviation between repeat checks of one config is SD. The difference between two single checks then has a standard deviation of √2 × SD, because each check carries its own noise. With only a handful of repeats you use Student's t instead of 1.96, with degrees of freedom from the repeats. That gives a 95% bound on how far apart two checks of the same config can land:

noise bound = t(0.975, df) × √2 × SD

Here, runs 1 to 3 were three checks of the unchanged config: $8.77, $8.86, $11.26. Their relative SD is 14.6%, with 2 degrees of freedom, where t is 4.303:

4.303 × 1.414 × 14.6% ≈ 89%

Throttle prints 89.1% because it uses the unrounded SD. Run 4 came in 11.0% below run 3. That's well inside ±89.1%, and the two intervals overlap as well, so this is the verdict:

before: $11.26/M output tokens (95% CI $10.31 to $12.21)
after:  $10.02/M output tokens (95% CI $7.41 to $12.62)
change: -11.0%
noise floor: calibrated: bound 89.1% (t x sqrt(2) x 14.6% run-to-run SD, 2 df)
Verdict: NO WINNER, the change (-11.0%) is not larger than the run-to-run
noise bound (89.1% = t x sqrt(2) x 14.6% run-to-run SD, 2 df); and the 95%
confidence intervals overlap, so the difference is within measurement noise

What about the +27.1% at run 3? Throttle didn't call it anything. With fewer than three earlier checks of the same config there's no noise estimate at all, so runs 2 and 3 both came back as:

Verdict: NOT CALIBRATED — run-to-run noise unknown. Two checks at different
times can differ from load alone. Run 'throttle check' at least 3 times without
changing anything (within 24 h) to measure your noise, or use 'throttle golden'
for a counterbalanced decision.

I built it that way because refusing to answer beats a confident wrong answer.

An 89% bound looks bad, and it's still the honest answer

A bound of ±89% means these three repeats can't resolve any realistic config change, and that's true. One slow run out of three blew the SD up, and with 2 degrees of freedom t is large. The bound is saying: on this machine, right now, you don't know enough to judge a change.

You can tighten it in two ways:

More repeats. t falls quickly with degrees of freedom: 4.30 at 2, 2.57 at 5, 2.23 at 10. More checks also give a better SD estimate, so one slow run matters less.
A quieter, longer measurement. Close what else is running, use longer blocks, and measure at the concurrency you run in production. Noise you remove is noise you don't have to bound.

For a decision you'll act on, like which config goes to production, don't compare two sequential runs at all. Interleave them (baseline, candidate, baseline, candidate, baseline, candidate) so drift hits both sides equally. I wrote up a real six-run example on an A100 in One vLLM flag, −68.6% cost per token.

What I do now

Run the unchanged config at least three times first, within the same day.
Change one thing.
Believe the change only if it's larger than the run-to-run noise bound and the 95% intervals don't overlap.
Otherwise it's NO WINNER. Don't write the Slack message.

Caveats

This was a laptop, a 3B model, a short workload and an assumed GPU rate. The size of the swing is specific to that machine on that morning. Your servers will have their own noise, which is exactly why you have to measure it rather than borrow mine.

These runs used Throttle 0.4.0, which repeats identical prompts, so the prompt cache was warm in all four runs. That rules out the cache as the cause of the swing, since it was the same in every run. Since 0.4.1, Throttle tags every request so a prefix cache can't make later checks look cheaper, and --warm-cache measures a warm cache on purpose.

Try it

pipx install throttle-pro
throttle demo      # no GPU needed; every number is SIMULATED

Then point throttle check at your own vLLM, SGLang or Ollama endpoint and run it three times before you change anything. If you get an interesting result, throttle check --share prints a sanitized summary you can post as a results issue. I'm collecting what real serving changes actually do to cost, and I read every one.

DEV Community: Throttle