A small Python tutorial: compute a speed ceiling, then test a change against controls run before and after it, so drift can't pass for a win.
A note on how this was written: an AI writing assistant drafted it from the source listed at the end, and every claim was checked against that source before it went out.
At a glance
- A new agent skill for tuning local LLM servers is built around one habit: compute the ceiling, then test each change between two control runs.
- I rebuild that habit in a few small Python cells with fake data, including a placebo test that checks whether the method fools itself.
- The idea is useful even if you never touch a GPU. Any before and after speed test can lie in the same way.
A speed skill that is mostly about not fooling yourself
I expected the new inference-optimisation skill on GitHub to be a list of server flags. The README says plainly that it isn't. It describes a loop for making a local LLM server faster (decode, speculative decoding, prefill, KV cache, startup and time to first token), and most of that loop is about measuring.
The workflow it lists is short: compute the theoretical ceiling, change one thing at a time, measure against interleaved controls, log every result including the failures, gate on long-context retrieval, and qualify a clean, pinned build before publishing. A change is accepted only if it beats both neighbouring control runs and makes nothing else worse by more than 3%. Each run takes at least 6 samples and uses medians.
The results table is what made me read closely. On an Intel Arc B390 laptop iGPU, single-user decode went from 43.6 to 53 tok/s (median), prefill at 256K went from 297 to 628 tok/s (2.1×), and startup went from 113 to 43 s. Those are the author's numbers on their hardware. I can't check them. What I can do is build the measuring habit myself and see why it matters. That is this post.
Start with the ceiling, not the flags
The skill's first step is a ceiling. For decode, each new token has to read the model's active weights from memory, so memory bandwidth divided by the bytes read per token gives a rough upper limit. The README writes it as bytes per token against bandwidth.
# Rough decode ceiling: each new token reads the active weights once
bandwidth_gb_s = 115 # memory bandwidth of the machine
active_params_b = 3.0 # billions of weights read per token
bits_per_weight = 4 # quantization
bytes_per_token_gb = active_params_b * bits_per_weight / 8
ceiling = bandwidth_gb_s / bytes_per_token_gb
print(f'GB read per token: {bytes_per_token_gb:.2f}')
print(f'decode ceiling, 1 user: {ceiling:.0f} tok/s')
for target in (40, 60, 90):
print(target, 'tok/s is', f'{target / ceiling:.0%}', 'of the ceiling')
Output:
GB read per token: 1.50
decode ceiling, 1 user: 77 tok/s
40 tok/s is 52% of the ceiling
60 tok/s is 78% of the ceiling
90 tok/s is 117% of the ceiling
The inputs here are my own round numbers, and the formula leaves out the KV cache reads, so treat it as a sketch. Look at the last three lines: they tell you how far each target sits from the limit before you spend an afternoon on it.
The README shows why this matters. For the B390 run, the ceiling said 100 tok/s for one user was only reachable with two useful speculative drafts. The draft head accepted about 14% at depth 2, so the practical target became about 88 to 97 tok/s. The log says the run reached about 55% of that, and puts the rest down to GEMV kernel efficiency. Prefill at 256K reached 628 tok/s against a ceiling of about 670 to 684. Knowing the ceiling tells you when to stop.
Why a plain before and after test can lie
Here is the trap the interleaved controls guard against. Machines drift: they warm up, clocks change, a laptop goes from wall power to battery (the skill's harness flags runs on battery for exactly this reason). If you run the old setup first and the new one after, any drift lands on the new one.
import numpy as np
rng = np.random.default_rng(7)
def run_bench(true_speed, minute):
# the machine slows a little as it warms up, plus noise
drift = 1 - 0.004 * minute
return true_speed * drift * rng.normal(1, 0.02)
# naive plan: 6 control runs, then 6 candidate runs; the candidate changes nothing
ctl = [run_bench(50, m) for m in range(6)]
new = [run_bench(50, m) for m in range(6, 12)]
print('control median:', round(np.median(ctl), 1))
print('candidate median:', round(np.median(new), 1))
print('change:', f'{np.median(new) / np.median(ctl) - 1:+.1%}')
Output:
control median: 49.0
candidate median: 48.3
change: -1.5%
The candidate has the same true speed as the control. Look at the change line anyway. Whatever it shows is drift and noise, not the change you made. Flip the sign of the drift and the same nothing becomes a win.
Put the candidate between two controls
The fix is to run the control before and after the candidate, round by round, so the candidate always sits between two controls in time. A median over the rounds then pools 6 samples per setup, which smooths out single bad runs but also hides any pattern inside the run. Keep the raw samples too.
def interleaved(new_speed, ctl_speed=50, rounds=6):
before, cand, after = [], [], []
minute = 0
for _ in range(rounds):
before.append(run_bench(ctl_speed, minute)); minute += 1
cand.append(run_bench(new_speed, minute)); minute += 1
after.append(run_bench(ctl_speed, minute)); minute += 1
return np.median(before), np.median(cand), np.median(after)
b, c, a = interleaved(new_speed=50)
print('control before:', round(b, 1))
print('candidate: ', round(c, 1))
print('control after: ', round(a, 1))
Output:
control before: 47.9
candidate: 47.3
control after: 47.4
If the machine is drifting, the two control medians will differ from each other. That gap is your noise floor for this session. A candidate that lands between them hasn't shown anything yet.
A rule that also guards the other numbers
The skill's acceptance rule has two parts: beat both neighbouring controls, and regress nothing by more than 3%. The second part catches the common case where a decode tweak quietly costs you prefill speed or multi-user throughput.
def accept(cand, before, after, others):
beats_both = cand > before and cand > after
# others: {metric: (control, candidate)}, higher is better
worst = min(new / old - 1 for old, new in others.values())
return beats_both and worst > -0.03, worst
others = {'prefill_32k': (1037, 1050), 'decode_4_users': (84.7, 83.0)}
ok, worst = accept(c, b, a, others)
print('accepted:', ok, '| worst other change:', f'{worst:+.1%}')
Output:
accepted: False | worst other change: -2.0%
The skill also adds gates I left out here. One is a long-context needle retrieval check, so a speedup that breaks answers at long context doesn't get through. Another is counting tokens from the server's usage stats, never from streamed chunks.
Placebo test: does the method fool itself?
This is the part I'd want to see before trusting any benchmark method. Run a change that does nothing many times and count how often it wins. Then run a small real gain and count how often it is caught.
def naive(new_speed, ctl_speed=50):
ctl = [run_bench(ctl_speed, m) for m in range(6)]
new = [run_bench(new_speed, m) for m in range(6, 12)]
return np.median(new) > np.median(ctl)
def bracketed(new_speed):
b, c, a = interleaved(new_speed)
return c > b and c > a
rng = np.random.default_rng(1)
for label, speed in (('placebo', 50), ('real +3%', 51.5)):
n = sum(naive(speed) for _ in range(300))
i = sum(bracketed(speed) for _ in range(300))
print(f'{label:9} naive wins {n/300:.0%} bracketed wins {i/300:.0%}')
Output:
placebo naive wins 3% bracketed wins 28%
real +3% naive wins 63% bracketed wins 94%
Read the two rows side by side. In the placebo row, any win is a false alarm. In the real-gain row, a low number means the design missed a real improvement. A good method keeps the first row low and the second high. Look at which design does that once drift is in play.
Check it yourself
Run the cells in order. Then make one change: in run_bench, set drift = 1 (no drift) and run the placebo cell again.
Watch the naive column. With drift gone, its placebo and real-gain numbers should move. How far they move shows how much of its earlier answer came from drift and not from the change.
If you want a second test, my interleaved function always runs the 'before' control first in each round. Shuffle the order of the three runs inside each round and see if the bracketed numbers change. If they do, order was part of the result, and that is worth knowing before you trust a 'win' on real hardware.
What I'm still unsure about
My drift model is a straight line and my noise is a bell curve. Real machines throttle in steps and have bursts of noise, so the size of the gaps in my output says nothing about your setup. The habit carries over. The exact numbers don't.
I also can't say how much of the B390 gain came from the method and how much from the specific levers. The README links its full hill-climb log for that, and that log is what I'd read next.
I drafted this post with AI help and checked the code and every claim against the source myself.
When you last measured a speedup on your own setup, did you run the old config again afterwards, and did it still score the same?
Sources
Top comments (0)