DEV Community

prodbymarcu
prodbymarcu

Posted on

An inference speedup has to beat its own error bar

Four levers measured against a live decode lane on my 3090 rig, medians in tokens per second

Every local-inference thread I have read this year ends with someone claiming a percentage. Nobody shows the spread, and half of those claims are measuring noise, that is what an afternoon of tuning on my own hardware proved. A tuning win needs four things before the number means anything: a median, an error bar wide enough to see, proof the changed flag actually loaded into the running process, and a stability gate where a crash at load is a discard. I got all four working on a live coding lane of my dual-3090 Arch box on November 17, 2026, and shipped nothing to production. The ledger came back empty of wins, and that empty result is more useful than a 3% headline would have been.

The protocol came from a stranger's repo

I cloned inference-optimisation-skill on Nov 17 and ran its methodology on my own lanes, because a benchmark row only counts when the next person can reproduce it. Its rules: six reps per arm, median after a warmup run, one 2048-token prompt against 256 completion tokens, a stability gate demanding zero errors, no OOM, no hang and VRAM under cap, plus a ceiling estimate computed BEFORE any tuning, so a lever only gets tried when the measured read-rate says there is headroom. Every arm lands in a ledger as baseline, candidate or cut, with the reason written next to it.

Two pins mattered here. Never park a lane just to bench an API, the lanes were serving real work and I was not going to blind them for an hour. And a config edit needs a full service restart, ready in 16 to 21 seconds, which sets the floor on how cheap a swap can be.

Proof the change loaded is the trap

At 13:15 I stopped the supervisor and pkilled the engine tree, then launched an A/B run. Numbers printed. Every single one of them was garbage. The manager respawned the live engine seconds later on its own recorded command line, so both arms ran the same binary and the diff measured nothing but sampling noise. That is the failure mode nobody shows a screenshot of, a harness that reports a delta when the delta never existed.

The guard I wrote after: parking goes through the manager's own model swap, never through stop plus pkill, and each arm prints its resolved config into the ledger row before comparison means anything. Some process put both lanes back at 13:15:27, I never confirmed which caller did it, and I did not need to, the lanes came back and the bogus numbers stayed out of the ledger. Two agent turns inside that window ended with Your request was not processed, fallback providers empty, so the box refused those requests on the spot and nothing answered them from a substitute model. Failing loud is what saved me a day of chasing phantom numbers.

Ceiling math said the levers were allowed to matter

Before tuning I read the model file itself. 256 experts, 10 of them used per token, 49 MoE layers, 25.146 GB of expert weights plus a 1.475 GB dense pool, 24 heads with 2 kv, indexer top-k 2048. The measured decode pulls about 2.46 GB per token, which at 84.55 tok/s works out near 208 GB/s effective off one GDDR6X card. The provisional ceiling table, assuming an optimistic 400 GB/s, says 159 tok/s is reachable with speculative drafts at four context slots, so I was running at 53% of that number and spec or cache work had room to pay off. Same script's prefill and MFU columns were unreliable on this IQ1_M dequant path, so I dropped those rows before writing any conclusion.

Measured read rate against the assumed bandwidth ceiling on the IQ1_M lane

Four levers tried, zero banked

Baseline stock args gave samples 83.86 and 84.44, median 84.44 warm, plus one cold artifact at 65.09 that I kept in the row because it shows what a bad sample looks like next to a good one.

Arm A, --spec-min-p from 0.70 down to 0.60: three samples at 86.66, 86.34 and 83.63. Median +2.6%, spread straddles the baseline line, verdict candidate, not banked. That is the honest shape of most claims you will see online, a real direction that survives its own error bar about half the time.

Arm B, --mtp-q4 projection: the engine crashed at load. Stability gate discard, no number reported for it, because publishing "it broke" as a speed figure is how tuning folklore gets written.

Arm C, --lookup-chain 4: 80.34, down 4.2%. Arm D, --spec raised from 4 to 6: 80.64, down 3.8%. Two slower arms out of four tried, which means the stock config was already close to the local optimum this harness can see.

I also pinned five known dead ends so nobody re-litigates them on this box: STRATA_KV_PREFETCH, STRATA_PF_FUSED, --prefill-head-alias, MAP_HUGETLB, --peer-device (108 threads measured below a single card's worth), plus conv-cache slots 4 with mib 4096, which taxes small reads. A ledger that records cuts, with reasons, is what makes a second afternoon cheaper than a first one.

What an empty ledger buys you

Nothing shipped. The other lane never got touched and production configs stayed exactly as they were, arm A lives in the file as a candidate for a future re-test against agent-shaped prompts, which is where the traffic actually looks like my 2048-token fixture anyway. Verification after teardown: end-to-end completions through both routers came back 200 at 1.03s and 0.72s, the stale marker got cleared before I removed it, the watchdog timer showed active, and the two paused crons went back on the list.

The claim survives, "a speedup has to beat its own error bar" is what the afternoon tested, and the answer for four levers was no. Anyone telling me tuning folklore works on their box is welcome to prove it.

Show me your spread

Post your median AND your worst sample, plus the resolved config line for each arm. If the delta sits inside the spread between those two numbers, label it a candidate and go re-measure, that is the honest headline. Which lever do you think deserves another afternoon before I bank it, --spec-min-p at 0.60 with three samples straddling baseline, or nothing at all?

Top comments (0)