Every post so far has been about flags: flags that died, flags that did nothing, flags that flipped. This one is about the uncomfortable truth on the other side of all that tuning: on fixed hardware, the biggest speed wins in this project didn't come from configuration at all. They came from what the model is and what the server can do with it.
Ranked by measured impact on my dual Tesla P40s:
1. Sparse MoE: 41 tokens/sec on 2016 GPUs
The single largest lever, by a wide margin. Gemma 4 26B-A4B is a mixture-of-experts model: 26B parameters on disk, about 3.8B active per token. My dense 27B models decode at 8.5 to 20 tokens/sec depending on config. The MoE runs 41.
Nothing about my hardware changed. The model simply does less work per token, by design. Sparse activation bought more throughput than every flag decision in this series combined, and it's not close. If your hardware is old and your workload tolerates the model family, MoE is the first question to ask, not the last.
2. MTP speculative decoding: +57% single-stream
The daily-driver stack (27B dense, Q6_K) measured 8.46 tokens/sec single-stream after the row-split removal. Enabling MTP speculative decoding, where the model's own multi-token-prediction layers draft ahead and the main pass verifies, lifted that to about 13.3 tokens/sec in a controlled A/B. Acceptance rates ran 0.38 to 0.63, and output correctness was verified against non-speculative runs.
Free speed is rare. This is the closest thing to it I found: no quality cost, no extra VRAM of consequence, one flag, 57%.
3. Parallel slots: aggregate throughput nearly doubles
Same stack, measured at 1, 2, and 4 concurrent slots: 8.46 / 12.8 / 15.0 tokens/sec aggregate. Per-request latency degrades gently until the slots saturate, which means one swapped-in model can serve several concurrent agents acceptably instead of queuing them.
For an agent workload, aggregate is the number that matters. My traffic is dozens of tool-call round-trips, not one human reading one stream. Four slots turned "one user at a time" into "the whole agent fleet."
The stacking effect
These compose. MoE where the family fits, MTP and parallel slots where it doesn't. The result across the fleet: the post-row-split era ended up faster than the row-split era it mourned. The 12-14 tokens/sec I lost to a deleted flag came back as 15 aggregate with better concurrency, and the fast path tripled it.
There's a budget lesson in that. I spent months on flags worth 10 to 40% each, some of which later turned out to be dead code or expired rules. The architecture-level choices were worth 100 to 400%, and they were sitting in plain sight the whole time: pick a sparser model, turn on the decoding feature the model ships with, let the server batch.
Tune the flags, but tune them last. On fixed hardware, architecture is the knob with the range.
Top comments (0)