A sequel to the $2,000 Inference Server series: a 125B model on two Tesla P40s, stuck at 17 tokens per second for a week, and the community bug report that found the problem in one line.
The machine from the first post in this series is still the same machine. An EPYC 7302, 128 GB of DDR4, and two Tesla P40s that cost $403 for the pair on eBay. Pascal, compute capability 6.1, no Tensor Cores, no NVLink. The newest silicon in it is from 2016.
In October I put a 125 billion parameter model on it.
That sounds like it shouldn't fit, and on paper it doesn't. Qwen3.8-Flash-Next is a mixture-of-experts model: 125B parameters in total, but only about 6B of them do work on any given token. The engine I ran it on, Strata, keeps the dense weights and attention on the GPUs, pins the 43 GB expert arena in host RAM, and caches the experts that actually get used in VRAM on each card. On this box, 93% of the experts end up resident in VRAM, and 99.9% of expert lookups at decode time hit that cache. Almost nothing crosses PCIe.
Credit where it belongs: Strata is Niko's project (Niko1221), open source under the MIT license. It is the reason a model this size runs on cards this old at all, and nothing in this post is a fix of mine. Every speedup below came from Strata's own releases, and the regression at the center of it was found by its community and fixed by its maintainer.
The bar it had to clear was the daily driver it was meant to replace, a dense 27B model at 23 tokens per second.
For a week it ran at 17.
The ceiling at 17
From October 10 to October 16, Strata 0.1.40.3 decoded at 17 to 18 tokens per second. The same model on upstream llama.cpp ran at 21.8 to 23.2. A model with a fifth of the active parameters per token was losing to a dense one, and losing to a different engine running the same weights.
There were three good explanations. Each one got ruled out by a measurement, not an argument.
The split mode. Every fast llama.cpp stack I run uses row split. On these cards it beat layer split 12 to 14 tokens per second against 7. That's the flag we tuned around, and upstream deleted it on July 6. Support for this model landed on August 27. No build that loads this model has row split. The replacement, tensor split, isn't implemented for this architecture and crashes on Pascal. Strata only does layer split and closed tensor parallelism as not planned. A backport of row split onto an older tree was on the table, and I stopped it: row split never split the expert tensors in the first place, and the old tree would have lacked this model's draft layer and a Pascal KV cache fix. It would have been slower than the llama.cpp stack by construction. Not built.
The container. Maybe Docker and Portainer were costing something. Same engine binary, same config, same prompts, on bare metal: 16.6 to 17.0 tokens per second, against 17.8 in the container. Identical placement. The container costs nothing on this box.
The configuration. Five A/B tests over two days. Conversation cache, pipeline windows, stage trimming, batch slots, context length from 32k to 128k to 262k, the sampling block. Some were worth keeping for other reasons. The conversation cache alone turned a 38 second follow-up turn into 1.5 seconds. None of them moved single-stream decode.
So it wasn't the hardware, the container, the multi-GPU mode, or the config. When three independent explanations each fail a controlled measurement, the remaining variable is the one you didn't think to vary.
Here that was the engine version.
A community number, to the decimal
The evidence that found it wasn't a profiler. It was a bug report.
Strata issue #1469, filed by paulhothersall: a Tesla P40, CUDA 12.9, the same IQ3_XXS quantization, a 4k prompt. Decode at 36.4 tokens per second on v0.1.40, 17.7 on v0.1.40.2, and 36.8 on v0.1.40.4. A second reporter on a GTX 1080 and 1070 went from about 18 to 31.7 on 0.1.41.
My 17 to 18 matched their regressed number almost exactly.
The report didn't stop at the number. It bisected the slowdown to a single commit and named the line that gave most of it back. Niko pushed the fix about 75 minutes after the issue was filed. I confirmed the mechanism in my own clone of the source. One commit had removed __restrict__ from the activation pointers in the small-batch matrix kernels, and added a prefetch that Pascal can't overlap with compute. Every decode step on compute capability 6.1 took about twice as long. The fix, "keep __restrict__ and drop the PDL prefetch below sm_70," was not in my build. On newer cards the two versions compile to byte-identical machine code, which is why almost nobody but Pascal owners ever saw it.
That's the lesson I'd hand anyone running old cards: a community number twice your own on the same hardware is a version question before it is a hardware question. Check the engine's issue tracker and release notes before you benchmark bare metal. I did those in the other order. The bare metal run came first, and all it proved was that the container was innocent.
One tag
The fix was a checkout. Same Dockerfile, same entrypoint, same run config, one version tag: 0.1.40.4. The build script got versioned image tags in the same change, so the old image stayed on disk as the rollback.
| 1,024-token answers at a 4k prompt | 0.1.40.3 | 0.1.40.4 |
|---|---|---|
| Decode, tokens per second | 17.5, 18.0, 18.1 | 35.4, 32.6, 33.2 |
| Prefill at 4k, tokens per second | 321 | 332 |
| Resident experts | 22,964 | 22,964 |
| Correctness, 20 fixed greedy prompts | 16 identical, 0 wrong |
Twice the decode rate, same placement, same answers.
Hold the defaults
The next step was the newest release, 0.1.44.1. It added real Pascal work: wider dequantization kernels, a BF16 fix, a compile fix for older CUDA. It also changed two defaults, and one of them is documented as "changes output."
An engine upgrade that changes your answers and your speed at the same time can't be measured, because you can't tell which change bought what. The engine's own docs for older GPUs say to pin both new defaults off when comparing builds. So the trial ran with both held off and nothing else changed.
| 1,024-token answers at a 4k prompt | 0.1.40.4 | 0.1.44.1, defaults held |
|---|---|---|
| Decode, tokens per second | 35.4, 32.6, 33.2 | 36.0, 35.7, 40.3 |
| Prefill at 256 / 4k / 32k | 152 / 332 / 623 | 173 / 356 / 653 |
| Correctness vs 0.1.40.4 | 14 identical, 0 wrong |
About 8% more decode, 5 to 13% more prefill, same placement. Kept.
Decode is flat from a 256-token prompt to a 32k prompt, 36.9 down to 35.6. Prefill rises with prompt size, from 173 to 653 tokens per second. That's the opposite of a dense model's curve, and it's the expert-streaming design showing through: attention cost grows with context, expert cost doesn't.
The knob that raised acceptance and not speed
Another community report, a benchmark PR from gun-powder, showed 35 tokens per second on two P40s with two tuning knobs: a stricter gate on the model's own draft tokens, and a fixed share of PCIe traffic. Strata takes both as per-request overrides, so the A/B ran on the live stack with no restart, arms rotated each round.
The stricter gate won the sweep: a median of 40.2 against 36.5, acceptance up from 0.61 to 0.79. So I deployed it and measured again. The long pass came back at 36.5, 41.3 and 35.3, under the bar I'd set to keep it. Acceptance rose exactly as the sweep predicted. Tokens per second didn't follow. A stricter gate makes every draft window surer and shorter, and on this box the two cancel out. Reverted. The fixed PCIe share cost about 4 tokens per second in every round, because with 99.9% of lookups already in VRAM there's nothing for it to govern. Left on auto.
A per-request knob is the cheapest A/B there is. Use it to screen, then deploy the winner and measure again before you believe it. Three lucky runs are not a result.
Apples to apples
The old daily driver had one more test in its gate: two clients at once. Strata runs one sequence unless you tell it otherwise, and a per-stream rewording of that row would have passed a capability the new stack didn't have. The row stayed as written. Two batch slots went into the config, and two direct streams now finish together in 33 seconds where the single-sequence engine took 53. The cost was about 1,300 fewer resident experts and a cache hit rate that moved from 99.9% to 99.5. Nothing visible. Kept.
With that, every row of the gate passed on its original wording. Strata is the engine.
Twice as fast, and it doesn't look it
The live proof was a 13,083-token conversation turn in Open WebUI. The model generated 13,327 tokens at a sustained 38.9 tokens per second for six minutes. About 11,700 of those tokens were the model thinking before it typed a word.
That's worth saying plainly, because it's what anyone sitting in front of the chat window actually sees. Twice the decode rate on a 10,000-token think is still minutes before visible text, and then the text arrives in bursts of one to four tokens per draft window. The decode rate is the engine's. The thinking length is the model's. Judge the engine by its own log line, not by the typing.
What the week was for
The whole gain came from one tag change. Everything before it was spent proving, by measurement, what the ceiling wasn't. Everything after it was worth about three tokens per second.
That sounds like a wasted week, and it isn't. A one-line fix you can't explain is a fix you can't trust, and every one of those dead ends is now a row in a table with a date and a directory behind it. The next time something on this box runs at half the speed someone else gets on the same card, the first stop is the issue tracker.
125 billion parameters, two cards from 2016, 36 to 40 tokens per second. It took knowing the machine in front of me, not the one on the spec sheet, and reading someone else's bug report closely enough to see my own number in it. Thanks to Niko for building Strata and fixing it fast, and to the P40 owners who wrote their numbers down where the rest of us could find them.
Top comments (0)