My LLM tuner picked a configuration that ran almost four times faster. I had passed a flag to limit quality loss, and the setting appeared in the saved profile.
Then I checked whether the quality gate had actually run.
It hadn’t.
A guard checked for prompts before the file-based workload had loaded them. The probe was never created. Nothing had compared the answers. The flag recorded my intention while the search carried on without enforcing it.
That was one of several assumptions that broke when I benchmarked PolyServe, the open-source LLM tuner I’ve been building.
The idea is straightforward: give it a model and sample traffic, let it measure serving configurations on your hardware, then serve its pick through an OpenAI-compatible API. It searches across vLLM, SGLang, and llama.cpp, with options for weight precision, batching, KV cache, and speculative decoding.
I wanted to find out which settings helped. I also found out which parts of my own reasoning needed fixing.
1. I thought a quality gate could tell me whether faster answers were still good
Fixing the missing probe exposed another bug.
A precision rejected in an early stage could still become the final winner. I had removed it from further tuning, but the final selection still considered every successful trial.
After fixing that, I could finally evaluate the idea behind the gate.
It compared generated answers with a reference precision. If the answers changed too much, it would reject the cheaper configuration.
On 48 prompts, comparing the first 32 generated tokens, the same fp8 weights disagreed with themselves on about 10% of answer openings across two runs. Greedy decoding did not make the batched execution bit-for-bit reproducible.
Other precisions disagreed with the reference on 83–100% of openings.
That sounds disastrous until you grade the answers. A separate GSM8K check found a much smaller accuracy difference. Exact text disagreement was a poor substitute for task correctness.
I withdrew the rejection gate. The current --quality-probe reports output drift; it does not claim to certify accuracy. Task quality needs a separate, labelled evaluation.
A safety setting needs evidence that it ran, enforced its decision, and measured something meaningful. I had problems at all three levels.
2. My default search space excluded the fastest configuration
On an RTX 4090 with 24 GB of memory, I tested Qwen2.5-14B-Instruct.
Calibration used 300 Dolly prompts. Evaluation used 300 different prompts, with three interleaved runs per configuration.
Allowing pre-quantized checkpoints expanded the search space:
| Configuration | Output tokens/s | Measured J/token |
|---|---|---|
| Previous fp8 pick, re-measured | 365 | 0.751 |
| Selected GPTQ 4-bit configuration | 1,412 | 0.261 |
That’s roughly 3.9× the throughput, with about 65% less measured energy per token.
This compares complete configurations. The winner combined 4-bit weights with different context, memory, cache, and decoding settings. It does not isolate a 3.9× effect from changing weight precision alone.
It also came with a quality question:
| Precision | Correct on 200 GSM8K problems |
|---|---|
| fp8 | 190 |
| GPTQ 4-bit | 186 |
Four answers changed from correct to incorrect; none changed in the other direction.
That is an observed two-percentage-point drop on a small sample, with substantial uncertainty. It cannot establish a universal quality cost.
PolyServe’s default --quant auto excludes these 4-bit checkpoints because they can hurt quality. The fastest configuration was outside the space I had asked it to search.
That default remains deliberate. But users should know what performance they might be excluding and what evaluation they need before accepting the trade-off.
3. A multi-GPU result didn’t generalize to another setup
An earlier experiment with a 3B model on two PCIe-connected A40s made tensor parallelism look unattractive.
Splitting the model across both GPUs was slower than using one. Replicas did better.
Then I measured a 32B model on two NVLinked A100s:
| Setup | Output tokens/s at eight concurrent users |
|---|---|
| Best tuned single A100 | 172.6 |
| Tensor parallel across two A100s | 295.8 |
Tensor parallelism delivered 71% more throughput than the tuned single-GPU configuration.
Both the model and hardware changed, so this does not isolate NVLink’s contribution. It does show why the A40 result was too narrow to turn into a general rule.
There is another limit: the time budget expired before replicas were measured on the A100 pair.
I can report a win over one GPU. I cannot claim tensor parallelism was the best two-GPU layout.
Even individual settings reversed direction. In the A100 session, an fp8 KV cache reduced vLLM throughput by about 4% while increasing SGLang throughput by about 8%.
The hardware name alone wasn’t enough to predict the winning settings.
4. My staged search didn’t consistently beat random sampling
This was the comparison I most needed to run.
PolyServe searches in stages. It prunes configurations that should not fit, tries decoding strategies, tunes other settings, and re-measures contenders.
Would random sampling from the same filtered space find an equally good configuration?
Often, yes.
On the smaller 14B search space, staged search won: 385 versus 369 tokens/s, with similar time budgets and non-overlapping ranges across three repeats.
That is about 4.4%, on one card and one random seed. A draft-model compatibility bug also wasted a small part of the random baseline’s budget, so this result needs replication.
After expanding the space to include more quantized checkpoints, random search reached 1,416 tokens/s against PolyServe’s 1,412. Their measured ranges overlapped.
Random search also used less time: 2,524 seconds against 3,028.
One win in the smaller space. One tie in the larger one. The earlier 8B experiments also failed to establish a consistent staged-search advantage.
The useful work is in constructing a valid search space, measuring configurations fairly, applying latency constraints, and producing a configuration someone can actually serve.
The current evidence gives me much less reason to be attached to the search order.
Even the shortcut I considered had a problem: a fitted predictor ranked the actual winning configuration a median of 15th out of roughly 25, and never in the top five across eight held-out calibration folds.
Good overall ranking correlation did not translate into finding the winner.
What you can reuse from these experiments
Whether you use PolyServe or write your own harness:
- Trace constraints through final selection. Rejecting a candidate in one stage means little if a later stage can select it again.
- Keep calibration and evaluation prompts separate. Re-measure the winner on work it did not help select itself on.
- Warm up the concurrency you measure. First-use compilation distorted my tail-latency measurements until the warm-up exercised the busiest level.
- Check prompt reuse. An earlier harness replayed prompts across concurrency levels and inflated results through prefix caching. Those throughput numbers were withdrawn.
- Give random search a fair budget. Include the time spent launching, failing, and measuring configurations.
- Put the quality cost beside the speedup. Readers need both to decide whether the result is useful.
Try it on traffic I haven’t seen
With a compatible serving backend installed:
pip install polyserve
polyserve serve Qwen/Qwen2.5-3B-Instruct --workload chat
The repository’s README covers backend dependencies and hardware requirements. You can supply representative prompts with --workload-file prompts.jsonl.
Calibration takes GPU time; the result is cached for later launches.
If you try it, I’d like an issue with your GPU, model, engine version, workload shape, latency target, and the configuration that won. A slower result or a failed launch is useful evidence too.
Repository:
https://github.com/Aagam-Bothara/polyserve
The full measurements and limitations are in docs/benchmarks.md. The reasoning behind the design choices is in docs/decisions.md.
So far, nobody outside this project has run PolyServe. If your results contradict mine, that’s the next experiment I need.
Top comments (0)