SGLang v0.5.20's release notes credit a nine-line change in the sampler with "Qwen3.5-2B decode +75% at batch 1 and +23% at batch 32". The nine lines are real, they are in the published wheel, and they do less work on the host than the line they replace. What the note leaves out is the predicate they sit behind: if a single request in the decode batch asks for top_k, top_p or min_p, your tokens never reach them. Serve a Qwen3 model through /v1/chat/completions with SGLang's own defaults and none of your traffic does, because the model's generation_config.json sets top_k and top_p for you.
This is the first of our claim checks: one published number, a test plan written before anything runs, hardware anyone can get, and a plain account of which parts hold. Nobody paid for this one.
Nine lines, and exactly one call site
PR #38117 landed on 6 September 2026 and shipped in v0.5.20 (tag commit 94602c9c, 18 September; v0.5.19 is 0bcd8223). Inside sampling_from_probs_torch in python/sglang/srt/layers/sampler.py, v0.5.19 drew the next token like this:
sampled_index = torch.multinomial(probs, num_samples=1)
v0.5.20 keeps that line as a fallback and puts the new rule in front of it, behind an environment switch that defaults to on:
if envs.SGLANG_OPT_USE_GUMBEL_SAMPLE.get():
q = torch.empty_like(probs, dtype=torch.float32).exponential_(1.0)
q.clamp_min_(torch.finfo(torch.float32).tiny)
sampled_index = (probs.float() / q).argmax(dim=-1, keepdim=True)
else:
sampled_index = torch.multinomial(probs, num_samples=1)
Dividing the probabilities by an Exp(1) draw and taking the argmax is the Gumbel-max trick, and it picks tokens from the same distribution as a categorical draw. The code comment is specific about where the time goes: torch.multinomial keeps CPU-side philox offset bookkeeping and runs a distribution-validity assert, both on the decode critical path. That work is on the host, between kernel launches, which is why the note's number is large at batch 1 and smaller at batch 32, and why your CPU decides how much you get.
SGLANG_OPT_USE_GUMBEL_SAMPLE is declared EnvBool(True) in python/sglang/srt/environ.py and accepts 0, false, no or n. That switch is the honest way to A/B this on your own hardware: same wheel, same weights, one environment variable.
One request with top_p takes the whole batch off the fast path
In v0.5.20 that function has one caller, Sampler._sample_from_probs, under if simple_sampling_case. The flags come from SamplingBatchInfo.from_schedule_batch, with any() over the requests: need_top_p_sampling is any top_p != 1.0, need_top_k_sampling is any top_k != TOP_K_ALL (1 << 30), need_min_p_sampling is any min_p > 0. Before that, is_all_greedy (every request with top_k <= 1) sends the batch to the greedy path, and --enable-deterministic-inference sends it to the seeded path.
So there are four outcomes, and only one of them is the new code:
| Batch | Path in v0.5.20 |
|---|---|
| Every request plain temperature sampling | Gumbel-max, the new code |
| Every request greedy (temperature 0) | greedy argmax, untouched |
| Any request with top_k, top_p or min_p | the filtered path, untouched |
| Server with deterministic inference | seeded multinomial, untouched |
The third row is the one that bites. A greedy request has top_k = 1 after normalization, and 1 != 1 << 30, so one temperature: 0 request sharing a decode batch moves everyone onto the filtered path.
It also lasts longer than the request that caused it. The flags are computed when requests are prefilled, and merge_batch ORs them into the running decode batch when new requests join. When a request finishes, filter_batch drops its rows and leaves every flag as it was. So one top_p request keeps the whole running batch on the filtered path after it has finished, until the running batch empties completely and the next prefill batch starts it fresh. On a server that's never idle, that can be a long time.
This port of SGLang's own arithmetic tells you which row a batch is in:
TOP_K_ALL = 1 << 30 # sampling_params.py: top_k = -1 ("whole vocabulary") becomes this
SAMPLING_EPS = 1e-6 # sampling_params.py: _SAMPLING_EPS
def normalize(params):
"""SamplingParams.normalize(): temperature below _SAMPLING_EPS means greedy (top_k = 1)."""
p = dict(params)
if 0 <= p["temperature"] < SAMPLING_EPS:
p["temperature"] = 1.0
p["top_k"] = 1
if p["top_k"] == -1:
p["top_k"] = TOP_K_ALL
return p
def path_for_batch(requests, enable_deterministic=False):
"""The sampler path for one decode batch. `requests` are normalized parameter dicts.
Returns "greedy", "gumbel" (the v0.5.20 fast path), "deterministic" or "filtered".
"""
if not requests:
raise ValueError("a batch needs at least one request")
is_all_greedy = all(r["top_k"] <= 1 for r in requests)
need_top_p = any(r["top_p"] != 1.0 for r in requests)
need_top_k = any(r["top_k"] != TOP_K_ALL for r in requests)
need_min_p = any(r["min_p"] > 0 for r in requests)
if is_all_greedy:
return "greedy"
if not (need_top_p or need_top_k or need_min_p): # simple_sampling_case
return "deterministic" if enable_deterministic else "gumbel"
return "filtered"
Your model's generation_config decides, and only on the chat endpoint
SGLang's --sampling-defaults defaults to model: the server reads temperature, top_k, top_p, min_p and repetition_penalty out of the model's generation_config.json (ModelConfig.get_default_sampling_params). They're applied in OpenAIServingChat, whose request conversion documents the order as user value, then model generation_config, then OpenAI defaults. /v1/responses is served by a subclass of it and takes the model's temperature and top_p too. /v1/completions and /generate never read the file, so they keep top_p = 1.0 and top_k = -1 and stay on the fast path, unless you started the server with --preferred-sampling-params, which the tokenizer manager merges into every request on every endpoint.
Here is that merge for the chat endpoint, from the same port:
# ChatCompletionRequest._DEFAULT_SAMPLING_PARAMS; /v1/completions and /generate use the same
# numbers from their own defaults, but never read the model's generation_config.
OPENAI_DEFAULTS = {"temperature": 1.0, "top_p": 1.0, "top_k": -1, "min_p": 0.0}
MERGED_KEYS = ("temperature", "top_k", "top_p", "min_p") # model_config.get_default_sampling_params()
def effective(request, endpoint="chat", generation_config=None, sampling_defaults="model"):
"""The sampling parameters the scheduler will see, before normalization.
endpoint: "chat" for /v1/chat/completions, "generate" for /generate and /v1/completions.
sampling_defaults: the server's --sampling-defaults ("model" is the default).
"""
from_model = {}
if endpoint == "chat" and sampling_defaults == "model" and generation_config:
from_model = {k: generation_config[k] for k in MERGED_KEYS if generation_config.get(k) is not None}
out = {}
for key in MERGED_KEYS:
if request.get(key) is not None:
out[key] = request[key]
elif key in from_model:
out[key] = from_model[key]
else:
out[key] = OPENAI_DEFAULTS[key]
return out
Pass a request body's sampling fields (missing ones as None) through effective, then normalize, then put the results in a list for path_for_batch.
Qwen's documentation recommends Temperature 0.6, TopP 0.95, TopK 20 and MinP 0 for thinking mode and calls those "the default setting in generation_config.json". Serve a Qwen3 model that ships that file, send a plain chat completion, and need_top_k_sampling is true before your request reaches the scheduler. The model in the release note is the exception, but only half of one: Qwen/Qwen3.5-2B had no generation_config.json when we checked on 26 October, so plain chat requests to it do land on the new path. Its model card, though, recommends four sampling configurations, and every one of them sets top_k=20. Clients that follow the card are on the filtered path anyway.
To see what your own server merged, grep its startup log for "Using default chat sampling params from model generation config", which OpenAIServingChat logs once with the dictionary. No line, no merge.
What a free T4 can and cannot tell you
The plan, written before the run: time both versions' rules on the same probability tensors at batch 1, 8, 32 and 64 and at vocabulary 32,768 and 151,936, once measuring the host-side call alone and once synchronizing the GPU, 200 calls a cell after 30 warm-ups, as medians with their interquartile spread; download both wheels to confirm the timed lines are the published code; compare the two rules' distributions over 200,000 draws; then install v0.5.20 and run a decode benchmark with the switch on and off.
The session gave us a Tesla T4 (sm_75, 15,360 MiB, driver 580.178.04), four vCPUs and torch 2.11.0+cu128. Both wheels still carry the lines printed above, so these numbers come from published code. The new rule landed 0.0023 from the target distribution in total-variation distance, and SGLang's two spellings of the trick picked the same token in 100% of 1,024 rows on shared randomness. Qwen/Qwen3.5-2B answered 404 for generation_config.json, which checks the paragraph above.
At Qwen3.5-2B's vocabulary of 151,936 tokens, per sampler call, median of 200 calls with every interquartile spread under 25%:
| Batch | v0.5.19 multinomial | v0.5.20 Gumbel-max | Saved per call |
|---|---|---|---|
| 1, host-side only | 143.2 µs | 45.1 µs | 98.1 µs |
| 1, GPU synchronized | 170.6 µs | 61.9 µs | 108.7 µs |
| 32, host-side only | 693.5 µs | 46.1 µs | 647.4 µs |
| 32, GPU synchronized | 725.4 µs | 628.1 µs | 97.3 µs |
Three separate sessions produced the same table within 10%.
The host-side column is the mechanism the patch targets, and it behaves as the code comment says: the new rule's host cost stays near 46 µs at every batch size, while multinomial's climbs. That's what you'd expect if its validity check makes the host wait for the device: at batch 32 its host-side time (693.5 µs) is almost all of its synchronized time (725.4 µs). The synchronized column is what a decode step pays, and there the saving is about 0.1 ms a step at both batch sizes.
That is where the claim and our measurement part company. A 0.109 ms saving becomes +75% decode throughput only if the old step cost about 0.25 ms (0.109 × 1.75 ÷ 0.75), near 4,000 tokens a second on one stream. We measured no step: pip install sglang==0.5.20 would not build in the session, because outlines_core has no wheel for its Python 3.13. So the verdict is narrow on purpose. The mechanism holds and is worth about 0.1 ms a decode step here; the +75% as worded is unverified rather than disproved, and a saving in host work scales with the host, which on Kaggle is four vCPUs and not the authors' machine. Nothing here speaks to Hopper, Blackwell or multi-GPU setups.
Gotchas
Benchmark with temperature 0 and you measure nothing. SGLang's own one-batch benchmark defaults --temperature to 0.0, which normalizes to greedy, which never calls the changed function. Qwen's docs separately tell you not to use greedy decoding with their models.
Comparing wheels is not comparing the patch. Between the two tags sampler.py alone gained 288 insertions across five commits, sampling-mask capture among them. To measure this change, flip SGLANG_OPT_USE_GUMBEL_SAMPLE inside one install.
Expect different tokens, not different quality. The two rules draw from the same distribution but consume randomness differently, so a test that pinned exact token IDs will diff after the upgrade. If you need reproducibility, --enable-deterministic-inference moves you to the seeded path, which also forces the pytorch sampling backend and skips this change.
Do this today
- Run the port above against your real request bodies and your model's
generation_config.json, and include the other traffic that shares your server: one request with top_k, top_p, min_p or temperature 0 in the running batch is enough. If it printsfiltered, this release note isn't about your server. - If it prints
gumbel, measure your own gain: start the same build twice, once withSGLANG_OPT_USE_GUMBEL_SAMPLE=0, and compare decode tokens/s. - To get the fast path on the chat endpoint, send
top_p: 1.0andtop_k: -1explicitly, or start the server with--sampling-defaults openaiand drop the model author's recommended settings. Then serve greedy and filtered traffic from another instance, because one such request holds the whole running batch off the fast path until it drains.
We also timed both rules at batch 8 and 64, and at a 32,768-token vocabulary next to Qwen3.5-2B's own 151,936, in the same run. Ask in the comments and we will post that table with its spreads.
Which vendor performance claim would you like checked next, and on what hardware?
Checked against SGLang v0.5.19 (0bcd8223), v0.5.20 (94602c9c), the v0.5.20 release notes and PR #38117's commit be00a543. InferPrice ran the sampler rules, the distribution checks and the wheel comparison on a free Kaggle Tesla T4 on 26 October 2026. We ran no SGLang server anywhere, so no end-to-end decode throughput here is ours, and we measured no Hopper, Blackwell or multi-GPU path.
Sources
- SGLang v0.5.20 release notes: https://github.com/sgl-project/sglang/releases/tag/v0.5.20
- The change itself, commit
be00a543(PR #38117), andpython/sglang/srt/layers/sampler.pyat tags v0.5.19 and v0.5.20: https://github.com/sgl-project/sglang -
SGLANG_OPT_USE_GUMBEL_SAMPLEinpython/sglang/srt/environ.py, the predicate inpython/sglang/srt/sampling/sampling_batch_info.py, and the defaults inpython/sglang/srt/sampling/sampling_params.py, all at v0.5.20 -
filter_batchandmerge_batchinpython/sglang/srt/sampling/sampling_batch_info.py, and the running-batch merge inpython/sglang/srt/managers/scheduler.py, at v0.5.20 -
OpenAIServingResponsesinpython/sglang/srt/entrypoints/openai/serving_responses.py, and--preferred-sampling-paramsinpython/sglang/srt/managers/tokenizer_manager.py, at v0.5.20 -
--sampling-defaultsinpython/sglang/srt/arg_groups/fields/serving.py,ModelConfig.get_default_sampling_paramsinpython/sglang/srt/configs/model_config.py, and the merge order inpython/sglang/srt/entrypoints/openai/protocol.py -
--temperaturedefault inpython/sglang/benchmark/one_batch_server.pyat v0.5.20 - Qwen's recommended sampling parameters: https://qwen.readthedocs.io/en/stable/getting_started/quickstart.html
- The
Qwen/Qwen3.5-2Bmodel card's recommended sampling parameters: https://huggingface.co/Qwen/Qwen3.5-2B - The
Qwen/Qwen3.5-2Bfile list, checked 26 October 2026: https://huggingface.co/Qwen/Qwen3.5-2B/tree/main
Originally published at inferprice.pages.dev.
Top comments (0)