Sampler Order: Why temperature and top_p Don't Compose

A team I worked with had a generation config they trusted: temperature=0.7, top_p=0.95. Someone wanted more variety in marketing copy, bumped temperature to 1.2, left top_p alone, and the model started emitting tokens that looked like they came from a different model entirely — stray Unicode, half-words, a citation to a paper that doesn't exist. The obvious read is "temperature 1.2 is just hot." The real cause is sampler order: top_p is computed on the temperature-scaled distribution, so raising temperature didn't just flatten the probabilities, it silently widened the nucleus from 5 candidate tokens to 63.

top_p is not a fixed diversity knob. It is a function of whatever the distribution looks like at the moment the truncation runs — and that depends on where temperature sits in your runtime's sampler chain.

TL;DR

top_p truncates the post-temperature distribution in HuggingFace transformers and vLLM. Temperature flattens the distribution, so the nucleus set grows non-linearly with T.
On a realistic logit vector, top_p=0.9 keeps 4 tokens at T=0.7, 7 at T=1.0, 22 at T=1.2, and 88 at T=1.5. The knob you didn't touch changed by 20x.
Sampler order differs across engines. llama.cpp's default chain runs truncation before temperature; HF and vLLM run temperature first. Same (T, top_p) pair, materially different sampling distributions.
Additive penalties have the same coupling: a penalty α applied pre-temperature shifts log-odds by α/T, so presence_penalty=0.5 at T=0.5 bites twice as hard as at T=1.0.
Fix: tune one knob. Pin top_p=1.0 and sweep temperature, or pin T=1.0 and sweep top_p. If you want a truncation that survives temperature changes, use min_p.

Why doesn't temperature compose with top_p?

Because nucleus sampling is defined over cumulative probability mass, and temperature changes how mass is distributed before that cumsum ever runs.

Temperature rescales logits: p_i(T) ∝ exp(z_i / T). Nucleus sampling then sorts descending and keeps the smallest set whose cumulative probability reaches p. Those two operations are not independent — the size of the nucleus is a function of the entropy of the tempered distribution, and entropy grows monotonically with T.

The growth is brutally non-linear once the tail wakes up. Take a plausible next-token distribution: ten reasonable continuations with logits from 14.0 down to 10.0, plus a long tail of ~4000 junk tokens starting at logit 8.0. Here's what each truncation keeps:

T	`top_p=0.9` keeps	`top_p=0.95` keeps	`min_p=0.05` keeps	p_max
0.5	3	4	4	0.641
0.7	4	5	5	0.507
1.0	7	10	7	0.369
1.2	22	63	9	0.295
1.5	88	140	10	0.204
2.0	179	248	10	0.107

Read the top_p=0.95 column again: 5 tokens at T=0.7, then 63 at T=1.2. The junk tail has a mass of 0.0034 at T=0.7 and 0.12 at T=1.2 — once the tail carries enough mass to matter, the cumsum has to reach deep into it to hit 0.95. That is the entire failure mode. You raised one knob 1.7x and the candidate set grew 12x.

Note the min_p column stays flat. min_p keeps every token with p_i ≥ min_p · p_max, a relative threshold. Because temperature scales the whole distribution roughly self-similarly near the head, the ratio test is far more stable than a cumulative-mass test. That's the practical argument for min_p, and it holds up in the numbers.

Does sampler order actually differ between runtimes?

Yes, and this is the part that breaks reproducibility across deployments.

HuggingFace transformers: the temperature warper is constructed before the top-k/top-p warpers, so logits are tempered first, then truncated.
vLLM: penalties → temperature → top_k/top_p → sample. Same effective order as HF.
llama.cpp: the chain is explicit and configurable via --samplers, and the default runs the truncation samplers before temperature (with the final distribution sampler last). Check your build's default — it has changed across versions — but the "temperature near the end" convention is the norm there.

Truncate-then-temper is a genuinely different algorithm. Truncation on the raw (T=1) distribution fixes the candidate set once; temperature then only reweights within that set. It can never resurrect a token that fell outside the nucleus.

Running the same logits through both orders at top_p=0.9:

T	temper→truncate (HF/vLLM)	truncate→temper (llama.cpp default)
1.0	7 tokens, 2.30 bits	7 tokens, 2.30 bits
1.2	22 tokens, 2.88 bits	7 tokens, 2.43 bits
1.5	88 tokens, 4.28 bits	7 tokens, 2.55 bits

At T=1.5, the HF/vLLM order gives a 26.9% chance of sampling a token that wasn't in the raw top-7 at all. The llama.cpp order gives 0%, by construction. Identical config file, identical weights, two runtimes, and one of them is drawing from a distribution with nearly 2 extra bits of entropy.

Here's the script — it's short enough to just run against your own logits:

import math

def softmax(z, T=1.0):
    m = max(zi / T for zi in z)
    e = [math.exp(zi / T - m) for zi in z]
    s = sum(e)
    return [x / s for x in e]

def nucleus(p, thr=0.9):
    """Indices kept by top_p, applied to whatever distribution you hand it."""
    order = sorted(range(len(p)), key=lambda i: -p[i])
    keep, c = [], 0.0
    for i in order:
        keep.append(i)
        c += p[i]
        if c >= thr:
            break
    return keep

# 10 plausible continuations + a long tail of junk tokens
z = [14.0, 13.4, 13.1, 12.6, 12.0, 11.6, 11.2, 10.9, 10.5, 10.0]
z += [8.0 - 0.02 * i for i in range(4000)]

for T in (0.7, 1.0, 1.2, 1.5):
    # Order A — HF / vLLM: temperature, then top_p
    a = nucleus(softmax(z, T), 0.9)
    # Order B — llama.cpp default: top_p on raw logits, then temperature
    b = nucleus(softmax(z, 1.0), 0.9)
    escaped = sum(softmax([z[i] for i in a], 1.0)[j]
                  for j, i in enumerate(a) if i not in b)
    print(f"T={T}: A keeps {len(a):>3}, B keeps {len(b):>3}, "
          f"P(outside B's set | order A) = {escaped:.3f}")

Swap in a real logit vector from output.logits[0, -1] and the effect is the same shape. Distributions with fat tails (open-ended prose, low-confidence positions) blow up fastest; distributions with a single confident token (code, JSON keys, tool names) barely move.

What does this break in production?

Temperature sweeps become confounded experiments. If your eval grid is T ∈ {0.3, 0.7, 1.0, 1.3} with top_p=0.95 pinned, you are not measuring temperature. You are measuring temperature and an uncontrolled nucleus width that varies with the entropy of every individual token position. A "temperature 1.3 is bad for our task" conclusion may just be "the tail leaked in at 1.3."

Cross-engine parity checks fail mysteriously. Prototype on llama.cpp locally, deploy on vLLM, keep the same YAML, and the served model is measurably more erratic at any T > 1. Nothing in the config diff explains it.

Cached prompts don't save you from sampler drift. Prompt caching and batching change nothing here; this is post-forward-pass math. But it does mean the same cached prefix can yield very different output-quality distributions across engines, which is a nasty thing to debug when the input side is byte-identical.

Do penalties interact with temperature too?

Yes, the same way, and it's easy to derive. In vLLM and OpenAI-compatible servers, frequency/presence penalties are applied to raw logits before temperature. An additive penalty α on token i changes its log-odds against any competitor by -α/T, so the odds multiplier is exp(-α/T):

presence_penalty=0.5 at T=1.0 → odds multiplier exp(-0.5) = 0.607
presence_penalty=0.5 at T=0.5 → odds multiplier exp(-1.0) = 0.368

On the same logit vector as above, a 0.5 penalty on the top token drops its probability by 29% at T=1.0 and 38% at T=0.5. So a penalty tuned at chat-style temperatures becomes noticeably more aggressive when you drop temperature for a structured-output path — usually the exact path where you least want the model steered off the token it was confident about. HF's multiplicative repetition_penalty has the same 1/T amplification, since it also edits logits pre-temperature.

What should I set instead?

Pick one truncation strategy and hold the other constant:

# vLLM — sweep temperature, leave the nucleus alone
SamplingParams(temperature=0.9, top_p=1.0, top_k=-1, min_p=0.03)

# Structured output / tool calls — no sampling knobs to misread
SamplingParams(temperature=0.0, top_p=1.0, top_k=-1,
               presence_penalty=0.0, frequency_penalty=0.0)

Rules that hold up:

Never sweep temperature and top_p together. Pin one at its no-op value (top_p=1.0 or T=1.0).
Prefer min_p for tail-cutting if your engine supports it. min_p=0.02–0.05 removes junk without collapsing under temperature changes.
Zero the penalties on structured paths. A repetition penalty at low temperature is a strong steering force, and it will happily corrupt long JSON or repeated code tokens.
Record the engine and version in eval artifacts, not just the sampling params. top_p=0.9 is not a portable specification.
On hosted APIs, change one or the other. Anthropic's and OpenAI's docs both recommend altering temperature or top_p, not both — this is exactly why. Claude's extended thinking mode goes further and requires temperature=1, which removes the interaction entirely for that path.

So why don't temperature and top_p compose?

Because top_p is a cumulative-mass cutoff evaluated on the distribution that exists at its point in the sampler chain, and in HuggingFace transformers and vLLM that distribution has already been divided by temperature. Raising T flattens the head and inflates the tail, so the cumsum has to walk much deeper to reach the same mass — top_p=0.9 keeps 4 tokens at T=0.7 and 88 at T=1.5 on a typical fat-tailed logit vector. Sampler order determines the outcome: engines that truncate before applying temperature (llama.cpp's default chain) hold the candidate set fixed and only reweight inside it, producing a fundamentally different sampling distribution from the same config. Treat (temperature, top_p, penalties, engine) as one coupled setting, sweep only one axis at a time, and use min_p when you want a tail cutoff that doesn't move when temperature does.

DEV Community