Min-p Sampling: Why top_p Truncates the Wrong Tail

Crank temperature to 1.5 with top_p=0.9 and watch a good model fall apart: mid-sentence topic switches, invented tokens, the occasional emoji in your JSON. The usual diagnosis is "temperature too high." The real culprit is that top_p truncates the wrong tail. Min-p sampling fixes the truncation criterion, and once you understand why, you'll stop reaching for top_p on self-hosted models.

TL;DR

• top_p (nucleus) keeps a fixed cumulative probability mass regardless of the distribution's shape. When temperature flattens the distribution, that fixed mass sweeps in a long tail of low-quality tokens.
• min_p keeps tokens whose probability is at least min_p × p_max — the cutoff scales with the top token's confidence, so it stays tight when the model is sure and relaxes when the model is genuinely uncertain.
• This decoupling lets you run higher temperature for diversity while keeping output coherent — top_p can't do both at once.
• Sampler order matters: apply min_p relative to the peak, and be deliberate about whether temperature is applied before or after truncation. Get it backwards and min_p silently degrades.
• min_p is a self-hosted feature (vLLM, llama.cpp, Hugging Face transformers, TGI). The Anthropic and OpenAI APIs don't expose it — they give you temperature and top_p only.

What is min-p sampling, exactly?

Min-p sampling truncates the next-token distribution using a threshold relative to the most likely token, then renormalizes and samples. You pick a base value p_base (say 0.1). The sampler computes p_max, the probability of the top candidate, sets threshold = p_base × p_max, and discards every token below it.

Concretely: if the top token has probability 0.6 and p_base = 0.1, the cutoff is 0.06. Any token under 6% survives only if it clears that bar. If instead the top token is 0.2 (the model is unsure), the cutoff drops to 0.02, so the sampler admits more candidates — exactly when you want more diversity.

That single "relative to the peak" idea is the whole trick. top_p and top_k both use an absolute criterion; min_p uses a relative one.

Why does top_p truncate the wrong tail?

top_p keeps the smallest set of tokens whose cumulative probability reaches p, and that criterion is blind to the peak's height. The failure shows up when the distribution has a confident head plus a long, flat tail.

Say the top token sits at 0.45 and the remaining mass is spread across dozens of tokens each around 0.01. With top_p = 0.9, the sampler keeps adding tokens — 0.45, then 0.46, 0.47… — dragging in 40-plus low-probability tokens just to reach cumulative 0.9. Many of those are semantically wrong. You've told the sampler "capture 90% of the mass," and it obeyed by scooping up garbage.

Temperature makes this worse because it flattens the distribution before top_p runs. Higher temperature lowers the peak and fattens the tail, so a fixed 0.9 cutoff now spans even more junk tokens. That's the mechanism behind "high temperature breaks the model" — it isn't the temperature alone, it's temperature feeding a shape-blind truncation.

min_p dodges this. With p_base = 0.1 and a peak of 0.45, the cutoff is 0.045, so every 0.01 tail token is cut in one shot. The number of survivors depends on how many tokens are genuinely competitive with the leader, not on an arbitrary mass target.

Show me the truncation difference in code

Here's a minimal NumPy implementation of both, so you can see the survivor sets diverge on the same logits:

import numpy as np

def softmax(logits, temp=1.0):
    z = logits / temp
    z -= z.max()
    e = np.exp(z)
    return e / e.sum()

def top_p_keep(probs, p=0.9):
    order = np.argsort(probs)[::-1]
    cum = np.cumsum(probs[order])
    # keep tokens up to and including the one that crosses p
    cutoff = np.searchsorted(cum, p) + 1
    return set(order[:cutoff].tolist())

def min_p_keep(probs, p_base=0.1):
    threshold = p_base * probs.max()
    return set(np.where(probs >= threshold)[0].tolist())

# a confident head + long flat tail
logits = np.array([6.0, 3.2, 3.1] + [0.5] * 40)

for temp in (1.0, 1.8):
    probs = softmax(logits, temp)
    tp = top_p_keep(probs, 0.9)
    mp = min_p_keep(probs, 0.1)
    print(f"temp={temp}: top_p keeps {len(tp)}, min_p keeps {len(mp)}")

Run it and the pattern is stark: as temperature climbs, top_p's survivor set balloons (the flat tail now fits under 0.9), while min_p's set stays small because the tail never gets within 10% of the peak. That growing top_p set is your incoherence.

Does min-p let me raise temperature safely?

Yes — that's the practical payoff. Because min_p's cutoff tracks the peak, you can push temperature into a range that would wreck top_p and still keep tokens coherent. The distribution flattens, the relative gap between the leader and the tail shrinks proportionally, and the threshold moves with it.

The knobs invert in a useful way:

• temperature now controls how much you sample among the legitimately competitive tokens — pure diversity.
• min_p controls how competitive a token must be to enter the pool at all — the coherence floor.

With top_p those two concerns are tangled, because temperature changes the token count that a fixed mass implies. With min_p they separate cleanly. For creative generation, people report running temperatures well above 1.0 with a min_p around 0.05–0.1 and getting varied but sane text. Treat those as starting points, not gospel — the right value depends on your model and task, and you should eval it.

What min_p value should I start with?

Start at min_p = 0.05 to 0.1 and disable top_p and top_k so you're testing one truncation rule at a time. Lower p_base (0.02–0.05) admits more of the tail — more diverse, more risk. Higher p_base (0.15+) clamps hard toward greedy, useful when you want near-deterministic structure but a little wiggle.

Rules of thumb:

• Structured output / code: keep min_p higher (0.1+) or just use greedy. You want the peak to dominate.
• Creative / brainstorming: min_p around 0.05 with elevated temperature.
• Don't stack it with top_p: running both means two truncation stages fighting each other, and you lose the clean mental model. Pick one.

Does sampler order matter for min_p?

It matters a lot, and it's the most common way min_p silently underperforms. Two ordering questions decide the behavior:

1. Temperature before or after truncation. If temperature scales the logits before min_p computes its threshold, the peak height itself moves, which shifts the cutoff. If min_p runs on the raw distribution and temperature is applied only to the surviving tokens before the final draw, the survivor set is temperature-independent. These produce different outputs. Inference stacks differ here, so read your engine's docs instead of assuming. In vLLM you control this through SamplingParams; the sampler applies penalties, then top_k/top_p/min_p filtering, then temperature scaling for the draw — check the version you run, because the pipeline has changed across releases.

2. min_p relative to which peak. min_p must key off the current top probability after any earlier filters, not a stale value. If a repetition penalty has already reshaped the logits, the peak it sees is the post-penalty peak. Chaining penalties, top_k, and min_p in the wrong order can move the threshold somewhere you didn't intend.

A typical, sane vLLM config:

from vllm import SamplingParams

params = SamplingParams(
    temperature=1.4,
    min_p=0.08,
    top_p=1.0,   # disabled — don't stack truncation rules
    top_k=-1,    # disabled
    max_tokens=512,
)

Setting top_p=1.0 and top_k=-1 is the point: you're isolating min_p as the only truncation stage.

Can I use min_p with the Claude or OpenAI API?

No. The Anthropic API (Claude Opus 4.x, Sonnet 4.x) exposes temperature, top_p, and top_k; the OpenAI API (GPT-5.x) exposes temperature and top_p. Neither surfaces min_p. Min-p is a decoding-time sampler, so you only get it where you control decoding: vLLM, llama.cpp / GGUF stacks, Hugging Face transformers (model.generate(..., min_p=...)), and TGI.

If you're on a hosted API and fighting incoherence at high temperature, your only real levers are lowering temperature or tightening top_p — you can't get min_p's shape-adaptive behavior. That's a genuine argument for self-hosting when sampler control is part of your product (creative writing tools, agents that need diverse-but-valid samples, synthetic data pipelines). For most production API work, the honest move is to keep temperature moderate and stop pretending top_p at 0.9 is protecting you at temperature 1.5 — it isn't.

The direct answer: why does min-p beat top_p?

Min-p sampling beats top_p because it truncates the next-token distribution relative to the model's confidence — it keeps tokens scoring at least min_p × p_max — instead of top_p's shape-blind rule of grabbing a fixed cumulative mass. top_p's fixed mass drags in a long tail of low-probability, incoherent tokens whenever the distribution flattens, which is exactly what temperature does; so "high temperature breaks the model" is really "top_p truncates the wrong tail." By scaling its cutoff to the peak, min_p stays tight when the model is certain and opens up only when the model is genuinely uncertain, which lets you raise temperature for diversity while keeping output coherent. Start around min_p = 0.05–0.1 with top_p and top_k disabled, watch your sampler's temperature-vs-truncation ordering, and remember it's a self-hosted feature — the Claude and OpenAI APIs don't expose it.