A language model never hands you a sentence. At every single step it hands you a probability distribution over its entire vocabulary — P(next token | everything so far). The word "decoding" is the name for the rule you use to turn that distribution into one actual token, append it, and ask the model again. Change the rule and the same model behaves completely differently. That one choice is why a model can be a precise translator and a chatty story-writer at the same time.
Here are the five rules everyone should know.
Greedy: always take the most-likely token
The simplest possible rule: at each step, pick the single highest-probability token (argmax). It's fast and fully deterministic — same prompt, same answer, every time.
The problem is that the locally-best token is not the globally-best sentence. Greedy is short-sighted, and in practice it walks straight into repetition loops: "the movie was very very very very...". Once it enters a high-probability rut, it can never look sideways to escape. Fine for very short, factual completions; poor for anything open-ended.
Beam search: keep the top-B sequences
Instead of committing to one token, beam search keeps the B best partial sequences alive (the "beams"). Each step it expands every beam over the vocabulary, scores each candidate by its cumulative log-probability — you sum log-probs, because adding in log space avoids the numerical underflow you'd get multiplying hundreds of tiny probabilities — and then prunes back to the top B. At the end it returns the highest-scoring complete sequence.
Beam search searches a much wider slice of the tree than greedy, and it shines where there is essentially one correct answer: machine translation, summarization, speech-to-text. But for open-ended writing it produces generic, safe, oddly bland text. Two reasons. First, cumulative probability rewards unsurprising sequences — and the most probable sentence is usually the most boring one. Second, unless you length-normalize the score, beam quietly prefers shorter outputs. It can still repeat, too.
Why we sample
For stories, brainstorming, and chat we don't want the single most-probable sequence — we want variety. So instead of maximizing, we sample: draw the next token at random, in proportion to its probability. High-probability tokens still usually win, but the model can occasionally take an interesting turn. This is exactly why "regenerate" gives you a fresh answer, and why greedy and beam never do.
The next three rules all shape how we sample.
Temperature: sharpen or flatten first
Before sampling, divide the logits by a number T and re-softmax. T below 1 sharpens the peak — more confident, closer to greedy, lower entropy. T above 1 flattens the distribution — more uniform, higher entropy, more surprising. T approaching 0 collapses to greedy; T approaching infinity is uniform noise. This is the "creativity dial" behind the temperature parameter in every API.
Top-k: keep a fixed number of candidates
Pure temperature sampling has a flaw: the long tail of thousands of tiny-probability tokens can, collectively, still get picked, producing an out-of-nowhere junk word. Top-k fixes it bluntly — keep only the k highest-probability tokens, zero the rest, renormalize, then sample. Its weakness is that k is fixed: too wide when the model is confident, too narrow when it's genuinely unsure.
Top-p (nucleus): an adaptive cutoff
Top-p fixes top-k's rigidity. Sort the tokens by probability and keep the smallest set whose probabilities sum to at least p — the "nucleus". When the model is confident, that's one or two tokens; when it's unsure, it's many. The cutoff adapts to the shape of each step. That's why top-p around 0.9 is the workhorse default for chat.
The real recipe
Modern chat models combine these: apply a repetition/presence penalty to tokens already used (kills loops), scale by temperature (~0.7–1.0), trim with top-p (~0.9, often with a top-k safety cap), then sample.
It all reduces to one tradeoff: determinism versus diversity. Need a single correct, reproducible answer — code, extraction, classification, translation? Use greedy, temperature 0, or beam. Need variety and natural conversation? Sample with temperature ~0.8 and top-p ~0.9. The model is identical; only the decoding rule changes.
I built an interactive explorer where you pick a strategy, tune its knobs, and watch it generate token by token over a small scripted model — greedy loops, beam finds the safe complete sentence, and sampling gives you a different answer every run. Every distribution, log-prob, beam prune and sampled draw is real code running in the page.
Try it here: https://dev48v.infy.uk/ai/days/day33-beam-vs-sampling.html
Top comments (0)