A model picks every word with a weighted dice roll, and one bad roll in a long answer is a broken JSON field. Top-p blocks fewer of those rolls than you think, because temperature runs first.
Engineer's view. This is a per-call error rate in a long batch job. A dependency that fails 0.2% of calls breaks most 2,000-call jobs. Each token is one weighted random draw, and your sampling settings set how often it lands on a token the model rated near zero. The bill is broken JSON that tests never catch.
TL;DR. In LLM sampling, temperature reshapes the model's odds first, and top-p or min-p then decides which unlikely tokens may still be drawn. Top-p lets more of them back in as temperature rises while min-p keeps blocking them, so use min-p when one stray word breaks your output.
Explain Like I'm 5
Imagine writing a story by pulling words out of a bag, one word at a time. The bag holds lots of good words and a few silly ones. Most pulls are fine. But you pull five hundred times, so sooner or later a silly word sneaks in.
You can shake the bag harder to get more surprising words. You can also throw words out before you pull. One rule says: keep the best words until you hold most of the bag. Another says: throw out any word far less likely than the best one. Shake hard, and the first rule keeps lots of silly words. The second rule still throws them out.
The Problem
You have shipped this bug before, in a system that had nothing to do with AI.
A payment service fails one call in 500. Nobody worries, because the user retries and it works. Then you write a nightly job that makes 2,000 of those calls in a row. It fails almost every night. A failure rate you cannot see per call becomes a certainty per job.
A language model writes an answer the same way, one token at a time. A token is a word or a piece of one. At each step the model scores every token it knows. Then a sampler picks one at random, weighted by those scores. Most of the weight sits on a few sensible tokens. A thin layer is spread across thousands of tokens the model itself rates as nearly impossible.
Each draw has a small chance of landing in that layer. Over a 500-token answer, small chances add up. In the toy model built for this article, plain sampling put about eleven such tokens into every answer. One of them inside a JSON key or a SQL column name is the whole bug.
Most teams reach for top-p, the setting meant to cut that layer off. It works at the default temperature. Raise the temperature for more variety, and top-p starts letting the tail back in. In the same test it admitted about forty stray tokens per answer.
The fix: cut the tail by each token's odds against the best one, with min-p, and remember that temperature runs before any cutoff.
Read the full post on The AI Commit
How Temperature, Top-p and Min-p Decide What a Model Writes carries the rest: the fix and why it works, the code to change for each role, how to tell it worked, when it is the wrong tool, and an interactive model you can play with.
Originally published at theaicommit.com, a daily AI engineering lab for software engineers. The original has an interactive version you can play with, and the full runnable code is on GitHub.
Top comments (0)