Temperature doesn't make your model creative

What temperature, top-k, and top-p actually do to the next token and why none of them can change its ranking.

If you have used a language model API, you would have seen the advice to turn temperature up for more creative output and down for more focused output. It is not wrong about what you will observe, but it is misleading about what is actually happening under the hood.

Here is the claim of this article, stated plainly: temperature does not add creativity to a model. It controls how far the model reaches into variety it already had. The model does not invent a new token because you raised the temperature. It only changes the willingness to pick tokens that were already on the table.

We will start one step earlier, with the raw numbers the model produces when choosing the next token.

I assume you have the rough understanding of -

• how a language model generates text which is one token at a time and each depending on everything before it.
• Probability distribution where there are options with probabilities that sum to 1.

Raw scores to probabilities

When a model chooses the next token, its final step produces one raw number for every token in its vocabulary. These raw numbers are logits, also plain scores where higher means more favored. Logits/scores can be any real number and they do not sum to anything meaningful. You can think of them as a cook's preference score for every dish on a fixed menu.

We cannot really make decision based on raw scores, so need a confidence score i.e. probabilities. To transform the scores into probabilities, Softmax function is used: it exponentiate each logit (which makes every value positive and stretches the gaps), then divide by the total so everything sums to 1.

Suppose the prompt is The weather today is and model produced five candidate next tokens with the logits in the table below. Working through e³ ≈ 20.1, e² ≈ 7.4, e¹ ≈ 2.7, e⁰ = 1.0, e⁻¹ ≈ 0.37, the total is ≈ 31.6, and dividing each by it gives:

Token	Logit	Probability
sunny	3	63.6%
cloudy	2	23.4%
warm	1	8.6%
terrible	0	3.2%
purple	-1	1.2%

(table 1) Logits converted to probabilities.

As a formula, for token i:

zᵢ is the logit for token i, and the denominator sums the exponentiated scores over all tokens j, which forces the result to sum to 1.

Before we touch anything, notice that purple is at 1.2%. Take a mental note of that.

Temperature: how strictly the model follows its own ranking

Temperature is a single number T, introduced with one change to softmax: divide every logit by T before exponentiating and that is the entire mechanism.

Notice that temperature is not a different function from softmax but it is the same softmax with one extra step. At T = 1 you get plain softmax back. Temperature generalizes softmax rather than replacing it.

In the kitchen picture, temperature is how strictly the cook follows their own ranking where low T is rigid and sticks to top rated items in the menu, while high T reaches further down the menu but the menu itself never changes.

Same logits, three temperatures:

Token	Logit	T = 0.5	T = 1	T = 2
sunny	3	86.5%	63.6%	42.9%
cloudy	2	11.7%	23.4%	26.0%
warm	1	1.6%	8.6%	15.8%
terrible	0	0.2%	3.2%	9.6%
purple	-1	0.03%	1.2%	5.8%

(table 2) The same five tokens at three temperatures. The ranking is identical in every column.

As we lower T it sharpens the distribution and probability mass concentrates at the top of the rankings. sunny climbs to 86.5% and the tail collapses. Raising T flattens it, so sunny falls to 42.9% and the tail fills in where purple is now 5.8% which is about five times its T = 1 value. From a limits perspective, as T → 0 (as T approaches 0) everything piles-up on the top token, and T → ∞ (as T approaches infinity) it approaches uniform where every token is equally likely and logits no longer matter.

Note

T = 0 would divide by zero so implementations never compute it. They take the highest logit (argmax) token directly (greedy decoding). Also, some APIs clamp T to a tiny positive epsilon value to avoid division by zero error.

Basically, the value of T decides how much the strong candidates dominate versus how much visibility the weak ones get - all without touching the ranking.

Why the ranking never changes

The order is identical within table 2 and it will never change. Let's take the ratio of two tokens probabilities.

The softmax denominators cancel as we divide softmax of 1st token by 2nd. The whole ratio comes down to e raised to (gap / T). If we check it against the table at T = 2, the gap is 3 − 2 = 1, so the ratio should be and 42.9 / 26.0 ≈ 1.65.

Two things to note here:

• Order is set by the sign of the gap - T is always positive, so dividing by it can never flip a sign. So as long as sunny's logit is larger, sunny stays more likely at every temperature value. Temperature cannot reorder tokens.
• Spacing is set by the magnitude of the gap over T - Large T shrinks the exponent toward 0, so ratios head to 1 and the distribution flattens. Small T blows the exponent up, and the mass piles onto the top token.

Temperature just scales the gaps between probabilities but it cannot change which way the gaps point. And everything about "creativity" follows from this.

So where does the "creativity" come from?

If temperature cannot promote a better token or add a new one, why does raising it feel creative?

If we look at purple specifically - at T = 2 it is about five times more likely than it was at T = 1, so when the model says something unexpected, that looks like creativity but the thing is purple was already in the distribution at 1.2%. Temperature value did not invent it, it just fattened its slice. The variety was always there in the tail and temperature value decides how far into the tail you reach.

Temperature can produce output which will look creative but what it is not doing is generating that creativity. It is simply controlling the access to variety the model already had in its distribution. And a practical consequence of this is:

• Temperature only has leverage where the model was already uncertain. If one logit has a very large spike over the rest, in that scenario there is barely any tail to reach into, so high T does not change things very much. On the other hand, if several tokens have similar logits, temperature has high leverage. So high-temperature "creativity" does not show up evenly across a generation. It shows up exactly at the positions where the model already had several reasonable continuations. The model's uncertainty decides whether there is variety or not and then temperature decides how much of it you sample.

Neither extreme is creative. As decoding becomes greedy and deterministic. Models tend to assign a repeated phrase even higher probability each time it appears, so once a repetition starts it self reinforces and greedy has no randomness to break the cycle. Long greedy generations often collapse into loops for exactly this reason. Very high T samples the tail indiscriminately and the tail holds terrible as well as purple, so output degrades toward noise.

top-k and top-p

Temperature keeps all the tokens and it only reweights, so every token keeps a nonzero probability at any finite T, whereas top-k and top-p truncate because they delete tokens from consideration and sampling happens only among the surviving tokens. In the kitchen example, they mark dishes off the menu rather than changing the cook's preference.

• top-k is very simple as the name is self explanatory. It sorts tokens by probability, keeps the top k, discards the rest, and finally renormalizes the survivors so they sum to 1 again. From our example, with k = 3, only sunny, cloudy, and warm will survive. k is a fixed number which is also its weakness. When the model is very confident and suppose k = 40 - It still pulls in the 39 tokens which it did not want but when the model is spread across a hundred options, k = 40 arbitrarily drops 60 of them. So a fixed count cannot differentiate between those two situations.

• top-p which is also known as nucleus sampling fixes the top-k weakness. top-p also sorts tokens by probabilities and only keeps the smallest set of tokens whose probabilities sum to at least defined p. It also includes the token that tips p over and then renormalizes. With p = 0.8 on table 1, sunny is 63.6%, add cloudy to reach 87%, stop and the survivors are {sunny, cloudy}. The size of the set is variable because a confident distribution covers p with two tokens, an uncertain one needs dozens. top-p sizes the candidate pool based on the model's confidence, which is why it is usually preferred over top-k.

	temperature	top-k	top-p
What it does	reweights the odds	deletes tokens	deletes tokens
Removes tokens?	never	yes, fixed count	yes, variable count
Adapts to confidence?	—	no	yes

*(table 3) Two delete tokens; one reweights. None reorder.*

Putting them together, then the draw

In most implementations, the flow is in a specific order:

logits -> temperature reshapes -> top-k / top-p truncate -> sample

temperature and top-p

top-p considers cumulative probabilities after temperature has reshaped them, so raising temperature also widens the top-p set as a side effect. A flatter distribution needs more tokens to reach p. One knob, two entangled effects. This is why the usual advice is to adjust temperature or top-p, not both.

The final step is sampling. Put the surviving tokens on a number line from 0 to 1, where each token's segment is as wide as its probability. Draw one uniform random number and whichever segment it lands on is your token. A 60% token gets hit about 60% of the time, a 1.2% token is a very thin slice and occasionally wins. And that thin slice is where the unexpected token finally comes from (which gets the name "creativity").

So the three knobs only set the segment widths. If you skip the draw and always take the widest segment then none of them change the output at all, because the widest segment is the same at every T. The ranking never changes and these knobs matter only because a random draw follows.

When to use what

• One right answer (code, extraction, arithmetic): low T (0–0.3). When a correct token exists, you do not want the tail winning.
• Open-ended generation (brainstorming, fiction, varied phrasings): T around 1 or above because the plausible alternatives live in the tail, and you want access to them.
• Conversation: around 0.7 so you get enough variety to avoid robotic repetition.
• Truncation: top-p around 0.9–0.95 could be a sane default because it adapts to confidence and reach for top-k when you specifically want a hard cap on the candidate count.

Avoid T much above 1.5 unless you want near-random output because past a point the tail stops being interesting and becomes noise.

Scope

Deliberately not covered:

• how the logits themselves are produced
• controls applied at the logit stage before these parameters are applied (repetition penalties, logit bias, constrained decoding)
• why even T = 0 is not always bit-for-bit deterministic in practice
• subject of the follow-up to this post: why newer reasoning models restrict temperature entirely and steer you toward higher-level controls instead.

Thank you for reading! It's very useful to know what these parameters cannot do along with what they can do.

---

References & Further Reading

1. Is Temperature the Creativity Parameter of Large Language Models?
2. The Curious Case of Neural Text Degeneration