Two people send an LLM the identical prompt and get different answers. The model isn't confused — it's rolling dice on purpose, and a few settings control exactly how loaded those dice are. Miss this and your app is either boringly repetitive or wildly unreliable.
Temperature, top-p, and their cousins are the least glamorous and most practically important knobs in applied AI. Here's what they actually do.
The model doesn't pick a word — it picks from a distribution
At every step, an LLM doesn't decide on the next token. It produces a probability distribution over all possible next tokens — "cat" 40%, "dog" 25%, "car" 8%, and so on across the whole vocabulary. Something then has to sample an actual token from that distribution. The sampling settings control how that choice is made, and that's where randomness enters.
Temperature: how adventurous the model is
Temperature reshapes the distribution before sampling.
- Low temperature (near 0) sharpens it — the most probable token becomes almost certain. Output is focused, deterministic, repetitive. This is what you want for factual answers, extraction, and code.
- High temperature (up toward 1 and beyond) flattens it — less-likely tokens get a real chance. Output is more varied, creative, and surprising. Good for brainstorming and writing, dangerous for anything that must be correct.
Think of it as a creativity dial. Turned down, the model plays it safe. Turned up, it takes risks — including the risk of wandering into nonsense. For anything where correctness matters, keep it low; high temperature just gives a model more room to embellish, which is a fast route to confident errors.
Top-p: trimming the long tail
Top-p (nucleus sampling) takes a different cut. Instead of reshaping probabilities, it restricts the pool of candidates: keep only the most likely tokens that together add up to, say, 90% of the probability mass, and sample from just those. It dynamically ignores the absurd long tail — the tokens that are technically possible but almost always wrong — while still allowing variety among the sensible options. Temperature and top-p are often used together.
Why this matters for reliability
Here's the part people learn the hard way: non-determinism is a feature you must manage, not a bug to ignore. If your system needs the same input to reliably produce the same structured output, a high temperature will quietly sabotage you with occasional off-format responses. Turning the randomness down is often the cheapest reliability fix there is — a lesson I apply across the AI systems I build.
Conversely, if you crank everything to deterministic and wonder why your "creative writing assistant" feels lifeless — that's the same knob, turned the wrong way for the job.
Matching the setting to the task
- Facts, extraction, classification, code, structured output: low temperature. You want the confident, consistent answer.
- Brainstorming, creative writing, generating variety: higher temperature, so the model explores.
- Reproducibility for testing: pin it as low and fixed as you can, so behavior is stable enough to debug.
The knob isn't "make it smarter" — it's "how much should it gamble." Set it to match what the task can tolerate, and a whole class of flaky behavior disappears. More at www.divyakush.com.
Related reading
- Why LLMs hallucinate — and the patterns that actually stop it — why high randomness makes fabrication worse.
- Prompt engineering that actually works (and what does not) — the prompt patterns that measurably work.
- Guardrails: keeping LLM systems from going off the rails — making an LLM safe to expose to users.
Divyakush Punjabi · Full-Stack & AI Engineer
Portfolio · GitHub · LinkedIn
Top comments (0)