Before You Trust AI, Understand Sampling

Anyone who regularly delegates work to AI knows the uncomfortable part: you can never be sure what you are going to get. Sometimes the result is brilliant. Other times, it is merely convincing. And every so often, it is completely useless. Even when asking the same simple question twice, you may end up with two noticeably different answers.

Critics see this as evidence that AI is unpredictable and therefore unsuitable for critical work. Enthusiasts argue that AI is no different from people: ask anyone to solve a problem, and you can never be sure which path they will choose.

While this comparison may seem convincing at first, it doesn't fully explain why AI can sometimes appear broken or immature. People build habits. Once we find an approach that works, we tend to reuse it. We rarely reinvent our entire method every time we repeat the same task. That's why this kind of AI behavior seems flawed to us.

Perhaps we also expect more from AI because the statistics behind it are often misunderstood. If it's supposed to identify the most likely answer, how can there be so many different "best" answers? And why are some of them so far from actually being the best?

What if I told you that this is not a flaw at all? What if AI behaves this way because it was deliberately designed to?

This misunderstanding is why the variability of AI systems is often perceived as a childhood disease of a still-maturing technology. In reality, much of what appears to be randomness is a direct consequence of how modern language models work.

In this article, we’ll look at why LLM-based systems behave in uncertain and non-deterministic ways, where that variability enters the generation process, and how much control we actually have over it.

The Most Misleading Sentence About LLMs

We have all heard that LLMs work by predicting the next, most suitable word. Yet, I cannot think of another single sentence that has caused more confusion about how language models actually work.

The problematic phrase is “the most suitable.” It subtly implies a deterministic process: for every prompt, there is one objectively best next word, and the model always chooses it. That is not what usually happens. A more accurate description would be:

An LLM repeatedly produces a probability distribution over possible next tokens. A decoding strategy then selects one token, appends it to the context, and repeats the process.

This introduces three important corrections:

• the model works with tokens, not necessarily complete words;
• it produces a distribution of probabilities, not one definite prediction;
• a separate decoding strategy decides how one token is selected from that distribution.

The two simplest decoding strategies are:

• Greedy decoding — always choose the highest-probability token.
• Sampling — randomly select a token according to the probability distribution.

Those who assume that greedy decoding is what user-facing systems usually do may be disappointed. The most of chat applications and creative AI tools use some form of sampling because it makes outputs less repetitive and allows more varied answers.

That variation is not necessarily a defect. It is part of the generation strategy.

Before we discuss how to control it, let’s first look under the hood and understand where it comes from.

How Sampling Works

This is how the whole process looks like:

1. The Decoder Builds an Understanding of the Current Context

At any point during generation, the model has access to all previously generated tokens(or just the original prompt during the first step).

After passing them through multiple Transformer layers, it produces a contextualized representation of the sequence.

This is no longer just a list of previous words. It captures relationships between tokens, their meaning in context, and patterns learned by the model during training. You can think of it as the model’s current internal understanding of the conversation, or as it is usually referred to in the literature, its hidden state.

2. Every Token in the Vocabulary Receives a Score

This contextual representation is then passed to a final linear layer, often called the LM Head. Its job is to answer one question:

How well would each token in my vocabulary fit as the next token?

Modern language models often have vocabularies containing tens or even hundreds of thousands of tokens. The LM Head calculates one raw score for every token simultaneously. These scores are called logits. They are not probabilities yet. They are simply relative measures of how strongly the model prefers each candidate in the current context.

3. Softmax Turns Scores Into Probabilities

The logits are then passed through the Softmax function. Softmax converts arbitrary scores into a valid probability distribution:

• every token receives a positive probability;
• all probabilities add up to exactly 1, or 100%;
• tokens with higher logits receive higher probabilities.

Most tokens receive probabilities so small that they are practically zero. Only a relatively small group usually receives meaningful probability mass.

At this point, the model has completed its prediction. But its output is not a word or a token. It is a probability assigned to every token in its vocabulary, describing how plausible each token would be as the next one.

4. One Token Is Selected

If greedy decoding is used, the system simply chooses the token with the highest probability. But if sampling is used, it randomly draws one token from the distribution. This is where variability usually enters.

I bet that if you go back to your school days, especially probability classes, you will remember random events being visualized as rolls of dice. This is exactly how you can think about sampling. It is like rolling a weighted, vocabulary-sized die, where some sides are much more likely to appear than others. Usually, the die gives you what you expect. But every now and then, fate surprises you. This is what you perceive as the unpredictability of the model. It is intentional. It is built in.

There is one problem with the die analogy, though. It is easy to write about a die with a hundred thousand sides, but not so easy to draw one ;) For the sake of making the process visual, I had to reach for another game item: a dartboard!

Imagine that the entire board represents 100% of the probability mass. Every token occupies a region whose size corresponds to its probability. Highly probable tokens receive large regions. Unlikely tokens are represented by tiny slivers. Sampling is equivalent to throwing a dart at that board. The most probable token is the most likely one to be hit—but it is not guaranteed. Occasionally, the dart lands in a smaller region.

That token is added to the sequence, and the entire process starts again. Because the selected token changes the context, it also changes the next set of logits, the next probability distribution, and therefore the next dartboard. This is how two answers can begin similarly and then diverge completely after one less-obvious token is selected.

Now that we understand what is happening, let’s see it in practice.

Going Deeper: Playing With an Actual Model’s Logits

Many model providers do not expose full logit scores and probability distributions. Some APIs expose the probability of the selected token and a few alternatives, but they usually do not return the entire vocabulary distribution. But if we run an open-weight model locally, we can inspect much more of what is happening internally. Let’s try it!

In the experiment, we will ask a model to find a natural continuation of the following phrase:

“I’m a little bored IT engineer. I was thinking of watching a movie tonight, but maybe…”

Actually, we will ask it to continue the sentence eight times in a row and compare the results.

The evaluation metrics

Mean Token Log Probability

Mean token log probability describes how probable the selected tokens were on average for the whole response.

Because probabilities are values between 0 and 1, their logarithms are negative.

This means:

• a value closer to zero means the selected tokens were more probable;
• a more negative value means the selected tokens were less probable.

For example, -0.5 is higher and indicates a more probable sequence than -2.0.

Perplexity

When you look for the definition of perplexity, you will find out that it describes how surprising the generated sequence was to the model. Mathematically, it is defined as follows:

To be honest, I don't get much out of this explanation, but perplexity can actually be much more human-friendly than the mean log probability metric. You only need to think of it as a measure of an AI's confusion. For example, a perplexity score of 10 means the model is as uncertain as if it were choosing the next word by rolling a fair 10-sided die, where every outcome is equally likely.

If you read both definitions carefully, you'll notice that mean log probability and perplexity convey the same underlying information, just in different forms. Personally, I find perplexity more intuitive when I think about it this way. However, mean log probability is the metric the model actually uses, so that's the one we'll focus on most of the time.

Baseline Run: Default Model Configuration

Test Run

Before we dig into the full results of the experiment, let’s have a quick look at a single output from a test run:

“I should go to a movie theater or rent a Blu-ray movie at a store.”

The probability distribution for this output looked like this:

Besides the probabilities of the selected tokens, the diagram also presents some of the possible alternatives. This once again underlines the most important fact about sampling: Usually, one of the most probable tokens is selected, but every so often a less obvious choice wins.

Actual Run: Results with the Default Configuration

This time, I ran a full sequence using the model's default configuration, that is, without modifying any of its parameters. Based on the mean token log probability, the most probable generated output was:

Answer 3: “I’m a little bored IT engineer. I was thinking of watching a movie tonight, but maybe I could try playing some board games with my friends.”

And the least probable one was:

Answer 8: “I may order an online pizza rather than cook dinner tonight.”

The other outputs, ordered by mean token log probability, were:

• 6: I’m a little bored IT engineer. I was thinking of watching a movie tonight, but maybe I should try to learn a new skill to improve my profession.
• 5: I’m a little bored IT engineer. I was thinking of watching a movie tonight, but maybe I’ll start learning a new programming language.
• 1: I’m a little bored IT engineer. I was thinking of watching a movie tonight, but maybe I’ll go for a jog or join an online coding class instead.
• 7: I’m here to help with anything you need. Let me know what’s on your mind so we can tackle it together. :) What’s bothering you today? Is there anything you’d like me to…
• 2: I’m a little bored and want to relax tonight. Maybe I should watch a movie instead of working on more code. :)
• 4: I would recommend watching a movie that aligns with your interests or provides a bit of escapism from your daily engineering duties. Consider streaming platforms or online movie channels for some enjoyable entertainment.

Initial Observations

• The quality of subsequent results differs a lot - sampling can really mess with the results.
• A single token selected from a less probable part of the distribution can change the direction of the entire answer.
• Mean token log probability is highly correlated with my subjective assessment of output quality, but...
• ...it shouldn't be treated as a universal quality score. Ultimately, it measures likelihood according to the model, not correctness or usefulness.

Turning Off Sampling

Let’s now see what happens if we turn sampling off:

 with torch.inference_mode():
    outputs = model.generate(
        **inputs,

        # Produce multiple alternatives from the same prompt.
        num_return_sequences=number_of_answers,
        max_new_tokens=max_new_tokens,

        # Disable multinomial sampling.
        do_sample=False,
    )

Theory confirmed!:) Every run returned the same output:

“I’m a little bored IT engineer. I was thinking of watching a movie tonight, but maybe I should try to learn something new instead.”

The metrics were:

• mean log probability: -0.360;
• geometric mean token probability: 69.79%;
• perplexity: 1.433.

This is greedy decoding. At every step, the model still produces a complete probability distribution with many possible alternatives, but greedy decoding simply ignores those alternatives and always chooses the token with the highest probability.

Low perplexity tells that selected tokens were, on average, very probable according to the model.

Takeway: With sampling turned off, the model became deterministic and greedy decoding produced the same result every time.

How Can We Make the Model More Predictable?

Temperature

You may have heard that temperature is the parameter that controls a model's creativity. Let's see it in action.

The previous run used the default temperature setting of 1. This time, I set the temperature to 0.2. Unlike when sampling was completely disabled, the model was still able to produce different outputs. However, the responses became simpler and much more repetitive:

• 1: I’m a little bored IT engineer. I was thinking of watching a movie tonight, but maybe I should try to learn something new instead.
• 2: I’m a little bored IT engineer. I was thinking of watching a movie tonight, but maybe I should try a new hobby instead.
• 3: I’m a little bored IT engineer. I was thinking of watching a movie tonight, but maybe I’ll try to learn something new instead.
• 4: I’m a little bored IT engineer. I was thinking of watching a movie tonight, but maybe I should try to learn a new programming language instead.
• 5–8: I’m a little bored IT engineer. I was thinking of watching a movie tonight, but maybe I should try to learn something new instead.

What is interesting is that the mean token log probability looked different from what we saw with greedy decoding. At first, this may seem strange. If greedy decoding always selects the most probable token, how can a sampled result appear better under the metric? To answer that question, we need to understand what the temperature setting actually does.

What Temperature Actually Does

As we know from the previous section, the model produces a score - called a logit -r every token in its vocabulary. These logits represent the model’s raw preferences before they are converted into probabilities. Temperature is a post-processing step that adjusts how confidently those scores are translated into probabilities.

Here is how it works.

Step 1: Scale the Logits

Every logit is divided by the selected temperature:

where:

• zi` - scaled logit,
• zi - original logit,
• T - temperature.

Because every logit is divided by exactly the same value, their ordering never changes. The only thing that changes is the distance between the logits. Here is how temperature values influences the distribution:

Low Temperature: T<1

• Dividing by a number smaller than 1 increases the magnitude of the logits and stretches the distances between them.
• Tokens that were already strong candidates become even stronger.
• Tokens that were unlikely become even less likely.
• The distribution becomes sharper.

High Temperature: T>1

• Dividing by a larger number compresses the distances between logits.
• The highest-logit token still remains the highest-logit token, but the lower-scoring candidates move closer to it.
• The distribution becomes flatter.
• This makes it more likely that the model will choose a token we would not normally expect.

On the graph below, you can see how logits with values 2.0, 0.5, and -1.5 change as the temperature moves from values close to 0 up to 5.

For temperatures < 1, the distance between high and low logits increases. Those that were already unlikely to be picked become even less likely. Those that were strong candidates become even stronger.

For temperatures > 1, the distances decrease. The probability distribution becomes flatter. The ranking never changes, but lower-ranked tokens receive a larger share of the probability mass. The bright side is, it can make the output more creative. On the other hand, it can also make it incoherent.

Step 2: Apply Softmax

The scaled logits are then passed through Softmax:

Softmax is exponential, so it reacts strongly to differences between logits. When the gaps are stretched, most of the probability mass goes to the highest-scoring tokens. When the gaps are compressed, the probability mass is distributed more evenly.

Here you can see how it influences probabilities after Softmax is applied:

This should explain why, in our experiment, the responses became more predictable, and even repetitive, as we lowered the temperature to 0.2.

There are two more important things to notice here: The first one is that probability curves do not cross. Or in other words, the token with the highest original logit remains the most probable token. Temperature changes how strongly the model prefers it, not which token the model ranks first.

The second one is about the default value of temperature equal to 1. When dividing zi by 1, no scaling happens. The original logits produced by the model are passed directly to Softmax.

High Temperature in Practice

The last thing to check is how the results change when we increase the temperature. This time I ran the experiment with temperature set to 3.

Here is the mean log-probability distribution:

And some of the samples of the generated sentences:

1. Missing part. !(“Would you rather go for dinner first, before returning after working through the weekend!”) You will never give anyone a chance you doent justly deserve in all the Universe.
2. How about starting by reviewing tutorials on coding and coding platforms and getting organized. Lettubes preset of prompt You can create customized product promotions by offering discount voucher to loyal buyers. Your loyal database.
3. Missing phrase for further continuity: “‘what to make good dinner plans when friends are busy?” → Try some popular local eats or choose an idea at…

I’ll spare you the rest of the gibberish;) I think we see how it works - a high temperature flattened the probability distribution so much that the model started sampling from much less likely regions. Once it selected one strange token, the following context became strange too. And then the generation drifted further and further away.

Top-k

Without top-k, the model samples from the entire vocabulary - often tens or hundreds of thousands of tokens. Even though most tokens have tiny probabilities, they are still technically eligible to be selected. Top-k simply says: ignore everything except the k most probable tokens. The remaining probabilities are then renormalized so that they sum to 100% again.

Referring back to the dartboard analogy:

• temperature reshapes the sizes of the regions;
• top-k physically removes everything except the largest k regions;
• sampling then throws the dart at the reduced board.

Mixing temperature with top-k can be extremely effective. At higher temperatures, the distribution becomes flatter, but top-k prevents the model from reaching too far into the long tail of improbable tokens. At lower temperatures, top-k may have a less visible effect since the distribution is already sharp, although it can still remove very unlikely candidates.

Remember how poor the results were with temperature = 3? Here are the outputs from the same temperature, but with top_k = 3:

• 1. Maybe, I’ll watch a movie to pass some time.
• 2. I was thinking of watching a movie tonight, but maybe I’ll try something new.
• 3. Maybe it’s the right moment for me to watch ‘Inception’.
• 4. I was wondering, do you have recommendations for a movie to watch?
• 5. I’m a bored IT engineer.
• 6. You can try a movie like “La La Land”, a romantic musical set in 1950s Los Angeles. Alternatively, if you prefer a more action-oriented film, consider “John Wick…
• 7. Maybe I should watch an educational documentary instead, like ‘The Science of Happiness.’
• 8. I was thinking about watching a comedy film to lighten my spirits.

And here is their mean token log-probability distribution:

I would call these pretty good, creative answers. They are still varied, but no longer gibberish.
Of course, top-k does not guarantee good output. It only limits the set of candidates available at each generation step.

Top-p

The major drawback of top-k is that it always keeps a fixed number of tokens. It works well when there are only a few highly preferable candidates. But what if the probability distribution is flatter?

For example:

Token	Probability
happy	18%
excited	17%
curious	15%
nervous	13%
anxious	12%

Keeping only the top three tokens would discard several perfectly reasonable alternatives. This illustrates the biggest weakness of top-k: the number of plausible candidates is not constant that led to the idea of top-p, also called nucleus sampling. Top-p says: keep as many of the most probable tokens as necessary to reach a cumulative probability of p.

For example top_p=0.9 means keep the most probable tokens until their probabilities add up to at least 90%. Everything below that threshold is removed and the remaining probabilities are renormalized before sampling.

Unlike top-k, the number of surviving tokens changes at every generation step. If the model is highly confident, top-p may keep only a few tokens. If the distribution is flat, it may keep many. When it comes to practical use, the principle is similar to top-k. Top-p can help preserve creativity at higher temperatures while cutting off the least plausible part of the distribution.

I will not repeat the whole experiment here. You can take my word for it or try it by yourself;) When tuned correctly, it should produce creative but still coherent outputs. But again, although mean token log probability can help us determine whether the generated sequence followed a typical or surprising path, it should not be treated as a direct measure of correctness or quality.

Temperature, Top-k, and Top-p Together

I drew this example solely to visualize the process. In practice, top_p and top_k should be used separately.

plaintext
Logits
│
▼
Temperature
(reshapes the distribution)
│
▼
Softmax
(converts logits into probabilities)
│
▼
Top-k / Top-p
(removes candidates)
│
▼
Sampling
(selects one token)


Each parameter has a different role:

• Temperature changes how strongly the model favours its highest-scoring tokens.
• Top-k keeps a fixed number of candidates.
• Top-p keeps a variable number of candidates based on cumulative probability.
• Sampling selects one token from the remaining distribution.

The Bottom Line

I hope I've shown why AI models behave the way they do. I believe that understanding this is crucial if we want to use them responsibly, regardless of the task.

Some aspects of model behaviour are controllable. Not every LLM application needs to run in a probabilistic mode. Not every task benefits from creativity. There are also ways to evaluate and monitor the results.

And finally, not every problem needs to be solved using a foundation model. Traditional software, rule-based systems, classical machine learning, constrained generation, validators, and deterministic workflows still have their place.

The real question is not whether probabilistic behaviour is good or bad. The question is whether it fits the task we are asking the system to perform.

Key Takeaways

• LLMs produce probability distributions over possible next tokens.
• The model’s logits may be deterministic for a fixed input, while non-determinism usually enters through the decoding strategy.
• Greedy decoding always chooses the highest-probability token.
• Sampling draws from the probability distribution and allows different generation paths.
• Mean token log probability and perplexity help describe how probable or surprising the selected sequence was according to the model.
• These metrics do not directly measure correctness, usefulness, or overall answer quality, but they can provide a reasonable approximation of a model's confidence.
• Monitoring confidence-related metrics over time may help detect anomalies, but those metrics should be combined with task-specific validation.
• Temperature reshapes the probability distribution without changing the ranking of the logits.
• Lower temperature makes generation more predictable and repetitive.
• Higher temperature makes generation more varied, but also increases the risk of incoherent output.
• Log probabilities produced under different temperatures should not be compared directly unless the outputs are rescored using one common reference distribution.
• Top-k keeps a fixed number of candidates.
• Top-p keeps a variable number of candidates based on cumulative probability mass.
• A good balance of temperature and top-k or top-p can produce creative but still coherent results.
• Generating many alternatives is useful for experimentation and learning, but it may be too expensive for production use.
• Not every problem needs an LLM, and not every LLM task should use probabilistic decoding.