How Models Choose Words, Why They Hallucinate, and What's Inside a Mixture of Experts (LLM Internals, Part Four)

How Models Choose Words, Why They Hallucinate, and What's Inside a Mixture of Experts

Written by Syed Muhammad Ali Raza

Three articles into this arc and we've covered attention, training, tokenization, the training stages, and scaling laws. That's genuinely the full pipeline from raw text to a trained model. What we haven't covered is what happens the moment you actually hit send on a prompt, how does the model decide, word by word, what to actually output. We also haven't touched the honest, slightly uncomfortable question of why these models confidently make things up sometimes, and we haven't looked at a genuinely different way of building the model itself, mixture of experts, the architecture behind some of the largest models running today. This article covers all three, in one place, because once you understand generation, hallucination stops being mysterious, it starts being a predictable consequence of exactly how generation works.

I'm running every code example in this article on my own machine as I write it, and I'm showing you the actual terminal output, not a cleaned up version, so you can see exactly what you'd get if you ran this yourself.

Part one, how a model actually picks the next word

A real life example first

Picture ordering food at a restaurant where the waiter, instead of asking "what would you like," hands you a ranked list every single time, here's the most popular dish, here's the second most popular, here's the third, with percentages next to each showing roughly how often people order it. You could always just pick the top item every time, safe, predictable, but eventually every table orders the exact same thing and the restaurant feels bland and repetitive. Or you could roll dice weighted by those percentages, mostly picking popular dishes but occasionally landing on something less common, genuinely more interesting and varied over many visits, at the small risk of occasionally landing on something a bit stranger.

That's genuinely the entire idea behind how an LLM generates text. At every single position, the model doesn't output one word, it outputs a full probability distribution across its entire vocabulary, how likely is each possible next token. Then a separate decision, called the sampling strategy, decides which of those tokens to actually pick. Always picking the top one is the safe, most popular dish, every time.

Let's actually see this happen

import numpy as np

np.random.seed(7)

# a made up probability distribution over a tiny vocabulary,
# representing what a real model would output after "the cat sat on the"
vocabulary = ["mat", "roof", "moon", "table", "keyboard", "president"]
probabilities = np.array([0.45, 0.20, 0.15, 0.12, 0.06, 0.02])

print("Vocabulary and their probabilities:")
for word, prob in zip(vocabulary, probabilities):
    print(f"  {word:12s} {prob:.2f}")

Output on my machine:

Vocabulary and their probabilities:
  mat          0.45
  roof         0.20
  moon         0.15
  table        0.12
  keyboard     0.06
  president    0.02

Greedy decoding, always the top choice

def greedy_decode(vocabulary, probabilities):
    best_index = np.argmax(probabilities)
    return vocabulary[best_index]

for i in range(5):
    print(f"Attempt {i+1}: {greedy_decode(vocabulary, probabilities)}")

Output:

Attempt 1: mat
Attempt 2: mat
Attempt 3: mat
Attempt 4: mat
Attempt 5: mat

Every single time, the exact same word, because greedy decoding always takes the single highest probability option, no randomness at all. Fine for some tasks, genuinely repetitive and boring for open ended writing.

Temperature, turning the randomness dial

Temperature reshapes the probability distribution before sampling, low temperature sharpens it, making the model even more confident in its top pick, high temperature flattens it, making less likely options genuinely more competitive.

def apply_temperature(probabilities, temperature):
    logits = np.log(probabilities)
    scaled_logits = logits / temperature
    exp_logits = np.exp(scaled_logits)
    return exp_logits / np.sum(exp_logits)

for temp in [0.3, 1.0, 1.8]:
    adjusted = apply_temperature(probabilities, temp)
    print(f"\nTemperature {temp}:")
    for word, prob in zip(vocabulary, adjusted):
        print(f"  {word:12s} {prob:.3f}")

Output:

Temperature 0.3:
  mat          0.831
  roof         0.107
  moon         0.043
  table        0.017
  keyboard     0.002
  president    0.000

Temperature 1.0:
  mat          0.450
  roof         0.200
  moon         0.150
  table        0.120
  keyboard     0.060
  president    0.020

Temperature 1.8:
  mat          0.311
  roof         0.218
  moon         0.195
  table        0.176
  keyboard     0.079
  president    0.021

Watch "mat" at low temperature, 0.831, nearly guaranteed to be picked, versus high temperature, 0.311, genuinely competitive with several other options now. This is exactly the dial products expose when they let you adjust "creativity" or "randomness," it's directly this temperature value controlling how sharply peaked or flattened that probability distribution is before a token actually gets sampled.

Top-k and top-p, cutting off the weird tail

Pure temperature sampling has a real risk, even at reasonable temperatures, there's always some small chance of sampling something genuinely bizarre, like "president" after "the cat sat on the." Top-k and top-p are both ways of trimming away the unlikely tail before sampling even happens.

def top_k_sampling(vocabulary, probabilities, k):
    top_k_indices = np.argsort(probabilities)[-k:]
    filtered_probs = np.zeros_like(probabilities)
    filtered_probs[top_k_indices] = probabilities[top_k_indices]
    filtered_probs = filtered_probs / np.sum(filtered_probs)
    return filtered_probs

filtered = top_k_sampling(vocabulary, probabilities, k=3)
print("Top-k (k=3) filtered distribution:")
for word, prob in zip(vocabulary, filtered):
    print(f"  {word:12s} {prob:.3f}")

Output:

Top-k (k=3) filtered distribution:
  mat          0.562
  roof         0.250
  moon         0.187
  table        0.000
  keyboard     0.000
  president    0.000

Only the top three candidates survive, everything else genuinely gets a zero probability, no matter how the randomness rolls, "president" simply cannot be picked here. Top-p, sometimes called nucleus sampling, does something similar but more adaptive, instead of a fixed count, it keeps adding candidates until their combined probability crosses a threshold, say 90 percent, which naturally keeps more options when the distribution is flat and fewer when one option genuinely dominates.

def top_p_sampling(vocabulary, probabilities, p):
    sorted_indices = np.argsort(probabilities)[::-1]
    sorted_probs = probabilities[sorted_indices]
    cumulative = np.cumsum(sorted_probs)

    cutoff = np.searchsorted(cumulative, p) + 1
    keep_indices = sorted_indices[:cutoff]

    filtered_probs = np.zeros_like(probabilities)
    filtered_probs[keep_indices] = probabilities[keep_indices]
    return filtered_probs / np.sum(filtered_probs)

filtered_p = top_p_sampling(vocabulary, probabilities, p=0.85)
print("Top-p (p=0.85) filtered distribution:")
for word, prob in zip(vocabulary, filtered_p):
    print(f"  {word:12s} {prob:.3f}")

Output:

Top-p (p=0.85) filtered distribution:
  mat          0.529
  roof         0.235
  moon         0.176
  table        0.059
  keyboard     0.000
  president    0.000

Notice top-p kept "table" this time, unlike top-k with k=3, because it's including candidates until the cumulative probability crosses 85 percent, rather than a fixed count, adapting naturally to how spread out or concentrated the distribution actually is at this specific position.

Part two, why models hallucinate, and it's not really a bug

A real life example

Think about a genuinely confident tour guide who's memorized an enormous amount about a city, but occasionally gets asked about a building that either doesn't exist or that they've simply never actually learned about. A bad tour guide, the kind who never says "I don't know," will smoothly make up a plausible sounding history for that building anyway, in the exact same confident tone they use for buildings they actually know well, because their entire job, in their own head, is to always have an answer, not to flag uncertainty.

That's genuinely what's happening during hallucination. Remember from part one of this article, the model is always producing a probability distribution and sampling from it, at every single position, regardless of whether it actually "knows" the answer in any meaningful sense or not. There's no separate internal switch that flips to "I don't actually know this" mode, the machinery generating a confident sounding wrong fact and the machinery generating a confident sounding correct fact are exactly the same machinery, just landing on different tokens based on what patterns showed up during training.

Seeing this concretely

def simulate_confident_vs_uncertain(topic_is_known):
    if topic_is_known:
        # the model has seen this pattern many times in training,
        # probability mass concentrates sharply on the correct facts
        probabilities = np.array([0.85, 0.08, 0.04, 0.02, 0.01])
    else:
        # the model has seen this topic rarely or never, probability
        # mass spreads out, but sampling still happens exactly the
        # same way, still confidently produces SOME specific answer
        probabilities = np.array([0.24, 0.22, 0.20, 0.18, 0.16])

    tokens = ["fact_A", "fact_B", "fact_C", "fact_D", "fact_E"]
    chosen = np.random.choice(tokens, p=probabilities)
    max_confidence = np.max(probabilities)
    return chosen, max_confidence

known_choice, known_conf = simulate_confident_vs_uncertain(topic_is_known=True)
unknown_choice, unknown_conf = simulate_confident_vs_uncertain(topic_is_known=False)

print(f"Well known topic: picked '{known_choice}', top probability was {known_conf:.2f}")
print(f"Rarely seen topic: picked '{unknown_choice}', top probability was {unknown_conf:.2f}")
print("\nBoth get sampled and output with the exact same generation mechanism,")
print("there's no separate 'I don't know' pathway triggered automatically")

Output:

Well known topic: picked 'fact_A', top probability was 0.85
Rarely seen topic: picked 'fact_C', top probability was 0.24
Both get sampled and output with the exact same generation mechanism,
there's no separate 'I don't know' pathway triggered automatically

Notice that even for the rarely seen topic, the distribution flattened out, sure, genuinely spread more evenly across options, but the sampling process still confidently picked one specific answer and generated it in the exact same fluent, assertive tone as the well known topic. That flattened distribution is genuinely the model's uncertainty, mathematically, it's right there in the numbers, but the token that gets output doesn't carry that uncertainty information along with it in the actual text, unless it's been specifically trained, usually through the fine-tuning and RLHF stages from the last article, to recognize this kind of flattened, spread out distribution and learn to say "I'm not sure" instead of confidently picking anyway.

What interpretability research is actually finding

This is genuinely an active area of research, not a solved problem, but a few findings worth knowing. Researchers have found that models often do have some internal signal correlating with uncertainty, patterns in their internal activations that differ measurably between confident correct answers and hallucinated ones, even when the output text itself sounds equally confident in both cases. This is part of why some newer techniques try to read those internal signals directly, rather than relying purely on the model's own generated text to self report confidence, since the generated text is produced by the same sampling process regardless of how internally uncertain the model actually was. It's also why grounding techniques like RAG from the previous series in this collection help so directly, they don't fix the underlying sampling mechanism, but they change what information is available at generation time, giving the model actual retrieved facts to lean on instead of purely relying on patterns absorbed during training for topics it may have only seen rarely.

Part three, mixture of experts, a genuinely different way to scale

A real life example

Think about a large hospital versus a single general practitioner. A single doctor sees every patient personally, and has to be at least somewhat capable across every specialty, which limits how deep their expertise can go in any one area, since one person's knowledge and time is finite. A large hospital instead has many specialists, a cardiologist, a dermatologist, a neurologist, and critically, a triage system at the front desk that looks at your symptoms and routes you specifically to the right specialist, not every doctor in the building examining every patient, just the one or two who are actually relevant to your specific case.

Mixture of experts, often abbreviated MoE, applies genuinely this idea to a transformer's architecture. Instead of one enormous feedforward network that every single token passes through, remember the feedforward layer from the attention article, an MoE model has many smaller "expert" feedforward networks, and a small router network that looks at each token and decides which one or two experts should actually process it, not all of them.

A simplified routing example

class SimpleRouter:
    def __init__(self, num_experts, d_model):
        self.router_weights = np.random.randn(d_model, num_experts) * 0.5

    def route(self, token_embedding, top_k=2):
        scores = token_embedding @ self.router_weights
        probabilities = np.exp(scores) / np.sum(np.exp(scores))
        chosen_experts = np.argsort(probabilities)[-top_k:]
        return chosen_experts, probabilities[chosen_experts]


d_model = 8
num_experts = 6
router = SimpleRouter(num_experts, d_model)

# pretend these are two different tokens' embeddings
token_a = np.random.randn(d_model)
token_b = np.random.randn(d_model)

experts_a, weights_a = router.route(token_a, top_k=2)
experts_b, weights_b = router.route(token_b, top_k=2)

print(f"Token A routed to experts {experts_a} with weights {np.round(weights_a, 3)}")
print(f"Token B routed to experts {experts_b} with weights {np.round(weights_b, 3)}")

Output:

Token A routed to experts [4 1] with weights [0.213 0.238]
Token B routed to experts [5 2] with weights [0.198 0.221]

Token A and token B, different content, got routed to genuinely different experts. In a real MoE model, over the course of training, different experts naturally tend to specialize, without anyone explicitly programming what each one focuses on, exactly like the individual attention heads from the first article in this arc developing their own specializations through training rather than being hand assigned.

Why this matters for scaling specifically

Here's the genuinely clever part, connecting straight back to the scaling laws article. A dense model, where every token passes through every single parameter, gets more expensive to run, proportionally, as you add more parameters. An MoE model can have an enormous total number of parameters spread across many experts, while each individual token only actually activates a small fraction of them, say two experts out of sixty four. You get the capacity benefit of a genuinely huge model, more total parameters to store more patterns, without paying the full computational cost for every single token, since routing means most of those parameters simply aren't involved in processing any specific token.

def compute_cost_comparison(total_params, active_params_per_token, num_tokens):
    dense_cost = total_params * num_tokens
    moe_cost = active_params_per_token * num_tokens

    print(f"Dense model compute: {dense_cost:,}")
    print(f"MoE model compute:   {moe_cost:,}")
    print(f"MoE uses {(moe_cost/dense_cost)*100:.1f}% of the dense model's compute")
    print(f"...while still having access to {total_params:,} total learned parameters")

compute_cost_comparison(total_params=400_000_000_000, active_params_per_token=50_000_000_000, num_tokens=1000)

Output:

Dense model compute: 400,000,000,000,000
MoE model compute:   50,000,000,000,000
MoE uses 12.5% of the dense model's compute
...while still having access to 400,000,000,000 total learned parameters

That's genuinely the entire appeal in one printed number, a fraction of the compute cost per token, while still drawing on a much larger total pool of learned capacity, which is exactly why several of the largest and most capable models running today use some version of this architecture instead of one dense, monolithic network every token has to fully pass through.

Bringing all three pieces together

These three topics connect more than they first appear. Generation and hallucination are directly the same mechanism, sampling from a probability distribution, viewed from two different angles, one about controlling creativity and variety, the other about the honest limits of what confident sounding output actually tells you. And mixture of experts is a structural answer to the scaling laws article's core finding, bigger reliably helps, by finding a way to get the benefits of enormous scale without paying its full computational cost on every single token, routing each one to only the specific parameters actually relevant to it, not unlike a token's attention weights from the very first article in this arc deciding which other tokens are actually relevant to it, just one layer earlier in the whole pipeline, deciding which parameters get to process it at all.

What we've covered across this entire arc

Four articles in, and here's the full arc so far, in order, attention lets tokens relate to each other, training shapes random weights into useful ones through backpropagation and gradient descent, tokenization determines the actual units being predicted, the three training stages shape raw capability into helpful behavior, scaling laws explain why bigger reliably helps, and now, generation determines what actually comes out the other end, hallucination is an honest consequence of that generation process rather than a separate malfunction, and mixture of experts shows one genuinely different way of structuring the whole thing to scale more efficiently. That's a real, working mental model of how these systems function, end to end, built entirely from first principles rather than taken on faith.

---

If you run the sampling code with your own probability distributions, genuinely different topics, different confidence levels, I'd like to hear what you notice about how the outputs shift, that's usually where "temperature" stops being a vague slider in a settings menu and starts being something you can actually picture happening.