Tokenization, Training Stages, and Why Bigger Models Work (LLM Internals, Part Three)

Tokenization, Training Stages, and Why Bigger Models Work

Written by Syed Muhammad Ali Raza

Two articles into this arc, we've built attention from scratch and trained a tiny network with our own backpropagation code. Both articles quietly skipped over three questions that keep coming up in the comments, how does text actually become the numbers a model works with in the first place, what's the actual difference between pretraining, fine-tuning, and RLHF, terms people throw around like they're obviously different things, and why does making a model bigger seem to just make it better, almost like a cheat code. This article answers all three, together, because they genuinely connect to each other more than you'd expect.

Part one, tokenization, how text becomes numbers

A real life example first

Think about how you'd teach a child to read before they know the alphabet. You wouldn't hand them a dictionary with every possible English word as a separate entry, that's an impossibly long list, and it would miss every new word ever invented after the dictionary was printed. Instead, you teach letters, then common letter combinations, then whole common words, building up a flexible toolkit that can represent literally any word, even ones the child has never seen, by combining smaller known pieces.

That's genuinely the problem tokenization solves. A model needs a fixed, finite vocabulary of chunks it understands, but human language has effectively infinite possible words, new slang, names, typos, made up words, technical terms. Break everything down to individual letters, and your fixed vocabulary is tiny and flexible, but every sentence becomes an enormous sequence of single characters, expensive and hard to learn patterns from. Use whole words, and your vocabulary needs to be gigantic to cover everything, and it still breaks the moment it hits a word it's never seen. Tokenization is the actual compromise, chunks bigger than single letters, smaller than whole words, chosen specifically to balance vocabulary size against sequence length.

How the actual algorithm works, byte pair encoding

The most common approach, called byte pair encoding, or BPE, builds this chunk vocabulary in a genuinely simple, almost mechanical way. Start with every individual character as its own token. Look through a huge amount of text and find the single most frequently occurring pair of adjacent tokens. Merge that pair into one new token. Repeat, over and over, thousands of times, each time merging whatever pair is currently most common. Do this enough times, and you end up with a vocabulary containing individual characters for genuinely rare combinations, and larger chunks, often whole common words, for things that show up constantly.

Let's actually build a tiny version of this algorithm, so you see the mechanism rather than just the description.

from collections import Counter

def get_pair_frequencies(word_freqs):
    pairs = Counter()
    for word, freq in word_freqs.items():
        symbols = word.split()
        for i in range(len(symbols) - 1):
            pairs[(symbols[i], symbols[i + 1])] += freq
    return pairs


def merge_pair(pair, word_freqs):
    new_word_freqs = {}
    bigram = " ".join(pair)
    replacement = "".join(pair)
    for word, freq in word_freqs.items():
        new_word = word.replace(bigram, replacement)
        new_word_freqs[new_word] = freq
    return new_word_freqs


# start with a tiny toy corpus, each word split into characters,
# with a special end of word marker
word_freqs = {
    "l o w </w>": 5,
    "l o w e r </w>": 2,
    "n e w e s t </w>": 6,
    "w i d e s t </w>": 3
}

num_merges = 8
for i in range(num_merges):
    pairs = get_pair_frequencies(word_freqs)
    if not pairs:
        break
    best_pair = max(pairs, key=pairs.get)
    word_freqs = merge_pair(best_pair, word_freqs)
    print(f"Merge {i + 1}, combined {best_pair} into '{''.join(best_pair)}'")

print("\nFinal vocabulary state:")
for word in word_freqs:
    print(f"  {word}")

Run this and watch what happens, common patterns like "e s t" for words ending in "est," get merged into single chunks early, because they show up frequently across "newest" and "widest." Rare, one off letter combinations stay as separate characters, because merging them wouldn't save much. This is genuinely the exact algorithm, at a toy scale, behind the tokenizers used in real models, just run on billions of words instead of four, producing a vocabulary of tens of thousands of chunks instead of the handful we generated here.

Why this explains weird model behavior you've probably seen

This is genuinely useful to know because it explains real quirks. Models sometimes struggle with letter counting tasks, like counting the number of r's in "strawberry," because the model doesn't actually see individual letters most of the time, it sees whichever token chunks "strawberry" happened to get split into during tokenization, which might not align with letter boundaries you'd expect at all. It also explains why the same model can handle common English words efficiently but chews through more tokens, and therefore more cost, on rare words, made up terms, or non English text, since less common patterns didn't earn themselves an efficient single token chunk during that merging process.

Part two, the training stages, pretraining, fine-tuning, and RLHF

A real life example

Think about how someone becomes a doctor. First, years of general medical school, absorbing an enormous, broad foundation, anatomy, biochemistry, pharmacology, general medical knowledge across every field, without yet specializing in anything. That's genuinely the biggest, longest phase, building broad general competence.

Then, residency, specializing in a specific area, say emergency medicine, working under supervision, being shown specifically how to actually apply that broad knowledge to real emergency room situations, following established protocols, learning the specific behaviors and judgment calls that general medical school didn't cover in that kind of depth.

Then, ongoing feedback throughout an actual career, senior doctors and patient outcomes providing continuous signal about what worked and what didn't, gradually refining judgment in ways that go beyond textbook protocol, developing the kind of intuition that comes specifically from feedback on real decisions.

That three stage arc, broad general education, specialized applied training, then ongoing feedback driven refinement, maps almost exactly onto how a modern LLM actually gets built.

Pretraining, the medical school stage

This is the stage the last article's training loop was demonstrating, at a toy scale. The model trains on an enormous amount of raw text, predicting the next token, over and over, across a huge, broad slice of the internet, books, articles, code, conversations, essentially everything. There's no specific task here, no instructions being followed, just the single objective from the last article, minimize the loss of predicting the next token, applied at a genuinely massive scale.

def pretraining_objective_conceptual(text_sequence):
    # simplified conceptual view, real pretraining does this
    # across billions of sequences, not one
    loss = 0
    for i in range(1, len(text_sequence)):
        context = text_sequence[:i]
        actual_next_token = text_sequence[i]
        predicted_probabilities = model_predict_next_token(context)
        loss += cross_entropy(predicted_probabilities, actual_next_token)
    return loss

This stage is genuinely where the vast majority of a model's raw capability comes from, language patterns, world knowledge, reasoning patterns, all absorbed from predicting next tokens across an enormous, broad corpus. But a model that's only been through this stage tends to just continue text in whatever style it started, it hasn't specifically learned to behave like a helpful assistant answering your questions, exactly like a fresh medical school graduate who knows an enormous amount but hasn't yet learned the specific behavior of "how a doctor actually interacts with a patient in an exam room."

Fine-tuning, the residency stage

This connects directly to an entire earlier article in the previous series of this collection, but here's the training mechanics view of it. Instead of raw, broad internet text, the model now trains on a much smaller, curated set of examples specifically showing the desired behavior, instructions paired with genuinely good responses, question and answer pairs, conversations demonstrating exactly how a helpful assistant should respond.

fine_tuning_examples = [
    {
        "instruction": "Explain photosynthesis simply",
        "good_response": "Photosynthesis is how plants turn sunlight into food..."
    },
    {
        "instruction": "Write a professional email declining a meeting",
        "good_response": "Subject: Re: Meeting Request\n\nThank you for the invitation..."
    }
    # a real fine-tuning dataset has many thousands of these,
    # far fewer than pretraining's raw text, but far more targeted
]

Same underlying mechanism, forward pass, loss, backpropagation, gradient descent, but now the loss is measuring "how far off was this response from the specific good behavior we're demonstrating," not "how well did you predict generic internet text." This is genuinely the residency stage, taking that broad pretrained foundation and specifically shaping it toward the actual behavior you want.

RLHF, the ongoing feedback stage

Reinforcement learning from human feedback is genuinely a different mechanism from the previous two, worth understanding as its own thing. Instead of training directly on "here's the exact right answer," which is often genuinely hard to write out for subjective qualities like helpfulness or tone, humans instead compare pairs of model responses and say which one they prefer.

comparison_examples = [
    {
        "prompt": "How do I politely decline a party invitation?",
        "response_a": "Just say no, easy.",
        "response_b": "You could say something like, 'Thank you so much for thinking of me, I won't be able to make it this time, but I hope you have a wonderful time.'",
        "human_preference": "response_b"
    }
]

These preference comparisons train a separate reward model, essentially a model that learns to predict which kind of response humans would prefer, and then that reward signal gets used to further adjust the original model, again through the same fundamental gradient descent mechanism, nudging it toward producing responses more like the ones humans consistently preferred. This is genuinely the "ongoing feedback from senior doctors and patient outcomes" stage, refining judgment on qualities that are much easier to recognize and compare than to explicitly write out as a single correct answer.

Part three, why bigger models just seem to work better

A real life example

Think about the difference between a small local library and a massive national archive. The small library has maybe a few thousand books, genuinely useful, but ask an obscure question and there's a real chance nothing in that collection actually covers it. The massive archive has millions of documents, and even genuinely obscure, rarely asked questions have a real chance of being covered by something in that vast collection, simply because there's so much more material for a rare pattern to have shown up in even once.

Bigger models are genuinely something like that archive, more parameters means more capacity to store more distinct patterns learned from training data, and more training data means more chances for even rare, subtle patterns to show up often enough to actually get learned. Neither alone is the whole story, a huge archive with nothing in it is useless, and a small library that's read extremely carefully can still be quite good, which is exactly why both model size and training data amount matter together, not independently.

Scaling laws, the actual observed pattern

Researchers found something genuinely striking, empirically, plot model performance against model size, training data amount, and compute used, on a specific kind of chart, log scale on both axes, and you get a remarkably smooth, predictable line. Bigger, trained on more data, with more compute, reliably performs better, in a way that's predictable well before you actually finish training a specific model, letting researchers estimate roughly how a much larger model would perform before spending the money to actually build it.

import math

def rough_scaling_law_intuition(model_size_params, training_tokens):
    # a deliberately simplified illustrative version, real scaling
    # laws are empirically fit curves from actual research, not this
    # exact formula, but this captures the qualitative shape
    loss = 1.0 / math.log(model_size_params) + 1.0 / math.log(training_tokens)
    return loss


small_model_loss = rough_scaling_law_intuition(model_size_params=1_000_000, training_tokens=1_000_000)
large_model_loss = rough_scaling_law_intuition(model_size_params=100_000_000_000, training_tokens=1_000_000_000_000)

print(f"Small model illustrative loss: {small_model_loss:.4f}")
print(f"Large model illustrative loss: {large_model_loss:.4f}")
print("Lower loss means better performance, larger scale genuinely tends to help, predictably")

Emergent behavior, the genuinely surprising part

Here's the part that goes beyond just "bigger is smoothly better," and it's genuinely one of the stranger findings in this whole field. Some capabilities don't show up gradually as models scale up, they show up suddenly, past a certain size threshold, a model that couldn't reliably do a specific kind of multi step reasoning at all suddenly can, once it crosses some scale point, not gradually improving at it beforehand, genuinely closer to a switch flipping than a smooth ramp.

Nobody fully agrees on exactly why this happens, but the practical takeaway is genuinely important, you can't always predict a model's specific capabilities just by smoothly extrapolating a smaller model's behavior upward. This is part of why each new generation of larger models sometimes surprises even the researchers building them, and it's a big part of why scale, while not the only thing that matters, has remained such a central focus in this field.

Bringing all three pieces together

Here's how these three topics genuinely connect, not just three separate facts. Tokenization determines the actual units the model is predicting during that next token loss from the training article, get tokenization wrong and even a perfectly trained model struggles with tasks that don't align well with how text got chunked. The three training stages are all running the same backpropagation and gradient descent mechanism from the previous article, just with different data and different loss signals at each stage, broad next token prediction, then curated good examples, then human preference comparisons. And scaling laws describe what happens when you take that entire pipeline, tokenization, architecture, and training stages, and simply do more of it, bigger vocabulary handling, bigger models, more training data, more compute, with performance improving in a genuinely predictable way, occasionally punctuated by capabilities that emerge suddenly rather than gradually.

What's next in this arc

We've now covered attention, the mechanism for relating tokens to each other, training, the mechanism for learning weights from data, tokenization, the mechanism for turning text into the units models actually work with, and scale, why bigger consistently tends to help. Genuinely, at this point, you understand the core mechanics behind how every model in the entire previous series of this collection actually works, from the ground up. Where this arc goes next is an open question, and honestly a good one to point at whatever's still nagging at you about how these models work underneath.

---

If you run the BPE code on your own text, a different language, code, whatever you're curious about, I'd genuinely like to hear what merges show up, that's usually where this stuff stops feeling abstract and starts feeling like an actual mechanism you understand.