Tokenization: BPE from Scratch, Vocab Size, and What Tokenizers Hide

Tokenization: BPE from Scratch, Vocab Size, and What Tokenizers Hide

Before a language model reads a single word, a tokenizer has already decided how that word gets cut up. The model never sees letters. It never sees words either. It sees token IDs, numbers drawn from a fixed list, and everything the model can and cannot do starts with that list. We built byte pair encoding from scratch, trained it on 83,159 real words from our own published articles, and measured it against OpenAI's real tokenizers. The headline numbers: the same sentence costs 16 tokens in English and 60 in Urdu, a vocabulary stops paying for itself after about 5,000 merges, and the number 2026 is two tokens because the tokenizer chops digits into groups of three. This is Deep Part 2 of the AI Engineer Deep series, and it goes one layer deeper than Part 3 of the original series, How LLMs Actually Work, where we first showed that tokens, not words, are what you pay for.

The 30-second answer

Tokenization turns text into a sequence of integer IDs using a fixed vocabulary built by byte pair encoding, or BPE. BPE starts from single characters and repeatedly merges the most frequent adjacent pair into a new token until the vocabulary reaches its target size. Frequent words stay whole, rare words split into pieces, and any word at all can be represented because the vocabulary always contains single characters as a fallback. The size of that vocabulary is a real engineering tradeoff: a bigger vocabulary packs text into fewer tokens, which saves money and context, but every extra token adds a row to the embedding matrix, which costs parameters. And because the vocabulary is learned from data, it carries the biases of that data: English is cheap, Urdu is expensive, and a handful of tokens in every vocabulary were barely trained at all.

Where BPE came from

BPE started life as a compression algorithm. Philip Gage described it in 1994 as a way to squeeze data by repeatedly replacing the most frequent pair of bytes with a single unused byte. It sat in compression textbooks for twenty years until Rico Sennrich, Barry Haddow, and Alexandra Birch adapted it for neural machine translation in their 2016 paper, Neural Machine Translation of Rare Words with Subword Units. Their problem was concrete: word-level translation models had fixed vocabularies, so rare words became an unknown token and the model had nothing to say about them. BPE fixed that by letting any word be spelled out as a sequence of frequent pieces. On the WMT 2015 English to German and English to Russian tasks, their subword models beat a word-level model with a back-off dictionary by up to 1.1 and 1.3 BLEU. That is a modest number on paper and a large idea in practice: the same greedy merge loop now sits at the front of every major language model, from GPT-2 to Llama 3.

The experiment: BPE from scratch on the paper's own toy corpus

We implemented the exact algorithm from the Sennrich paper in plain Python, no libraries, and ran it on the paper's own example dictionary: low (5 times), lowest (2), newer (6), wider (3), new (2). The algorithm counts every adjacent symbol pair weighted by word frequency, merges the most frequent pair everywhere, and repeats. Ten merges in, the vocabulary already looks like a real tokenizer in miniature.

MergePair mergedResultCount
1e + rer9
2er + end of worder9
3n + ene8
4ne + wnew8
5l + olo7
6lo + wlow7
7new + ernewer6
8low + end of wordlow5
9w + iwi3
10wi + dwid3

After ten merges, the frequent words are whole tokens and the rare ones are pieces: low is one token, newer is one token, lowest splits into low plus e plus s plus t, and wider splits into wid plus er. That is the whole trick. Nothing in the algorithm knows English. It only knows frequency. The code is short enough to read in one sitting, and it is the same loop that builds production vocabularies, just with a smaller corpus and fewer merges.

import collections

def get_pair_stats(vocab, freq):
    pairs = collections.defaultdict(int)
    for word, symbols in vocab.items():
        for i in range(len(symbols) - 1):
            pairs[(symbols[i], symbols[i + 1])] += freq[word]
    return pairs

def merge_pair(pair, vocab):
    out = {}
    bigram = "".join(pair)
    for word, symbols in vocab.items():
        i, merged = 0, []
        while i "] for w in corpus}
for step in range(10):
    pairs = get_pair_stats(vocab, corpus)
    best = max(pairs, key=pairs.get)
    vocab = merge_pair(best, vocab)

The experiment: training BPE on 83,159 real words

A toy corpus proves the loop works. It says nothing about scale, so we trained the same code on every word in our own published articles: 83,159 words, 6,233 unique. Then we encoded a held out sentence the tokenizer had never seen, 25 words about tokenizers themselves, and measured how many tokens it took as the vocabulary grew.

MergesVocabulary sizeTokens for the held out sentenceCharacters per token
0271390.99
100127692.00
500527433.21
2,0002,027304.60
5,0005,027255.52
10,0009,958255.52

Two things stand out. First, the early merges do almost all the work: the first 100 merges cut the token count in half, and 2,000 merges get you to 4.60 characters per token. Second, the curve flattens hard. Going from 5,000 merges to 10,000 merges bought exactly zero fewer tokens on held out text, because the extra merges memorized quirks of our training articles instead of learning general pieces. A bigger vocabulary is not automatically a better tokenizer. It is a bigger vocabulary that has to earn its keep on text it has never seen. At 2,000 merges, our little tokenizer split unseen words the way a real one would: tokenizer became tokeniz plus er, unbelievable became un plus be plus li plus ev plus able, and transformers became trans plus form plus ers. The 45 letter word pneumonoultramicroscopicsilicovolcanoconiosis fell apart into 21 small pieces, which is exactly what should happen to a word the tokenizer has never met.

The experiment: OpenAI's real tokenizers on real text

Our toy BPE proves the mechanism. To see what production tokenizers actually do, we ran the same text through tiktoken, OpenAI's own BPE library, with three real encodings: r50k_base from GPT-2, cl100k_base from GPT-4, and o200k_base from GPT-4o. A 196 character paragraph about attention took 39 tokens under all three, or 5.03 characters per token. On plain English prose, three generations of tokenizer agree almost exactly. The differences show up the moment you leave English.

LanguageSame sentence, charactersTokens (cl100k_base)Characters per token
English73164.56
German86263.31
Chinese19280.68
Urdu63601.05

The same meaning costs 16 tokens in English and 60 in Urdu, a 3.75x multiplier, and Chinese needs 28 tokens for a 19 character sentence because each character carries a whole syllable of meaning and still splits into more than one token. This is not a rounding error. It is a price difference. Every API call is billed per token, every context window is measured in tokens, and a user writing in Urdu pays nearly four times as much as a user writing the same thing in English, for the same model and the same answer. Published analyses of token pricing across languages have found gaps as large as 10x for some languages. Our own measurement sits inside that range, and it is the reason newer vocabularies keep growing: o200k_base roughly doubles cl100k_base to about 200,019 tokens, and Llama 3 moved to 128,256 tokens, largely to buy back efficiency in non English text.

Numbers: the tokenizer chops digits into groups of three

Numbers get special treatment, and it explains a famous model weakness. Under cl100k_base, 123456789 becomes three tokens, 123 plus 456 plus 789. The year 2026 becomes two tokens, 202 plus 6. Pi to fourteen decimal places becomes seven tokens, 3 plus a dot plus 141 plus 592 plus 653 plus 589 plus 79. The model never sees a number as a number. It sees digit chunks of up to three, glued together by position. That is why a model can write a poem about arithmetic and still fumble a five digit addition: the digits it must add were never presented as a single unit. The same blindness shows up in spelling. Ask a model how many r letters are in strawberry, and remember that it never sees the word strawberry. It sees str plus aw plus berry, three tokens, and the letter r is smeared across token boundaries the model cannot inspect. The failure is not in the model's reasoning. It is in the input representation, and no amount of prompting fixes a representation problem.

Vocab size: the tradeoff nobody prices

Every token in the vocabulary costs parameters twice over: once in the input embedding matrix that turns IDs into vectors, and once in the output head that turns vectors back into probabilities over the vocabulary. For GPT-2 small, the embedding matrix is 50,257 tokens times 768 dimensions, about 38.6 million parameters, or 31.1 percent of the whole 124 million parameter model. For Llama 3 8B, the matrix is 128,256 tokens times 4,096 dimensions, about 525 million parameters, or 6.6 percent of the model. The share falls as models grow, but the absolute cost never goes away, and a vocabulary of 262,145 tokens like Gemma 3 pays for it in both matrices. This is the real reason vocabularies do not grow without limit. A bigger vocabulary compresses text better, which saves context and money on every request, but it also makes the model bigger, slower to load, and more expensive to train, and our sweep shows the compression gain flattens long before the parameter cost does. Picking a vocabulary size is picking a point on that curve, and most teams pick it once and live with it for the life of the model.

What tokenizers hide

Three things about tokenizers surprise working engineers, and all three follow from how BPE is actually built. First, tokenization happens before the model, and it happens in two steps: a regex first splits text into chunks, then BPE merges inside each chunk. The regex decides that a leading space belongs to the following word, which is why hello world is two tokens, hello and space plus world, and why tokenization with a leading space is two tokens, space plus token and ization. Change the space and you change the tokens, which is why the same word can tokenize differently at the start of a sentence and in the middle of one. Second, the tokenizer and the model are trained separately, on different data, and that gap creates glitch tokens. A string that was frequent in the tokenizer's training text earns its own token, but if it barely appears in the model's training text, its embedding never gets trained. The famous case is SolidGoldMagikarp, a Reddit username that appeared in so many counting threads that GPT-2's tokenizer gave it a token of its own, while the model itself almost never saw it in training. Asked to repeat it, GPT-2 era models produced evasions, insults, or unrelated text. We verified the fix with tiktoken: the old r50k_base encoding splits it into two tokens, Solid and GoldMagikarp, and both cl100k_base and o200k_base split it into five, Solid plus Gold plus Mag plus ik plus arp. A 2024 paper by Land and Bartolo, Fishing for Magikarp, showed these undertrained tokens are common across many open model families and can be found automatically. Third, tokenizers are frozen the day the model ships. You cannot add a token for your product name, your file format, or your language without retraining the model, so every prompt you write is at the mercy of merge decisions made months or years ago on someone else's corpus.

Our opinionated take

Most teams treat the tokenizer as plumbing and never look at it, and that is a mistake we think costs real money. If you build on top of language models, the tokenizer is the first thing you should measure, not the last. Count tokens before you count dollars: our paragraph of English cost 39 tokens, and one million tokens of English prose is roughly five megabytes of text, so a retrieval pipeline that stuffs whole documents into context is paying a token bill it never budgeted for. Test your prompts in the languages your users actually write, because a 3.75x token multiplier is a 3.75x cost multiplier and a 3.75x smaller effective context window, and no one will tell you that until the invoice arrives. And if you train anything yourself, do not chase the biggest vocabulary you can afford. Our sweep flattened at 5,000 merges on real text, and the production lesson is the same: the vocabulary that wins is the one trained on your data, sized to the point where the curve bends, not the biggest number on a spec sheet. The tokenizer is the cheapest part of the model to understand and the most expensive part to ignore.

What is next in this series

This is Deep Part 2 of the AI Engineer Deep series, a line by line revisit of the original 8 part series with real research and real measurements. Deep Part 3 takes on training LLMs: pretraining, loss curves, and scaling laws, including what a loss curve actually tells you and when more data stops helping. The original series still stands as the map: Deep Part 1, Transformers from Scratch is where attention was built by hand, Part 3, How LLMs Actually Work is where tokens and temperature first appeared, and Part 8 on LLMOps shows what the token bill looks like in production.

Previous: Transformers from Scratch: How Attention Actually Works

Next: Deep Part 3: Training LLMs: Pretraining, Loss Curves, and Scaling Laws (coming next in this series)

---

Originally published on LogicStack. Run the code examples right in your browser with our free interactive Python and SQL playgrounds — no setup, no sign-up.