AI Engineering Illustrated

Posted on Oct 13 Originally published at ragmehta.substack.com

Why Can't LLMs Count the R's in "Strawberry"? How Tokenizers Work

#ai #machinelearning #nlp #llm

The preprocessing step behind spelling slips, shaky arithmetic, glitch tokens and your API bill

Ask a language model how many r's are in "strawberry" and, for a long time, the popular answer was "two." The same models could write a sonnet about strawberries, explain their genetics and translate the word into forty languages. So why does counting letters go wrong?

The model never sees the letters. Before your prompt reaches the first transformer layer, a separate program called the tokenizer chops the text into chunks and swaps each chunk for an integer. The model only ever reads those integers. If "strawberry" becomes a single integer, the three r's are hidden inside a number the model can't take apart.

Tokenization is the least glamorous part of the LLM stack, and it causes a surprising number of problems: shaky multi-digit arithmetic, higher costs for non-English users, strange "glitch" words that make models act up, and token counts that don't match your character-based estimates. In this issue we'll open the tokenizer up: how it turns text into IDs, how Byte Pair Encoding (BPE) learns its vocabulary, and how its choices show up as spelling, arithmetic, language and glitch-token problems you can design around.

Every token split in this issue was measured with OpenAI's open-source tiktoken library on the actual GPT-2, GPT-4 (cl100k_base) and GPT-4o (o200k_base) vocabularies, so you can reproduce them yourself.

Tokens: the Model's Only Alphabet

A neural network can't do math on strings. It needs numbers. The simplest scheme would give every character a number, but then a 2,000-word document becomes more than 10,000 positions, and every transformer layer pays for each one. The other extreme gives every word a number, but no fixed list covers every word, name, typo, URL and code identifier people will ever type.

Rico Sennrich and colleagues measured this trade-off in their 2016 paper Neural Machine Translation of Rare Words with Subword Units. On a German training corpus of about 100 million words, keeping whole words needed 1.75 million vocabulary entries and still left 1,079 unknown words in the test set. Splitting into characters needed only about 3,000 symbols but stretched the corpus to 550 million positions. Their subword method, BPE, landed in between: about 63,000 symbols, 112 million positions and zero unknown words.

That middle ground is a token: a chunk of text that's usually shorter than a word for rare words and exactly one word for common ones. Modern LLMs use vocabularies of 50,000 to 200,000 tokens. tiktoken's README puts the average at about 4 bytes of text per token. On the English text of last week's issue I measured 4.4 characters per token with the GPT-4o tokenizer, or roughly 0.63 words per token.

From Text to Token IDs in Five Steps

From raw text to token IDs: pre-split, apply BPE merges, look up IDs, then embed

Here's what happens to the sentence "I love strawberry" on its way into GPT-4o:

Raw text. Your string arrives as UTF-8 bytes. Some tokenizers normalize it first (Unicode normalization, lowercasing in older models), but GPT-style tokenizers keep the bytes exactly as typed.
Pre-split. A regular expression cuts the text into pieces that merges are never allowed to cross: letter runs, digit runs, punctuation, whitespace. The tiktoken source shows the patterns. GPT-2's version keeps a leading space attached to the following word, so the pieces are I, love, strawberry. That's why the dot in the figure matters: " strawberry" with a space and "strawberry" without one are different tokens.
BPE merges. Each piece starts as a sequence of bytes. The tokenizer then applies its learned merge rules, in the order they were learned, gluing adjacent pairs together until no rule applies. Common words collapse into one token. Rare ones end up as a few subword chunks.
Token IDs. Each final chunk is looked up in the vocabulary. With o200k_base, "I love strawberry" becomes [40, 3047, 101830]: three integers.
Embeddings. Inside the model, each ID selects one row of a learned embedding matrix, a vector of a few thousand numbers. From here on, the transformer only ever works with those vectors.

Two properties fall out of this design, and the tiktoken README calls both out. Tokenization is lossless: decoding the IDs gives back the exact original bytes. And it works on any text, even strings that never appeared when the tokenizer was trained. The second property depends on a trick we'll get to in a moment.

How BPE Learns Its Vocabulary

Byte Pair Encoding started life as a data compression algorithm (Philip Gage, 1994): repeatedly find the most common pair of adjacent bytes and replace it with a new, unused symbol. Sennrich et al. adapted the idea to build a vocabulary instead of compressing a file. Training goes like this:

Split every word in the training corpus into characters. The starting vocabulary is just those characters.
Count every pair of adjacent symbols, weighted by how often each word appears.
Merge the most frequent pair into a new symbol, add it to the vocabulary, and record the rule.
Repeat until the vocabulary reaches the target size.

The number of merges is the only real setting. As the paper puts it, the final vocabulary size equals the starting symbols plus the number of merge operations.

BPE training: count adjacent pairs, merge the most frequent, repeat until the vocabulary is full

The diagram uses the toy corpus from the Hugging Face LLM course: "hug" (10 times), "pug" (5), "pun" (12), "bun" (4) and "hugs" (5).

Start: the vocabulary is b g h n p s u.
Merge 1: the pair u + g appears 20 times (10 in hug, 5 in pug, 5 in hugs), more than any other, so ug becomes a token.
Merge 2: u + n now leads with 16 (12 in pun, 4 in bun), so un joins.
Merge 3: h + ug appears 15 times, and the first three-letter token, hug, is born.

Real tokenizers run this loop tens of thousands of times over gigabytes of text. GPT-2's vocabulary has 50,257 entries. GPT-4's cl100k_base has about 100,000 and GPT-4o's o200k_base about 200,000 (100,277 and 200,019 including special tokens, according to tiktoken). Meta says Llama 3 moved to a 128K-token vocabulary that yields up to 15% fewer tokens than Llama 2 on the same text.

Encoding new text replays the merges. Split the input into characters (or bytes), then apply the learned rules in priority order. A word seen often during training collapses into one token. A word never seen still gets encoded, just in smaller pieces. With GPT-4o's vocabulary, "unbelievably" becomes un | bel | ievably.

The order of merges also explains some quirks. Merges reflect the tokenizer's training data, which isn't necessarily the model's training data. Keep that in mind; it comes back when we get to glitch tokens.

Byte-Level BPE: No More "Unknown Word"

Classic BPE starts from characters, and Unicode defines roughly 150,000 of them. Any character missing from the training corpus becomes [UNK], the "unknown" token. The Hugging Face course notes this is one reason older NLP models handled emoji badly.

GPT-2 introduced a simple fix: run BPE over bytes instead of characters. There are only 256 possible byte values, so the base vocabulary has 256 entries and every possible input, whether emoji, Tamil script or binary garbage, can be written in them. Merges then build up common multi-byte sequences. As the course puts it, GPT-2 doesn't actually have an unknown token, because an unknown character can't occur with byte-level BPE.

You can see the byte fallback at work. In GPT-2's vocabulary, the emoji 🙂 (four UTF-8 bytes) becomes 2 tokens. GPT-4o's larger vocabulary learned the whole sequence, so it's 1 token. Either way, nothing is ever "unknown." The price is that a rarely seen script can get chopped into fragments that aren't even whole characters, which leads to the language tax below.

Why the Model Can't See the R's

Back to the strawberry. Here's what GPT-4o's tokenizer does with the classic question:

The model sees “ strawberry” as one ID with no letters inside; spelling it out or calling code exposes the letters

"How many r's are in strawberry?" becomes 8 tokens: How | ·many | ·r | 's | ·are | ·in | ·strawberry | ?. The word you care about is token #101830. To the model, that's one opaque symbol, mapped to one embedding vector. There's no "r" in it to count.

Whatever the model knows about how strawberry is spelled, it had to memorize from training text where the word happened to be spelled out, as in dictionaries, spelling-bee pages or kids' worksheets. That knowledge is thin and indirect, so the model guesses, and a plausible guess for a word ending in a double letter is "two."

The split also depends on context in ways people don't expect:

strawberry (leading space, the usual case mid-sentence) is 1 token.
strawberry (no leading space, for example at the start of a message) is 3 tokens: st | raw | berry.
Strawberry (capitalized) is a different 3: Str | aw | berry.

So the model may have learned slightly different things about the same word.

The fixes all expose the letters. Writing the word as s-t-r-a-w-b-e-r-r-y turns it into 10 tokens, one per letter, and now counting is something the model can actually do. Better still, when exact character-level answers matter (counting, reversing a string, checking a regex), have the model call a code tool. One line of Python, "strawberry".count("r"), returns 3 every time.

The Multilingual Token Tax

The tokenizer's merges come from its training data, and that data is mostly English and code. English words get dedicated tokens. Other languages, especially ones with their own scripts, get fewer merges and are left closer to raw bytes.

The token tax: the same sentence costs more tokens in some languages, and numbers split into left-aligned chunks

I ran one short sentence, "The weather is nice today," and simple translations of it through three of OpenAI's tokenizers. In English, it's 6 tokens in all three. In Tamil, it's 85 tokens with GPT-2's tokenizer, 42 with GPT-4's and 11 with GPT-4o's. Hindi drops from 27 to 21 to 5, and Russian from 27 to 13 to 6. That's just one sentence, not a benchmark, but the direction is clear. Larger, more multilingual vocabularies have closed much of the gap, though not all of it.

Aleksandar Petrov and colleagues studied this systematically in Language Model Tokenizers Introduce Unfairness Between Languages (2023). They found the same text can differ in token length by up to 15× across languages, and that even byte-level and character-level models show gaps of more than 4×. The consequences are concrete:

Cost: APIs bill per token.
Context: a 128K-token window holds far fewer pages of Tamil than of English.
Latency: output is generated one token at a time.

If you serve a non-English market, measure token counts on your actual traffic. Don't assume the "about 4 characters per token" rule of thumb.

Numbers: Why Long Addition Goes Wrong

The same pre-split regex decides how numbers are chunked, and labs have made very different choices. Aaditya Singh and DJ Strouse surveyed them in Tokenization counts: the impact of tokenization on arithmetic in frontier LLMs (2024):

Pure BPE (GPT-3, Claude 2.1, OLMo): whichever digit strings happened to be frequent got tokens. 710 might be one token while 711 isn't, so numbers of the same length split inconsistently.
Single digits (PaLM, Llama 1 and 2, Mistral): every digit is its own token.
Chunks of up to 3 digits, left to right (GPT-3.5, GPT-4): the \p{N}{1,3} rule in the tiktoken pattern. GPT-4o's o200k_base uses the same rule.

Left-to-right chunking means 1234567 becomes 123 | 456 | 7. The chunks line up with the start of the number, while place value, and the way carries move when you add, runs from the end. Insert commas and the tokenizer splits on them, giving 1 | , | 234 | , | 567. Now every chunk sits on a thousands boundary.

Singh and Strouse tested exactly this on GPT-3.5 with 7- to 9-digit addition. With plain digits, 8-shot accuracy was 75.6%. With commas, it was 97.8%. When the answer had more digits than the inputs (because of a final carry), plain-digit accuracy fell to 8.25%, and the errors weren't random: the model almost always got the first three digits right and the fourth digit wrong, right at the first token boundary. Asking the model to first rewrite the problem with commas recovered most of the lost accuracy. GPT-4 was less affected, but the effect didn't disappear.

The lesson isn't "add commas everywhere." It's that number formatting changes the problem the model has to solve. For anything that must be exact, like invoices or financial math, route the arithmetic to a calculator or code tool.

Glitch Tokens: When the Vocabulary Outlives the Data

In early 2023, Jessica Rumbelow and Matthew Watkins went looking at GPT-2 and GPT-3's token embeddings and found a cluster of strange entries: SolidGoldMagikarp, TheNitromeFan, cloneembedreportprint, petertodd. In their LessWrong write-up, they showed that asking a model to simply repeat these strings produced evasion, insults or substitutions. ChatGPT, for example, repeated " SolidGoldMagikarp" as "distribute." Some of these tokens even made GPT-3 give different answers at temperature 0, which is supposed to be deterministic.

Their partial explanation goes back to the point about merges. The tokenizer was trained on scraped web text that included Reddit usernames, e-commerce site internals and game logs. The model was trained on more heavily curated text. So strings like SolidGoldMagikarp (I confirmed it's a single token, #43453, in GPT-2's vocabulary) earned a vocabulary slot but almost never showed up during model training. Their embedding vectors stayed close to wherever they started, and the model doesn't know what to do when it meets one.

It's not just a GPT-2 curiosity. Fishing for Magikarp (Land and Bartolo, 2024) built automatic detectors that combine tokenizer analysis, weight-based signals and prompting, and found under-trained tokens across a wide range of models. In newer OpenAI vocabularies, " SolidGoldMagikarp" is 5 ordinary tokens (·Solid | Gold | Mag | ik | arp), but each new tokenizer can bring its own oddities. If you train or fine-tune models, scanning for under-trained tokens is cheap insurance.

The Other Tokenizer Families

BPE dominates today's LLMs, but you'll run into two other algorithms, often through one library:

WordPiece (developed by Google for BERT) also learns merges bottom-up, but according to the Hugging Face course it scores a pair as its frequency divided by the product of its parts' frequencies, which favors pairs whose parts are rare on their own. It keeps only the final vocabulary and encodes each word by taking the longest piece it knows, then the next. Pieces inside a word carry a ## prefix, as in token ##ization. (Google never open-sourced its training code, so the course describes a best-guess reconstruction.)
Unigram LM, from Taku Kudo's Subword Regularization (2018), works top-down. It starts with a large candidate vocabulary, scores how much each piece contributes to a probabilistic model of the corpus, and prunes the least useful pieces. Since the same word can be segmented several ways, Kudo trained translation models on sampled segmentations as a kind of noise, which made them more robust, especially in low-resource and out-of-domain settings.
SentencePiece (Kudo and Richardson, 2018) is a library rather than an algorithm. It implements both BPE and Unigram and trains directly on raw sentences, with no language-specific pre-splitting. Whitespace becomes the visible symbol ▁, so I saw a girl becomes ▁I ▁saw ▁a ▁girl (example from the SentencePiece README). That makes it a natural fit for languages that don't put spaces between words, like Japanese and Chinese.

Picking a Vocabulary Size

Why not a million tokens? Every entry costs a row in the input embedding matrix and another in the output layer that scores the next token (unless the two are tied). With 4,096-dimensional vectors, a 200,000-entry vocabulary is about 820 million parameters per matrix. More entries also means fewer training examples per rare token, which is the glitch-token problem again.

Smaller vocabularies mean longer sequences: attention costs more, the context window fills faster and generation takes more steps. Recent models have moved toward larger vocabularies (50K, then 100K, 128K and 200K) as the embedding table becomes a small share of total parameters. Meta credits Llama 3's 128K tokenizer with "substantially improved model performance."

A Cheat Sheet for Engineers

Common tokenizer gotchas, their causes and practical fixes

A few of these deserve a sentence more.

Count tokens with the real tokenizer. Use tiktoken.encoding_for_model("gpt-4o") for OpenAI models, or your provider's token-counting endpoint, before you enforce context limits or estimate costs. Character-based estimates are wrong for code, non-English text and JSON.

Watch the boundary at the end of your prompt. Because GPT-style tokenizers attach the space to the following word, " Paris" is a single token. A prompt that ends with a trailing space ("The capital of France is ") leaves a bare space token, and the model then has to continue with "Paris" without its usual leading space, an unusual sequence it saw less often in training. Trim trailing whitespace from completion-style prompts. The "token healing" technique cited by Singh and Strouse goes further: it backs up a few tokens and lets the model regenerate across the boundary.

Special tokens are control characters. Strings like <|endoftext|> map to reserved IDs (100257 in cl100k_base, 199999 in o200k_base) that mark document and message boundaries. By default tiktoken refuses to encode them from ordinary text: encode("hi <|endoftext|>") raises an error. Keep that protection on, so pasted text can't forge a document or message boundary.

Key Takeaways

LLMs read integers, not letters. A tokenizer turns text into IDs before the model runs, and anything inside a token, like its spelling, is invisible unless the model memorized it.
BPE builds the vocabulary by merging the most frequent adjacent pairs, and byte-level BPE guarantees that any input can be encoded with no "unknown" token.
Tokenization explains classic failure modes: letter counting, multi-digit arithmetic (75.6% vs 97.8% for GPT-3.5 depending only on commas), higher costs for some languages (up to 15× longer encodings) and glitch tokens.
The fix is usually to change the representation or hand off the work: spell words out, format numbers deliberately, and call code for anything that must be exact.
Measure with the actual tokenizer. "4 characters per token" is a rough English-only average, not a budget.

DEV Community