Ask an LLM how many R's are in "strawberry" and it will often say two. This isn't stupidity — it's a direct window into how these models see text, and understanding it explains a surprising number of their quirks.
The model never sees letters. It sees tokens. Once you understand tokenization, a whole class of "why did it do that" mysteries dissolves.
Text isn't characters to a model
Before a single word reaches the neural network, it's chopped into tokens — chunks that are often whole words, sometimes word-pieces, sometimes just a few characters. "strawberry" might become a single token, or split into "straw" and "berry." The model only ever operates on these chunks and their numeric IDs. It has no direct access to the individual letters inside a token.
That's why counting letters is hard for it: if "strawberry" is one opaque token, the model was never given the R's to count. It's not reasoning about spelling; it's reasoning about a symbol that stands for a word. Ask a human to count the letters in a word they can only hear, never see, and they'd struggle too.
Why tokenization shapes everything
This isn't a trivia detail — it has real consequences:
- Cost and limits are counted in tokens, not words. Your context window, your API bill, your rate limits — all measured in tokens. Roughly, English runs about 0.75 words per token, but code, punctuation, and other languages differ wildly.
- Some languages pay a tax. Tokenizers are usually optimized for English. The same sentence in a less-represented language can take two or three times as many tokens — meaning higher cost and less that fits in the context window, for identical meaning.
- Rare words fragment. A common word is one clean token; an obscure name or a novel string shatters into many pieces, which the model handles less gracefully.
- Whitespace and formatting are tokens too. How your text is spaced and punctuated quietly changes the token stream the model sees.
Understanding this changes how you write prompts and estimate cost — the kind of low-level detail I pay attention to across the systems I build.
Tokens become vectors
Here's how it connects to the rest of the stack. Each token ID is mapped to an embedding — a vector capturing its meaning — and it's those vectors the model actually reasons over. Tokenization is the doorway; embeddings are the room behind it. The choice of how to split text determines what the model gets to work with in the first place.
The practical takeaway
When a model does something baffling with spelling, character counts, or a rare string, your first thought should be: what did that look like as tokens? Half the time, the "reasoning failure" is really a tokenization artifact — the model was answering a question it literally couldn't see the pieces of.
Knowing where the model's vision starts and stops is what separates guessing at its behavior from understanding it. More at www.divyakush.com.
Related reading
- Embeddings and semantic search, from the ground up — what happens to tokens once they become vectors.
- The context window: an LLM's working memory and its limits — the model working memory and how to manage it.
- Multimodal AI: how a text model learns to see — how one model handles images and text.
Divyakush Punjabi · Full-Stack & AI Engineer
Portfolio · GitHub · LinkedIn
Top comments (0)