Modern AI feels like magic until you peek under the hood and realize it is more like a highly disciplined assembly line of linguistic fragments. This piece rebuilds the core ideas from Naveen Kumar VR’s original explanation into a more structured engineering perspective, while keeping the intuition intact and sharpening the technical clarity.
⸻
Tokens: The Fundamental Unit of Language Models
Large language models do not process text as humans do. They do not see full words or letters in isolation. Instead, they operate on tokens, which are compressed chunks of text.
A token can be:
- A full word (“error”)
- A word fragment (“stra”, “wberry”)
- A punctuation mark (“.”)
- Or even whitespace patterns
Think of text as a construction kit where tokens are modular components. Instead of building language from raw characters or entire words, models assemble meaning from pre-shaped linguistic “blocks”.
⸻
Byte Pair Encoding (BPE): How Tokens Are Created
At the heart of modern tokenization sits Byte Pair Encoding, a statistical compression method adapted for language.
Here is the intuition:
- Start with every character as a separate unit
- Scan massive text datasets
- Identify the most frequent adjacent character pairs
- Merge them into new tokens
- Repeat millions of times
Over time:
- Common patterns become single tokens (“the”, “ing”, “tion”)
- Rare words remain split into multiple pieces
- Code, typos, and unusual strings get fragmented further
This is why language models are efficient: they learn a reusable vocabulary of patterns instead of memorizing every possible word.
⸻
The Cost Reality: Tokens Are Money
One of the most important engineering truths:
You are not billed per word. You are billed per token.
A rough heuristic:
- 1 token ≈ 0.75 words (English average)
But inefficiencies in formatting can inflate token usage quickly.
Even small differences matter:
- “Fix the error.”
- “Fix the error .”
That extra space and punctuation irregularity can cause additional token splits, increasing cost at scale.
⸻
Token Economics: Input vs Output
In production systems, token usage splits into two categories:
- Input tokens: what you send to the model
- Output tokens: what the model generates
Output tokens are usually more expensive because they require sequential generation and higher compute workload.
Example pricing structure (illustrative):
Model Tier
Input Cost (per 1M tokens)
Output Cost (per 1M tokens)
Lightweight (e.g. mini models)
lower
moderate
Premium models like GPT-4o
higher
significantly higher
This is why architectural decisions in prompts and payload formatting directly translate into infrastructure spend.
Formatting Matters: JSON Isn’t Free
Even formatting becomes a cost variable.
Pretty JSON:
{
"status": "unhealthy",
"replica_count": 0
}
Minified JSON:
{"status":"unhealthy","replica_count":0}
At scale, the difference is significant:
- Pretty format introduces whitespace tokens and line breaks
- Minified format reduces token count dramatically
In real systems, minification can reduce token usage by 30–40% for structured payloads.
⸻
The “Strawberry Problem”: Why Models Miscount Letters
A famous demonstration shows models failing to count letters in “strawberry”.
This is not a math failure. It is a visibility limitation.
When a human sees:
strawberry → s t r a w b e r r y
A model sees something like:
[“straw”, “berry”]
Inside the tokenizer:
- The word is split into semantic chunks
- Individual letters are no longer directly accessible
So when asked:
“How many R’s are in strawberry?”
The model effectively cannot inspect the raw spelling. It only reasons over token fragments:
- “straw” → no visible letter-level structure
- “berry” → statistical association of letters, not explicit decomposition
Result: the reasoning becomes approximate rather than literal.
This phenomenon is often called token blindness.
⸻
Vocabulary Size: Why GPT-4o Became More Efficient
Earlier tokenizers (like cl100k_base) used ~100K token vocabularies.
With newer systems such as GPT-4o, vocabulary size expanded to ~200K tokens.
Impact:
- More compact encoding of multilingual text
- Better handling of non-English languages
- Reduced token fragmentation in code and structured data
For developers, this means:
- Lower token consumption per request
- Faster inference in many cases
- Reduced cost in global applications
⸻
Special Tokens: Invisible Control Signals
Not all tokens represent human-readable text. Some act as structural commands inside the model pipeline:
- <|endoftext|> → marks end of input stream
- <|im_start|> / <|im_end|> → separates system, user, and assistant roles
These are critical for:
- Conversation structure
- Instruction hierarchy
- Safety boundaries
However, if manipulated incorrectly, they can contribute to prompt injection-style vulnerabilities, where system boundaries are confused or bypassed.
⸻
Beyond Tokens: What Comes Next
Understanding tokens unlocks deeper concepts in LLM engineering:
Token IDs
Tokens are converted into integers before entering the model:
- “the” → 464 (example ID)
- “error” → 3921 (example ID)
The model only ever sees numbers.
⸻
Context Window Limits
Every model has a fixed memory window:
- Input tokens + output tokens must fit inside it
- Exceeding it causes truncation or loss of earlier context
This is why long conversations “forget” earlier details.
⸻
Rare or “Glitch” Tokens
Because tokenizers are trained on massive web corpora, they sometimes include:
- obscure strings
- leaked identifiers
- unusual code fragments
These can behave unpredictably in edge cases, especially in adversarial inputs.
⸻
Final Takeaway
Tokenization is not just a preprocessing step. It is the economic, structural, and cognitive backbone of modern language models.
Once you understand tokens:
- Prompt engineering becomes predictable
- Cost modeling becomes transparent
- Model limitations become explainable rather than mysterious
⸻
Credit
This rewritten technical synthesis is based on the original educational work and concepts by Naveen Kumar VR, from his MLOps and AI series on language model fundamentals.
Top comments (2)
So what do you think this article is saying, and why?
Any thoughts on this?