You've never sent a single word to an AI.
Every time you hit send on ChatGPT, Claude, or Gemini, your words get shredded before the model ever sees them. The process is called tokenization. Once you understand it, you'll never think about AI the same way again.
What is a token?
A token is the actual unit your AI reads — not a word, not a sentence. Sometimes it's a whole word. Sometimes just a fragment.
- "ChatGPT" is 2 tokens.
- "Unbelievable" is 3 ("un", "believ", "able").
- A single emoji can cost 3 tokens.
Rough rule: 1 token ≈ 4 characters ≈ 0.75 words.
This is what your message actually looks like to an AI. Not words. Colored chunks of subwords.
Why does this matter?
The model never sees language. It sees numbers. Every token gets converted into a vector — a list of hundreds of values that places it in mathematical space.
"King" and "Queen" sit close together in that space. "King" and "pizza" are far apart. The model understands meaning the same way a map understands geography — through distances, not definitions.
This is how AI understands the word "Queen". Not by reading it. By knowing where it sits relative to everything else.
The Strawberry Problem
Ask any LLM how many R's are in "strawberry." Most get it wrong.
Not because it's stupid — because it never sees the full word. It sees "straw" and "berry" as two separate tokens and tries to count letters inside chunks it can't see into.
This is the gap between what AI feels like and what it actually is.
"Straw" + "berry". Two tokens. This is why it can't count the R's.
What changed in 2026?
Tokens aren't just text anymore.
Every major model (GPT-5.5, Claude Opus 4.7, Gemini 3.1 Pro…) now handles a million tokens in a single conversation. Images cost tokens. Audio costs tokens. Video costs tokens. Your entire prompt — text, files, screenshots — becomes one long river of numbers the model processes simultaneously.
The one thing to remember
Your prompt is not a command. It's a statistical context — a sequence of numbers that nudges the model toward the most probable next output.
Better context = better predictions. That's the whole game.
How does the AI actually build its token vocabulary?
This is the part nobody explains and everyone should know.
The algorithm behind almost every major AI model (GPT, Claude, Llama, Mistral) is called Byte Pair Encoding, or BPE. It was originally invented in 1994 as a data compression algorithm. OpenAI repurposed it for AI in 2019.
Nobody designed it for language. It just happened to work extraordinarily well.
Here's how BPE actually works (in plain English):
- Start with every single character in the alphabet. "h", "e", "l", "l", "o" — each is its own token.
- Now scan through billions of words and find the two characters that appear next to each other most frequently. Merge them into a single token.
- Repeat. Over and over. Thousands of times.
The result is a vocabulary of subword units — chunks that are bigger than single characters but smaller than full words — that perfectly balances handling rare words and common ones simultaneously.
By the end, your vocabulary might look like:
- "ing" is one token (because it appears everywhere)
- "un" is one token (prefix of thousands of words)
- "2026" is one token (appears enough to earn its own slot)
- "xylophone" gets split (rare enough that it never earned a merge)
The final vocabulary of most modern LLMs sits between 32,000 and 128,000 tokens. GPT-4o uses around 100,000.
Every prompt you send gets mapped into this fixed vocabulary and nothing outside it can exist.
This is how AI builds its vocabulary. Not by learning words. By learning which letter combinations appear together most often and merging them.
The hidden bias nobody talks about
Here's something that will make you genuinely uncomfortable.
The training data for most LLMs is heavily skewed toward English. Llama 2, for example, was trained on 89.7% English content.
This means the tokenizer learned English patterns extremely well and everything else poorly.
The result is a hidden inequality baked into every AI interaction:
- Processing the same sentence in German or Italian costs about 50% more tokens than in English.
- In languages like Burmese or Tibetan, the same text can cost up to 15 times more tokens than the equivalent in English.
Why does this matter?
Because tokens = cost. Tokens = speed. Tokens = how much of your context window gets used up.
If you're using AI in Hindi, Tamil, Arabic, or any non-English language, you are paying more, getting slower responses, and fitting less into each conversation.
Not because the AI is worse. Because the tokenizer was never built with your language in mind.
A developer described this perfectly: "My agent works great in French or English. But if a user asks in Darija, it crashes in quality and becomes unusable."
That's not a bug. That's tokenization.
This is one of the most underreported fairness issues in all of AI right now.
Tokens are no longer just text
This is where 2026 changes everything.
Tokens aren't just words anymore. Every major frontier model now processes images, audio, and video as tokens too.
- An image tile costs tokens.
- A second of audio costs tokens.
- A video frame costs tokens.
What this means in practice:
When you send Claude a screenshot, it doesn't "see" it the way you do. It converts the image into a grid of tiles, encodes each tile into numerical vectors, and feeds those vectors into the same token stream as your text.
Your image. Your voice note. Your PDF. Your video. All of it — one long river of numbers. Processed the same way.
The model has no concept of "this is a picture" versus "this is a sentence." It's all just tokens in a sequence.
This is what people mean when they say AI is becoming truly multimodal. It's not that the model learned to see. It's that vision got tokenized.
Text. Image. Audio. Video. It all becomes the same thing before the model sees it.
Tokens.
What this means for how you use AI (starting today)
Understanding tokenization isn't just academic. It directly changes how you should prompt.
Front-load your important information
The model processes tokens left to right. The beginning of your prompt carries the most weight. Put your most critical instructions first — not buried in paragraph three.Be concise — you're paying per token
Every word you type is tokens. Every word the model generates is tokens. Verbose prompts with filler phrases waste your context window and cost more on paid APIs. Trim everything that doesn't add meaning.If you're coding, name things clearly
Code files often exceed the token limit of a single embedding and they have to be chunked. Variable names and function names become individual tokens. Clear, common naming patterns tokenize more efficiently than cryptic abbreviations."getUserData"tokenizes better than"gtUsrDt."Switch to English for complex reasoning tasks
This is uncomfortable but true. For tasks requiring deep multi-step reasoning, especially on smaller models, English prompts will outperform non-English prompts because the tokenizer was trained primarily on English. On frontier models this gap is shrinking. On smaller local models, it's still significant.Don't trust AI with character-level tasks
Spelling backwards. Counting specific letters. Rhyming on a specific syllable. Anagrams. These all require character-level processing that tokenization makes structurally difficult. Use AI for meaning-level tasks. Use code for character-level tasks.
The future of tokenization
Tokenization is not a solved problem. It's actively being reinvented right now.
Researchers are exploring byte-level transformer models that process raw bytes instead of subword tokens entirely. This would mean no vocabulary, no BPE merges, no tokenization bias against non-English languages. Every byte is treated equally.
The tradeoff is sequence length. Bytes are much smaller than subwords, so the same text becomes a much longer sequence, which is expensive computationally.
Other researchers are working on multilingual tokenizers with "language plasticity" — the ability to adapt post-training to new languages without retraining the entire model from scratch.
The goal is simple: an AI that processes every language on earth with equal efficiency, equal cost, and equal quality.
We're not there yet. But tokenization is exactly where that battle is being fought.
Takeaway
Every time you type a message to an AI, a silent, high-speed assembly line starts moving.
Your words get shredded.
Assigned numbers.
Converted into vectors.
Fed into a model that has never read a single word in its life.
And somehow on the other side it answers like it understood everything.
That gap between what's actually happening and what it feels like is the most fascinating thing in technology right now.
Source: Adapted from the excellent thread by Shiny (@Oishaaniii) on X. Original post contains additional visuals and examples.
Top comments (0)