Hello ♦
Central idea of the article: How does a machine turn human language into something it can actually process? Tokenization doesn't teach a machine what language means. It gives language a structure that a machine can process mathematically.
Reading time: 7-10 minutes
What's the problem? A computer doesn't see words the way we do
What do you see and feel when you read the sentence:
I like the JDM vibe.
Maybe you imagine the aesthetics of 1990s Japan, cool cars, The Fast and the Furious, and all that. You can build a mental image of the sentence and somehow feel what it means.
A language model, however, can't form such an image in the same way we do. It doesn't perceive words the way we do. For a language model, raw text is not yet a convenient mathematical representation.
To the model, the raw text is just a wall of symbols.
From Text to Something a Machine Can Process
In my article about vectors, I talked about how data can be represented mathematically.
For example, we can describe an apartment with a vector:
x = [45, 1, 6, 1]
where:
45 — area;
1 — number of rooms;
6 — floor;
1 — distance from the city center;
We decided ourselves which characteristics were important and turned them into numbers. With text, things get more complicated.
For machine learning algorithms to work with text mathematically, we need to convert it into a numerical representation. But how can we turn abstract text into numbers without losing its meaning?
The answer lies in tokenization — the first and one of the key steps in how modern LLMs process text.
What is a token?
A token is a unit of text that a model treats as an element of its input sequence.
For example:
['I', ' like', ' the', ' JDM', ' vibe', '.']
Here, each element of the list is a token.
But there's an important thing to keep in mind: a token is not necessarily a word.
For example, the same sentence could, in principle, be split like this:
['I', 'lik', 'e', ' the', ' J', 'DM', ' vi', 'be', '.']
The exact way text is split depends on the tokenizer and its vocabulary.
Why not simply use words?
At first glance, it seems logical to use:
one word = one token
In this approach, we create a vocabulary that our tokenizer can look up, where each word is assigned its own ID.
But language is enormous. People constantly create new words, use names, technical terms, slang, abbreviations, typos, and different forms of the same word. This means we'd need to store an enormous number of words in the vocabulary, which is impractical.
So a natural idea emerges:
Maybe we don't need to store every possible word. Maybe we can split words into smaller, reusable pieces.
Words are not always the smallest useful units
As we saw earlier:
['I', ' lik', 'e', ' the', ' J', 'DM', ' vi', 'be', '.']
We can use smaller parts of words — recurring fragments of text.
These fragments don't necessarily correspond to linguistic morphemes. Their purpose is not to be "correct parts of a word", but to serve as useful units for representing text.
This allows a model to process words and sequences that it has never encountered as whole tokens by combining smaller pieces that are already in its vocabulary.
But there are several other ways to split words into tokens.
Three ways to tokenize text
The most practical approach is subword-based tokenization, which is why modern language models typically use different variants of subword tokenization.
Tokenizer
Now we have a separate component — the tokenizer.
Very roughly, its job can be illustrated like this:
It's important to understand:
A tokenizer is not a language model.
A tokenizer doesn't try to reason about a sentence or answer the user's question. It performs a much more specific task: converting text into tokens.
In other words, it turns text into a sequence of discrete units.
But tokens are still not numbers
We have:
['I', ' like', ' the', ' JDM', ' vibe', '.']
But a neural network needs numerical data. That's why the tokenizer uses a special dictionary called a vocabulary. Each token is assigned a specific identifier:
I → 52
_like → 58
_the → 7005
_JDM → 786
_vibe → 5
. → 4
Now we've finally turned text into numbers. But this is where a very important trap appears.
A token ID is just an ID
Let's look at two tokens:
JDM → 9231
vibe → 2711
Can we say that "JDM" and "vibe" are similar because:
|9231 − 2711| = 6520?
No.
And can we say that their numerical IDs tell us anything about their semantic relationship?
Also no.
A token ID is just an identifier.
It answers the question:
Which token is this?
But it doesn't answer the question:
What does this token mean?
Now we can see the complete pipeline:
"I like the JDM vibe" → ["I", " like", " the", " JDM", " vibe"] → [40, 1093, 279, 41723, 215]
And that's what the process of converting text into numbers looks like!
That's it.
I hope you found something useful here.
My vibe today:
Top comments (0)