DEV Community: Danil'

Tokens: How Machines Break Language into Pieces

Danil' — Fri, 28 Aug 2026 14:58:09 +0000

Hello ♦

Central idea of the article: How does a machine turn human language into something it can actually process? Tokenization doesn't teach a machine what language means. It gives language a structure that a machine can process mathematically.

Reading time: 7-10 minutes

What's the problem? A computer doesn't see words the way we do

What do you see and feel when you read the sentence:

I like the JDM vibe.

Maybe you imagine the aesthetics of 1990s Japan, cool cars, The Fast and the Furious, and all that. You can build a mental image of the sentence and somehow feel what it means.

A language model, however, can't form such an image in the same way we do. It doesn't perceive words the way we do. For a language model, raw text is not yet a convenient mathematical representation.

To the model, the raw text is just a wall of symbols.

From Text to Something a Machine Can Process

In my article about vectors, I talked about how data can be represented mathematically.

For example, we can describe an apartment with a vector:

x = [45, 1, 6, 1]

where:

45 — area;
1 — number of rooms;
6 — floor;
1 — distance from the city center;

We decided ourselves which characteristics were important and turned them into numbers. With text, things get more complicated.

For machine learning algorithms to work with text mathematically, we need to convert it into a numerical representation. But how can we turn abstract text into numbers without losing its meaning?

The answer lies in tokenization — the first and one of the key steps in how modern LLMs process text.

What is a token?

A token is a unit of text that a model treats as an element of its input sequence.

For example:

['I', ' like', ' the', ' JDM', ' vibe', '.']

Here, each element of the list is a token.

But there's an important thing to keep in mind: a token is not necessarily a word.

For example, the same sentence could, in principle, be split like this:

['I', 'lik', 'e', ' the', ' J', 'DM', ' vi', 'be', '.']

The exact way text is split depends on the tokenizer and its vocabulary.

Why not simply use words?

At first glance, it seems logical to use:

one word = one token

In this approach, we create a vocabulary that our tokenizer can look up, where each word is assigned its own ID.

But language is enormous. People constantly create new words, use names, technical terms, slang, abbreviations, typos, and different forms of the same word. This means we'd need to store an enormous number of words in the vocabulary, which is impractical.

So a natural idea emerges:

Maybe we don't need to store every possible word. Maybe we can split words into smaller, reusable pieces.

Words are not always the smallest useful units

As we saw earlier:

['I', ' lik', 'e', ' the', ' J', 'DM', ' vi', 'be', '.']

We can use smaller parts of words — recurring fragments of text.

These fragments don't necessarily correspond to linguistic morphemes. Their purpose is not to be "correct parts of a word", but to serve as useful units for representing text.

This allows a model to process words and sequences that it has never encountered as whole tokens by combining smaller pieces that are already in its vocabulary.

But there are several other ways to split words into tokens.

Three ways to tokenize text

The most practical approach is subword-based tokenization, which is why modern language models typically use different variants of subword tokenization.

Tokenizer

Now we have a separate component — the tokenizer.

Very roughly, its job can be illustrated like this:

It's important to understand:

A tokenizer is not a language model.

A tokenizer doesn't try to reason about a sentence or answer the user's question. It performs a much more specific task: converting text into tokens.

In other words, it turns text into a sequence of discrete units.

But tokens are still not numbers

We have:

['I', ' like', ' the', ' JDM', ' vibe', '.']

But a neural network needs numerical data. That's why the tokenizer uses a special dictionary called a vocabulary. Each token is assigned a specific identifier:

I → 52

_like → 58

_the → 7005

_JDM → 786

_vibe → 5

. → 4

Now we've finally turned text into numbers. But this is where a very important trap appears.

A token ID is just an ID

Let's look at two tokens:

JDM → 9231

vibe → 2711

Can we say that "JDM" and "vibe" are similar because:

|9231 − 2711| = 6520?

No.

And can we say that their numerical IDs tell us anything about their semantic relationship?

Also no.

A token ID is just an identifier.

It answers the question:

Which token is this?

But it doesn't answer the question:

What does this token mean?

Now we can see the complete pipeline:

"I like the JDM vibe" → ["I", " like", " the", " JDM", " vibe"] → [40, 1093, 279, 41723, 215]

And that's what the process of converting text into numbers looks like!

That's it.

I hope you found something useful here.

My vibe today:

Vectors in Machine Learning: More Than Just Lists of Numbers

Danil' — Fri, 21 Aug 2026 09:19:07 +0000

Hello ♦

Central idea of the article: Machine learning works with data represented as numbers. Linear algebra gives us a language for reasoning about those numbers as geometric objects.

Reading time 10 minutes

The simplest data can be represented by a single number:

The speed of a car
The number of disks
The mass of a compressor

But why do we need to represent data as numbers? Obviously, to answer our questions:

How long will it take me to get from point A to point B?
What is the maximum number of disks that can fit in a box?
Will I be able to carry this compressor by myself?

You ask yourself questions like these every day, and numbers help you answer them and make the right decisions, so you don't have to suffer:

Finally, I can get 8 hours of sleep because I'll get there quickly.
So I need to buy 20 disks — I can't fit any more in the box.
I don't want to strain my back, so I'll ask Kamil to help me.

But what if we want to describe an object as a whole, rather than just its individual characteristics? For example, to estimate its value, efficiency, or something else.

Let's consider some characteristics of a car:

For example: mass = 1,500 kg, power = 200 hp, speed = 220 km/h.

We could store these as three separate numbers, but then we lose the idea that these three values describe the same object.

This leads to a natural idea: combine several numbers into a single structure.

A simple real-life analogy is a pencil case. People invented pencil cases to keep everything they need for studying in one place — from pencils to scissors. If we know we'll need scissors, pens, and pencils, we put them all in a pencil case rather than scattering them across separate pockets of a backpack.

In a similar way, something happened with data: several related numbers came to be treated as a single mathematical object — ♦ a vector ♦.

♦ Vector ♦

Originally, vectors did not exist as a separate mathematical concept. People worked with scalars — plain numbers. But as physics began to develop, scientists encountered quantities that had not only a magnitude, but also a direction.

17th century (René Descartes): He connected geometry with algebra by introducing the Cartesian coordinate system. A point in space could now be represented by a set of numbers: (x, y, z) in three-dimensional space and (x, y) in two-dimensional space. The coordinate system made it possible to represent the geometry of an object as a set of numbers.

In other words, we can turn an object or its properties into a numerical representation.

Apartments:

Let's say we have two apartments that we rent out to tourists visiting the city. We also know that our friend Tomi, who owns 25 apartments, has no trouble renting out any of them.
We'd like to understand why only one of our apartments is profitable — what's wrong with the other one?

Apartment A

Area: 35 m²
2 rooms
5th floor
5 km from the city center

Apartment B

Area: 50 m²
1 room
2nd floor
3 km from the city center

Approximate characteristics of Tomi's apartments:

Apartment B

Area: 45 m²
1 room
6th floor
1 km from the city center

And there are approximately 25 apartments with similar characteristics.

Let's try to represent all of this as a mathematical structure that describes the object as a whole — a vector.

x_{1} = [35, 2, 5, 5]
x_{2} = [50, 1, 2, 3]
x_{3} = [45, 1, 6, 1]
...
x_{25} = [49, 1, 5, 1.2]

We think about it, discuss it with our wife, and realize that people want to live closer to the city center so they can be closer to the city's main attractions. Tourists also generally don't need a lot of space. In other words, they don't need a two- or three-room apartment. For a family of three or four, a one-bedroom apartment is usually a perfectly acceptable option.

Assuming that the key factors are the apartment's size and its distance from the city center, let's plot a graph with:

the x-axis representing the apartment's area;
the y-axis representing the distance from the city center.

Looking at a large amount of data in the form of lists, it's difficult to tell how far our apartments are from the ideal profile.

Based on the two features we selected, Apartment B is closer to the typical profile of Tomi's apartments. We should probably sell Apartment A altogether, since it isn't a good fit for short-term rentals to tourists. It would be better suited for long-term rentals.

In the 19th century, the ideas of vector calculus developed further, particularly through the work of William Hamilton, Hermann Grassmann, James Clerk Maxwell, and other mathematicians and physicists.

A vector was no longer just a pair of numbers or a point in space [3, 5]. It became an object that could be operated on:

addition;
subtraction;
multiplication by a scalar;
dot product;
calculating its magnitude;
determining its direction.

In other words, a vector turned out to be a convenient way to represent the structure and relationships between numerical quantities.

This means that we can do more than simply represent points in space, as we did with the apartments.

For example, we have created a visual representation showing where our apartments are located.

But it would be useful to know the exact numbers — how different are our apartments?

This is one of the advantages of vectors: we can calculate the differences easily and get the complete picture:

visual representation + precise numbers

Let's calculate the average values for Tomi's apartments:

44.48 — average area

1.12 — average distance from the city center

Now let's compare our apartments with Tomi's average:

For Apartment A:

35 − 44.48 = −9.48 m²

5 − 1.12 = 3.88 km

For Apartment B:

50 − 44.48 = 5.52 m²

3 − 1.12 = 1.88 km

The minus sign in the first case simply shows where Apartment A's area lies relative to the average area.

Now let's visualize this average so that we can focus on a single object that represents all of Tomi's apartments.

Ideally, we want to buy apartments whose characteristics are as close as possible to the green point.

Analytically, we can calculate which apartment is closer to the green point:

Apartment A: A = [35, 5]

Apartment B: B = [50, 3]

Average apartment: C = [44.48, 1.12]

We can calculate the distance:
$d(A,C)=\sqrt{(35−44.48)^{2}+(5−1.12)^{2}}$
$d(B,C)=\sqrt{(50−44.48)^{2}+(3−1.12)^{2}}$

And if we can measure the distance between apartments, why not measure the distance between texts? Images? Users? Products? This could help us understand how similar they are.

I think you've already noticed that the mathematical framework allows us to work not only with 2D or 3D vectors, but with n-dimensional vectors as well. This is one of the biggest advantages of vectors: we can represent and analyze vectors with as many numbers as we need — or as many as our technical constraints allow.

This means we can represent almost anything. From pixels to chemical structures.

But there's an important point we need to stop and think about.

We don't simply turn an object into numbers. We create a representation of that object, and the quality of this representation matters enormously.

For example, we could represent an apartment like this:

[50, 2, 7, 1.5]

Or like this:

[2, 50, 7, 1.5]

But if the order of the features hasn't been defined beforehand, this vector is meaningless.

What I mean is that the numbers inside a vector don't mean anything by themselves. Their meaning comes from what each position represents. A vector only makes sense when we define what each dimension represents.

Or, in more mathematical terms:

A number by itself has no meaning until we fix a basis — what each axis of the space represents.

The hardest part is often not doing mathematics on data. It is deciding what the data should look like mathematically.

ML

ML uses vectors not just as convenient containers for numbers.

In human terms:

Let's represent each object as a point in space and try to find patterns in how these points are arranged.

For example, as shown in the figure, let's try to determine which object represents text, which represents a number, and which represents sound.

In other words, we encode the properties of an object as an ordered set of numbers — its coordinates. This turns the object into a point in a multidimensional space. Machine learning algorithms can then look for spatial relationships: distances between points, clusters, angles, and directions.

The important idea behind vectors isn't that they store numbers. It's that they allow us to turn objects into points in a mathematical space where we can measure relationships between them.

That's it.

I hope you found something useful here.

I’d describe my vibe today with this picture: