Tokenization in LLM

neha — Mon, 31 Aug 2026 13:19:17 +0000

Hii!!

I have started my journey to learn basics about AI and LLM and have decided to write something about what I learn along the way.

Need for tokenization

LLMs don't process text directly. Neural networks operate on numerical representations, so the text needs to be converted into numbers before it can be processed by the model. Aside from understanding, it also needs to relate it to other things. To process data, models need numbers.
A token is a piece of text chosen by the tokenizer.

Here,
Piece of text - Tokens
Number assigned to each token- Token ID

A token can be:

A whole word
Part of a word
Punctuation
Whitespace + a word
Sometimes even individual bytes

For example, the text I love kittens might be split into:

["I", " love", " kittens"]

These tokens are then mapped to token IDs such as:

[40, 821, 5632]

At high level, the token ID is used to look up the corresponding embedding vector. These vectors are then provided to the Transformer for further processing.

About token IDs

Tokenization happens at the very beginning before any processing happens.
Different models can use different tokenizers, so the token IDs generated for the same text can be different for different models.

For example,
Consider input as: hi
one model might assign token ID 12 and another might assign it 25.

Token IDs are integer numbers assigned to each token.

Vocabulary

In simple terms, it refers to mapping of chunk of words to a particular number.A vocabulary is essentially a collection of tokens and their corresponding IDs. The tokenizer learns or constructs this vocabulary during tokenizer training, and the resulting vocabulary is then used to tokenize text.

The chunks generated from data gets assigned to the number based on this vocabulary.

Why not use individual characters as tokens?

Every character has a corresponding Unicode code point. We could theoretically use characters directly as tokens, but this has some drawbacks:

Unicode contains a very large number of possible characters, making a character-level vocabulary large.
If we restrict the vocabulary to only a limited set of characters, characters outside that set can result in out-of-vocabulary (OOV) problems.
Character-level representations can also result in much longer sequences than subword representations.

To address these problems, different tokenization algorithms have been developed, such as:

Byte-pair encoding (BPE)
Byte-level BPE (BBPE)
Wordpiece

High level flow

We can understand at a very high level, flow is

Text
 ↓
Tokenizer
 ↓
Tokens
 ↓
Token IDs
 ↓
Embedding vectors
 ↓
Transformer

Example:

"I love kitten"
       ↓
Tokenizer
       ↓
["I", " love", " kitten"]
       ↓
[40, 821, 5632]       ← token IDs
       ↓
[vector, vector, vector]
       ↓
Transformer

Why does the number of tokens matter?

As many of you have observed, we see number of tokens we have used when we interact with models. For example, Copilot shows how many tokens we have used. Tokens are associated with both the user input and output generated by LLM.

The number of tokens affects how much text a model can process within its context window and how much computation is required. More tokens generally mean more computation.

That's it for now!

This is what I have understood about tokenization and token IDs so far. There is still a lot more to explore, especially how algorithms like BPE actually create these tokens.

I'll be diving into that next and sharing what I learn along the way. 🚀

If you spot anything I got wrong or have suggestions for what I should learn next, feel free to let me know!

DEV Community: neha