"How Text Becomes Numbers: Tokenization Explained"

This article covers how raw text becomes something that an NLP model can actually compute with.

Why Computers Cannot Understand Words?

A neural network, at its core, is a system of matrix multiplications and other numerical operations. It has no native concept of a "word" — it only operates on numbers. Feed it the string "cat" directly and a neural network will not be able to multiply a matrix by the letters c-a-t.

This means every NLP system, regardless of whether it is Rule-based, Statistical (n-gram), RNN / LSTM or Transformer — needs a conversion step: raw text in, numbers out. Tokenization is that conversion step. It is the process of splitting text into discrete units (tokens) and assigning each one a unique numerical ID.

Vocabulary

Before you can assign IDs to tokens, you need a vocabulary — a fixed, finite list of every possible token the model knows about, each mapped to a unique integer ID.

A simplified example vocabulary might look like:

{"the": 0, "cat": 1, "sat": 2, "on": 3, "mat": 4}

Given this vocabulary, the sentence "the cat sat" would tokenize to the ID sequence [0, 1, 2].

The vocabulary is fixed at training time and typically contains tens of thousands of entries. For e.g. GPT-2's vocabulary has roughly 50,000 tokens.

What Happens When The Model Encounters A Word That Isn't In Its Vocabulary?

A word like "the" is almost certainly in any English vocabulary, but what about a rare technical term, a typo, a name, or a word in a different language? This is exactly the problem the rest of this article solves.

What are Tokens?

A token is a discrete unit of text that raw text gets split into. Each token is then assigned a numerical ID.

It can be:

• A whole word: "cat" → 1 token
• A sub-word piece: "tokenization" → might split into "token" + "ization" → 2 tokens
• A single character: "x" → 1 token
• Punctuation: ",", "." → each their own token

Limitations of Early Tokenizers

Early tokenizers split on whitespace and punctuation, effectively treating each word as one token. The downside is that it cannot handle words the vocabulary hasn't seen before. If "tokenization" isn't in the vocabulary, a whole-word tokenizer maps it to a generic <UNK> (unknown) token, losing all information about the corpus text.

Sub-word Tokenization and its Advantages

Modern tokenizers solve the problem of tokens outside the vocabulary, and the loss of information that results from it, with sub-word tokenization.

Instead of a fixed list of whole words, the vocabulary contains common word pieces. A rare or unseen word gets split into pieces the vocabulary does recognize, rather than being discarded entirely. "Tokenization" becomes "token" + "ization" — both of which are common enough to be in the vocabulary on their own, even if the combined word is rare. This means a sub-word tokenizer can represent virtually any input, including words it never saw during training, by falling back to smaller and smaller pieces — even down to individual characters, if required.

BPE or Byte-Pair Encoding

BPE is the most widely used sub-word tokenization algorithm, and it's how the vocabulary itself gets built in the first place.

The core idea is to start with individual characters as your base vocabulary, then repeatedly merge the most frequently occurring adjacent pair into a new token, until you reach your target vocabulary size.

For e.g. your training corpus is dominated by the words "low," "lower," "lowest," and "newest." BPE takes into account words occurring with the highest frequency and starts by treating every word as a sequence of individual characters:

l o w
l o w e r
l o w e s t
n e w e s t

Step 1: find the most frequent adjacent character pair across the corpus. In this case, it's e s (appears in "lowest" and "newest"). Merge it into a new token: es.

Step 2: repeat. Now es t is frequent (both words end in "est" after the first merge). Merge into est.

Step 3: repeat again. l o is frequent (appears in every "low" variant). Merge into lo.

This process continues for a fixed number of merges (a hyperparameter chosen before training). The result is a vocabulary containing individual characters, common sub-word pieces (est, lo, low), and — for very frequent whole words — the whole word itself as a single token.

The key property is that common words end up as single tokens, while rare or unseen words get broken into smaller known pieces instead of being discarded.

SentencePiece

BPE has an assumption that it operates on text that is already split into words by whitespace. This works fine for English, but breaks down for languages that don't use whitespace to separate words at all, for e.g. Japanese and Chinese.

SentencePiece is a tokenization framework that solves this by treating the input as a raw stream of characters (including whitespace itself, often represented as a special character like ▁), rather than assuming words are pre-separated. It then applies a sub-word algorithm (BPE, or Unigram Language Model tokenization) directly on that raw stream. This makes it language-agnostic. This is why SentencePiece is widely used in multilingual models.

Context Length

Once text is tokenized into a sequence of IDs, that sequence has a length — a token count. Every transformer model has a maximum number of tokens it can process in a single input, called its context length (or context window).

This matters because if a document tokenizes to 5,000 tokens and the model's context length is 4,096, the input has to be truncated or split. In practice, this is handled either by simple truncation (cutting off whatever doesn't fit) or by splitting a document into smaller chunks processed separately. The latter is relevant to Retrieval-Augmented Generation and how large documents get broken into meaningful pieces before being fed to a model.

Sample Code

The example below uses Hugging Face's transformers library, a widely-used open-source Python library providing pre-trained models and tokenizers.

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("gpt2")

text = "Tokenization is fundamental to NLP."
tokens = tokenizer.tokenize(text)
token_ids = tokenizer.encode(text)

print(tokens)
print(token_ids)

Output:

['Token', 'ization', 'Ġis', 'Ġfundamental', 'Ġto', 'Ġ N', 'LP', '.']
[30642, 1634, 318, 7531, 284, 399, 12016, 13]

• Ġ is how GPT-2's tokenizer represents a space that comes before a token. This specific approach is called byte-level BPE.
• Ġis has the Ġ prefix, because there is a whitespace before "is" in the original text ("Tokenization is...")
• "Tokenization" splits into two pieces (Token + ization) — exactly the sub-word behavior described earlier.
• "NLP" splits into N and LP — an example of a rarer term getting broken into smaller known pieces rather than being mapped to a generic unknown token.
• The final output is a list of integers (token_ids) that gets fed into the model.

Where This Leaves Us?

Tokenization converts text into a fixed vocabulary of numerical IDs, using sub-word algorithms like BPE or SentencePiece so that any input — including words never seen during training — can be represented without loss. This conversion is the literal first step of the pipeline.

The next question is what a neural network actually does with the IDs assigned to tokens.