After closing Phase 1 with the Tensor structure that will support the rest of the project, it's time to solve a problem that looks simpler on the surface, but carries a design decision worth pausing to think about before jumping into coding: how to turn text into a number.
The Problem
A neural network doesn't process text. It processes numbers, and more specifically, it processes linear algebra operations on vectors and matrices, which means that before anything gets near an embedding layer, an attention layer, or whatever else, the text we type needs to become a sequence of integers. That's the sole responsibility of the tokenizer: to be the bridge between the language we understand and the numerical representation the model can manipulate.
Three Ways to Build That Bridge
There are essentially three levels of granularity for tokenization, and each one solves the problem differently, with clear trade-offs.
Using the phrase "the cat sleeps" as an example, you can see how each approach tackles the problem differently.
The first is char-level, where each character (including the space) becomes a token:
Text: "the cat sleeps"
Tokens: ['t', 'h', 'e', ' ', 'c', 'a', 't', ' ', 's', 'l', 'e', 'e', 'p', 's']
IDs: [1, 0, 5, 2, 8, 1, 0, 4, 1, 7, 6, 3, 9, 4]
The vocabulary here is all the distinct characters appearing in the corpus (letters, space, punctuation), which for English stays in the range of 60 to 100 symbols, and the implementation is straightforward, but in return the sequences get long, since each word becomes multiple tokens.
The second is word-level, where each word is a single token (typically separating punctuation as well):
Text: "the cat sleeps"
Tokens: ['the', 'cat', 'sleeps']
IDs: [12, 340, 891]
The vocabulary becomes all the distinct words in the corpus, which quickly turns into tens of thousands of entries, and any new word the model hasn't seen during training (a proper name, for example) simply has no ID for it—the classic OOV (out of vocabulary) problem.
The third, which real production models use (GPT-2 and up, for example), is subword via BPE (Byte Pair Encoding). The logic is to start char-level and merge the most frequent pairs of characters until forming a vocabulary of fixed size (10,000 tokens, for example), which makes common words into a single token and rare or compound words break into smaller pieces:
Text: "the cat sleeping peacefully"
Tokens: ['the', 'Ġcat', 'Ġsleep', 'ing', 'Ġpeace', 'ful', 'ly']
IDs: [12, 340, 55, 891, 203, 77, 88]
(the Ġ is the GPT-2 convention to mark word start with space before). Notice that "sleeping" became two tokens ("sleep" + "ing") and "peacefully" also ("peace" + "ful" + "ly"), the algorithm learned that these fragments are frequent enough to deserve their own token, without needing an entry for every complete word in the language. It's the middle ground that solves both the long sequence problem of char-level and the OOV problem of word-level, just at the cost of a much more complex implementation.
Why Char-Level, Here and Now
For this phase of the project, I opted for char-level, not because it's the "right" approach (it's not what production uses), but because the goal here is to understand the end-to-end mechanism without the complexity of the BPE merge algorithm getting in the way of learning. Char-level closes the encode-decode cycle quickly, with an implementation that fits in a few lines, and that's what matters in a phase that exists to consolidate concepts, not to compete with production tokenizers.
Separating Responsibilities
The design split into two classes: Vocabulary, which is the mapping between character and ID (and vice versa), and Tokenizer, which uses that vocabulary to convert text to IDs and back. This separation exists because it leaves the door open to swap tokenization strategies later (char-level for BPE, for example) without needing to touch whoever consumes the Tokenizer.
Vocabulary Consistency
The vocabulary cannot be rebuilt with every execution. If the Vocabulary is regenerated every time the program runs, the IDs assigned to each character might come out differently from one session to the next, and in that case, the model trained with one vocabulary becomes invalid if used with another, because each learned weight depends directly on the ID it represents.
The vocabulary is built once, during dataset preparation, and from that point on is persisted and only loaded (never rebuilt) in all subsequent training or inference. The encode and decode, those are used all the time, always against the same frozen vocabulary.
What Stays for Next Phase
With the tokenizer closed, the text sequence already becomes a sequence of integers, but those integers still don't mean anything to the model, they're just indices. That's what the next phase solves: transforming each ID into a dense vector, capable of carrying some notion of meaning in vector space. That's the gateway to embeddings.
Top comments (0)