Neural networks don’t understand words, sentences, or punctuation. They only understand numbers.

Core Definition: Tokenization is the process of breaking raw text into smaller discrete units—called tokens—which can be words, subwords, characters, or symbols that a computer model can process.

Numerical Mapping: Once split, each unique token is assigned a numerical identifier (Token ID) using a predefined vocabulary, allowing neural networks to perform mathematical operations on text.

Modern Standard (Subword Tokenization): Most large language models use subword algorithms (such as Byte-Pair Encoding or WordPiece) to balance vocabulary size, efficiently handle compound words, and avoid "out-of-vocabulary" errors for rare or unseen terms.