If you're reading about AI, LLMs, or RAG, you'll hear terms like tokens, vectors, and embeddings all the time. They may sound complicated, but the basic idea is quite simple.
Computers don't understand language the way humans do. Before an AI model can process text, it first converts it into numbers.
The journey looks like this:
Text
↓
Tokenizer
↓
Tokens
↓
Embeddings (Vectors)
↓
AI Model
↓
Output
Or simply:
Language → Small pieces → Numbers → Understanding
1. Tokens
A token is a small piece of text that the model processes.
For example:
"I love cricket"
may become:
["I", "love", "cricket"]
But tokens are not always complete words.
playing
may become:
["play", "ing"]
or
unbelievable
may become:
["un", "believ", "able"]
So, words and tokens are not always the same thing.
2. Tokenizer
A tokenizer is the tool that splits text into tokens.
Example:
Input:
unbelievable
Output:
["un", "believ", "able"]
Different AI models use different tokenizers, so the same sentence may be split differently.
You'll also notice that some tokens include spaces, such as " like" instead of "like". This helps the model better understand how text naturally appears.
3. Tokenization
Tokenization is simply the process of splitting text into tokens.
Text
↓
Tokenizer
↓
Tokens
The tokenizer is the tool.
Tokenization is the process.
4. How Does the Tokenizer Split Words?
The tokenizer doesn't know every word beforehand. Instead, it learns common patterns from large amounts of text.
You don't need to remember every method, but you'll often hear these names:
BPE (Byte Pair Encoding) – merges common pieces of words.
WordPiece – similar to BPE but chooses splits based on probability.
For example:
playing
may become:
play + ing
Modern LLMs mostly use subword tokenization, sometimes combined with byte-level tokenization, because it handles new words, different languages, emojis, and symbols efficiently.
5. Tokens Become Numbers
After tokenization, the computer still doesn't understand the text.
Computers only work with numbers, so every token is converted into a vector.
For example:
cat
might become:
[0.2, -0.5, 1.3, ...]
The numbers themselves are not important. They are simply how the computer represents text.
6. Vector vs Embedding
These two terms are closely related, but they are not the same.
A vector is simply a list of numbers.
[0.2, -0.5, 1.3, 0.8]
Those numbers could represent anything.
An embedding is a vector that has been learned by an AI model to capture meaning.
For example:
cat → [0.2, -0.5, 1.3, ...]
dog → [0.25, -0.45, 1.2, ...]
car → [-1.8, 0.7, 2.4, ...]
The important part isn't the numbers themselves. It's the distance between them.
Since cat and dog have similar meanings, their embeddings are close together. Car is a different concept, so it is farther away.
A simple way to remember it is:
Vector = just numbers
Embedding = numbers with meaning
In other words:
Every embedding is a vector, but not every vector is an embedding.
7. Token Embeddings
Every token gets its own embedding.
For example, king and queen have different embeddings, but they are close together because they are related.
The model learns these relationships during training.
8. Text Embeddings
Sometimes we don't want an embedding for each token. Instead, we create one embedding for an entire sentence, paragraph, or document.
For example:
I love cricket.
becomes a single vector representing the meaning of the whole sentence.
Text embeddings are commonly used for:
Semantic search
RAG
Recommendations
Finding similar documents
For example, if someone searches:
best sport in India
the system can still find a document saying:
Cricket is the most popular sport in India.
Even though the wording is different, the meaning is similar.
9. Chunking
Large documents are usually split into smaller pieces called chunks before creating embeddings.
Document
↓
Chunks
↓
Embedding for each chunk
Later, when someone asks a question, the system compares the question's embedding with all chunk embeddings and retrieves the most relevant chunks.
This is one of the core ideas behind RAG.
10. Other Important Terms
Vocabulary
The vocabulary is the collection of tokens the tokenizer knows.
Common vocabulary sizes are 30K, 50K, or 100K tokens.
Special Tokens
Models also use special control tokens such as:
Start of text
End of text
Separator
Padding
These help the model understand the structure of the input.
Context Window
Every model has a maximum number of tokens it can process at once. This is called the context window.
For example:
8K tokens
32K tokens
100K+ tokens
If the input is longer than the context window, some text has to be removed or handled separately.
Putting Everything Together
The complete flow looks like this:
Input Text
↓
Tokenizer
↓
Tokens
↓
Embeddings
↓
AI Model
↓
Output Tokens
↓
Readable Text
A ChatGPT Example
Suppose you ask ChatGPT:
"What is the capital of India?"
Behind the scenes, something like this happens:
Your Question
↓
Tokenizer splits the text into tokens
↓
Each token is converted into an embedding (a vector that captures its meaning)
↓
Similar words and concepts are placed close together in the embedding space
↓
The model processes these embeddings to understand the context and relationships
↓
It predicts the most likely next tokens
↓
"Delhi is the capital of India."
You only see the final answer, but internally the model never works directly with words. It works with tokens, embeddings, and mathematical operations on those embeddings to generate the response.
What's Next?
This blog covered the basics of how AI understands text using tokens and embeddings. These concepts are the foundation for almost everything you'll learn next.
In the upcoming blogs, we'll keep building on these basics and cover topics like:
RAG (Retrieval-Augmented Generation) explained in simple language
How context works in AI and why it matters
Different types of RAG, including Agentic RAG, Graph RAG, Corrective RAG, and more
Vector Databases and how they store embeddings
Semantic Search and how AI finds information by meaning instead of keywords
AI Agents and Agentic AI—what they are and when to use them
Each topic will be explained step by step, without assuming any prior AI knowledge.
Stay tuned!
This post was initially published on my blog. Check out the original source using the link below:
Top comments (0)