Let's start with the simplest definition:
A token is a small piece of text that an AI model processes as input or generates as output.
An AI model doesn't read text exactly the way humans do.
You see:
"I love learning AI."
The AI might break this into tokens somewhat like:
"I"
" love"
" learning"
" AI"
"."
The exact tokenization depends on the model and tokenizer.
So, a token can be:
- A complete word
- Part of a word
- A punctuation mark
- A space combined with a word
- Sometimes special symbols
Tokens Are Not the Same Across Models
Different AI models may tokenize text differently.
For example:
"GenerativeAI"
might be:
Model A:
["Generative", "AI"]
Model B:
["Gener", "ative", "AI"]
The exact number of tokens depends on:
- Model
- Tokenizer
- Language
- Text
Therefore, you should not assume: 1 word = 1 token
Why Does AI Use Tokens?
Computers work with numbers.
An LLM cannot directly process "I love learning AI".
Instead, the text goes through a process:
Human Text
↓
Tokenizer
↓
Tokens
↓
Token IDs (Numbers)
↓
Neural Network
↓
Predicted Token IDs
↓
Text
This whole process of breaking text into tokens is called Tokenization.
For example:
Input:
"I love Pizza!"
Tokenizer:
I
love
Pizza
!
Then each token is mapped to a number called a token ID.
I → 12345
love → 4567
Pizza → 234
! → 45
Is Token ID the same as Vectors?
Now, here comes an important question: if token IDs represent words as numbers, then what are vectors?
(I have already discussed about Vectors and Embeddings in my previous blog. I would recommend you to check that out to fully understand the context here.)
Coming to the point. The answer of the above question is No. Token IDs and vectors are not the same thing, although both are represented using numbers. This is a very common confusion when learning how LLMs work.
The process is:
Text → Token IDs → Vectors → LLM processing
HUMAN TEXT
│
▼
┌───────────┐
│ Tokenizer │
└───────────┘
│
▼
TOKEN IDs
[40, 1832, 2456]
│
▼
┌─────────────────┐
│ Embedding Layer │
└─────────────────┘
│
▼
VECTOR REPRESENTATION
[0.12, -0.45, 0.87, ...]
│
▼
LLM
│
▼
NEXT TOKEN PREDICTION
Token IDs are identifiers. Vectors are numerical representations.
Let's take an example to clearly understand the concept of Token IDs, Vectors and Embeddings:
"I love pizza"
↓
TOKENIZER
↓
Token IDs (Identifies a token)
[45, 782, 1234]
↓
EMBEDDING MODEL
↓
Embedding
[0.21, -0.45, 0.87, ...]
(This list of nos. is a vector created to represent meaning)
Explaination:
🏷️ Token ID
"Which token is this?"
pizza → 1234
It's just an identifier. 1234 does not mean "pizza" mathematically.
🔢 Vector
"What is the numerical format?"
[0.21, -0.45, 0.87, 0.12]
Just a list of numbers.
🧠 Embedding
"What does this data mean?"
An AI model converts:
"I love pizza"
into an embedding, which is represented as a vector:
[0.21, -0.45, 0.87, 0.12]
The numbers collectively capture semantic information about the text.
Remember this:
- Token ID = Identifier 🏷️ (tells the AI what token it is)
- Vector = List of numbers 🔢 () (numerical form used to represent data.)
- Embedding = Meaning represented as a vector 🧠 (represent what the data means)
Tokens and LLMs
Now let's understand how tokens actually work when you talk to ChatGPT or another LLM.
Suppose you type:
"What is RAG?"
The process looks like:
Your Question
↓
Tokenization
↓
Token IDs
↓
LLM processes tokens
↓
Model predicts next token
↓
Another token
↓
Another token
↓
Final Response
The model generates the answer one token at a time.
For example, it might generate:
RAG
→ stands
→ for
→ Retrieval
→ Augmented
→ Generation
→ ...
Each generated token is selected based on probabilities learned by the model.
Does the AI Generate the Whole Answer at Once?
No.
You must have noticed when ChatGPT generates a response. Word by word, right?! It keeps predicting the next token until it decides the response is complete.
This is why LLMs are sometimes called autoregressive models in the context of next-token prediction.
Tokens and AI Pricing 💰
Before I tell you about the token cost, I'll first explain to you what we mean by token usage.
Tokens are generally divided into two categories:
1. Input Tokens: The tokens you send to the model.
For example:
User:
Explain RAG in simple terms.
Your input might include:
System instructions + Your question + Conversation history + RAG context/documents + Tool results
All of these consume input tokens.
2. Output Tokens: The tokens the model generates.
RAG stands for Retrieval-Augmented Generation...
So, for a basic request:
Input Tokens
+
Output Tokens
=
Total Token Usage
For example:
Your request
↓
Input: 5,000 tokens
+
Output: 2,000 tokens
↓
Total: 7,000 tokens
If the model's available context window is 128K, you're well within the limit.
Wait, What Is a Context Window?
This is another extremely important concept.
An LLM has a maximum amount of text it can process at one time. This is called its context window.
┌───────────────────────────────┐
│ Context Window │
│ │
│ System Instructions │
│ + │
│ Previous Conversation │
│ + │
│ User Prompt │
│ + │
│ Retrieved Documents (RAG) │
│ + │
│ Generated Response │
│ │
└───────────────────────────────┘
Everything needs to fit within the model's context limits.
What Happens When You Exceed the Context Window?
Suppose:
Context Window = 100,000 tokens
But you try to send:
120,000 tokens
The model cannot process all of them within that request.
Depending on the application, you may see:
- An API error
- Input truncation
- Older conversation content removed
- Context being summarized
The exact behavior depends on the model and platform.
Coming to Token pricing. It is one of the most important topics to understand if you're building GenAI applications using APIs.
The basic idea is:
You are generally charged based on how many tokens your application sends to the model and how many tokens the model generates.
The cost may depend on:
- Input token price
- Output token price
- Model being used
This means developers need to manage tokens carefully.
Simple Example:
Let's imagine a fictional model with pricing:
Input:
$1 per 1 million tokens
Output:
$5 per 1 million tokens
Suppose your application uses:
- Input = 100,000 tokens
- Output = 20,000 tokens
Input cost
100,000 / 1,000,000 × $1
= $0.10
Output cost
20,000 / 1,000,000 × $5
= $0.10
Total
$0.10 + $0.10
= $0.20
So the request costs $0.20.
This is just a hypothetical example to explain the calculation. Real model prices vary significantly.
Token Cost vs Context Window
Don't confuse these two.
Imagine:
Model Context Window = 128K tokens
This means the model can process a certain amount of context within its supported limits.
It doesn't mean: "You have to use 128K tokens."
You could send:
2,000 tokens and still use the same model.
You pay based on your actual usage according to the provider's pricing, not simply because the model supports a 128K context window.
Think of it like a backpack:
🎒 Backpack capacity = Context window
📦 What you actually put inside = Token usage
💰 What you pay for = The actual usage according to the pricing model
One simple formula to remember:
Total LLM Cost ≈ (Input Tokens × Input Rate) + (Output Tokens × Output Rate)
Where Are Tokens Used?
Tokens are used across almost every part of modern Generative AI.
1. Chatbots
User Question
→ Tokens
→ LLM
→ Output Tokens
2. RAG
Question
→ Tokens
→ Retrieve Context
→ LLM
→ Answer
3. AI Agents
Goal
→ Tokens
→ Reason
→ Tool Call
→ Tool Result
→ Tokens
→ Final Answer
4. Summarization
Long Document
→ Tokens
→ LLM
→ Summary Tokens
The Simplest Way to Remember:
Think of an LLM like a chef.
- Tokens are the ingredients 🧱- The model processes language as small pieces.
- The context window is the kitchen counter 🏠- You can only have a limited amount of ingredients on the counter at once.
- The LLM is the chef 👨🍳- It uses the available ingredients to prepare the answer.
- Token limits are the counter size 📏- If you bring too many ingredients, everything won't fit.
- Token costs are the grocery bill 💰- The more ingredients you use, the more you may pay.
Top comments (0)