DEV Community: Prince Singh

Tokenization in AI: What Is It, Why Is It Used, and How Does It Work?

Prince Singh — Sun, 02 Aug 2026 01:00:49 +0000

Have you ever wondered how an AI model understands the sentence you type?

For example, when you ask:

«"How do I learn Python?"»

An AI model doesn't directly process the entire sentence like a human does.

Before the model can work with your text, the text usually goes through an important step called tokenization.

In this post, we'll learn:

What tokenization is
Why AI models use tokenization
How text is converted into tokens
What tokens and token IDs are
How tokenization works with code
How tokens are used by Large Language Models (LLMs)

Let's get started! 🚀

🧩 What Is Tokenization?

Tokenization is the process of breaking text into smaller pieces called tokens.

A token can be:

A complete word
Part of a word
A punctuation mark
A number
A special symbol

For example, the sentence:

«"I love programming!"»

Could be split into tokens like:

"I"
"love"
"programming"
"!"

The exact tokens depend on the tokenizer used by the AI model.

Tokenization is important because AI models work with numbers, not raw human text.

So the general process looks like this:

Human Text
↓
Tokenization
↓
Tokens
↓
Token IDs
↓
AI Model
↓
Output Tokens
↓
Detokenization
↓
Human Text

🤔 Why Do AI Models Need Tokenization?

Computers don't naturally understand words the way humans do.

A computer works with numerical representations.

For example, an AI model might convert:

Hello world

into something like:

[15496, 995]

These numbers are called token IDs.

The model can then process these numbers using mathematical operations.

So tokenization acts as a bridge between:

Human Language
↕
Tokenization
↕
Numerical Representation
↕
AI Model

🔢 What Is a Token ID?

A tokenizer usually has a vocabulary containing many tokens.

Imagine a very small vocabulary:

Token ID

hello 10
world 11
I 12
love 13
AI 14
! 15

If we write:

«"I love AI!"»

The tokenizer could convert it into:

I → 12
love → 13
AI → 14
! → 15

The AI model receives:

[12, 13, 14, 15]

The real vocabulary of modern AI models is much larger and more complicated than this simple example.

✂️ Tokens Are Not Always Complete Words

One of the most important things to understand is that one token is not always one word.

A long or uncommon word may be split into multiple tokens.

For example:

"unbelievable"

Might be divided conceptually into:

"un"
"believ"
"able"

The exact split depends on the tokenizer.

This approach is useful because the model doesn't need to store every possible word in its vocabulary.

Instead, it can combine smaller pieces.

For example:

"play"
"playing"
"played"
"player"

These words share similar pieces.

A tokenizer can represent words using reusable subword units.

🌍 Why Subword Tokenization Is Useful

Imagine you have a vocabulary containing only complete words.

What happens when someone types a word the model has never seen?

For example:

«"Supercalifragilisticexpialidocious"»

The model might not have this entire word in its vocabulary.

With subword tokenization, it can break the word into smaller pieces.

Conceptually:

Super
cal
ifrag
ilistic
exp
ial
idocious

The model can then process these pieces instead of treating the entire word as unknown.

This helps AI models handle:

New words
Rare words
Names
Technical terms
Programming code
Different languages

💻 Tokenization and Programming Code

Tokenization isn't only useful for normal text.

It is also important for AI coding tools.

Consider this Python code:

def hello(name):
return "Hello " + name

A tokenizer may break it into tokens representing things such as:

hello
(
name
)
:
return
"Hello "
+
name

Again, the exact tokenization depends on the model and tokenizer.

This allows AI models to process programming languages as sequences of tokens.

That's one reason modern AI coding assistants can work with:

Python
JavaScript
Java
C++
Rust
SQL
HTML
CSS

and many other programming languages.

🧠 How Tokenization Works in an LLM

Let's imagine you send this prompt:

«"Explain recursion in Python."»

A simplified pipeline looks like this:

Your Prompt
│
▼
"Explain recursion in Python."
│
▼
Tokenizer
│
▼
Tokens
│
▼
Token IDs
│
▼
AI Model
│
▼
Predicted Token
│
▼
More Predicted Tokens
│
▼
Generated Response

The model processes the input tokens and generates output tokens.

For example, the output might conceptually be:

"Recursion"
"is"
"a"
"technique"
"where"
"a"
"function"
"calls"
"itself"

The model generates these tokens sequentially.

The output tokens are then converted back into readable text.

This final process is called detokenization.

🔄 Tokenization vs. Detokenization

These are basically opposite processes.

Tokenization

Human-readable text → Tokens

Hello world
↓
["Hello", "world"]

Detokenization

Tokens → Human-readable text

["Hello", "world"]
↓
Hello world

So the complete flow is:

     TOKENIZATION

Text ───────────────────► Tokens
│
▼
AI Model
│
▼
Tokens
│
▼
DETOKENIZATION ◄──────┘
│
▼
Text

📏 What Are Context Windows?

You may have heard about AI models having a context window.

A context window is the amount of information a model can process within a particular interaction.

The size is often measured in tokens.

For example, imagine a model has a context window of:

100,000 tokens

That means the model can handle a large amount of tokenized information within that context.

The exact number of words represented by a certain number of tokens varies depending on the language and content.

For English, a rough rule of thumb is that one token is often around a fraction of a word, but this is only an approximation.

Code, punctuation, numbers, and other languages can tokenize differently.

💰 Why Do AI Companies Talk About Token Usage?

Tokens are also important for understanding AI API usage.

Many AI services measure usage based on tokens.

For example:

Input Tokens
+
Output Tokens
=
Total Token Usage

Imagine you send:

"Write a Python program to calculate Fibonacci numbers."

The tokenizer converts your prompt into input tokens.

The AI then generates output tokens containing the answer.

So the total usage could be represented as:

Input: 15 tokens

Output: 200 tokens

Total: 215 tokens

The actual number will depend on the tokenizer and the exact text.

This is why developers working with AI APIs often care about:

Token limits
Context windows
Input tokens
Output tokens
Token costs

🧪 A Simple Tokenization Example

Let's take:

«"AI is amazing!"»

Conceptually:

Text:
AI is amazing!

↓

Tokens:
["AI", "is", "amazing", "!"]

↓

Token IDs:
[123, 45, 678, 9]

The AI model doesn't necessarily see the words directly.

It processes numerical representations of these tokens.

The numbers then pass through the neural network.

The model performs many mathematical calculations and eventually predicts the next token.

🎯 Why Tokenization Is So Important

Tokenization is a fundamental part of modern language AI.

It helps models:

Process Human Language

It converts text into a format that AI systems can process.

Handle New Words

Subword tokens allow models to work with unfamiliar or rare words.

Process Code

Programming languages can also be represented as sequences of tokens.

Manage Context

AI models measure their input and output capacity in tokens.

Control API Usage

Many AI APIs use token counts to measure usage and calculate costs.

Generate Text

Language models generate responses as sequences of tokens.

🤖 Tokenization Is Not the Same as Understanding

One important point is that tokenization itself doesn't mean the AI understands the text.

Tokenization is simply a way of representing text.

For example:

"I love programming."

might become:

Tokens → Token IDs → Neural Network

The model then uses its learned parameters and mathematical computations to process those tokens and generate an output.

So tokenization is one step in a much larger AI pipeline.

🏗️ The Big Picture

A simplified Large Language Model pipeline looks like this:

             USER
              │
              ▼
        Your Prompt
              │
              ▼
        TOKENIZATION
              │
              ▼
         TOKEN IDs
              │
              ▼
      ┌───────────────┐
      │   AI MODEL    │
      │               │
      │ Neural Network│
      │       +       │
      │   Parameters  │
      └───────────────┘
              │
              ▼
      Generated Tokens
              │
              ▼
      DETOKENIZATION
              │
              ▼
      Human-Readable Text

This is a simplified explanation. Modern AI systems have many additional components and complex processes.

🚀 Final Thoughts

Tokenization may sound like a small technical detail, but it is a very important part of how modern AI systems process language.

The basic idea is:

«Humans communicate using words and sentences. AI models process numerical representations of tokens.»

The process is roughly:

Text
↓
Tokenization
↓
Token IDs
↓
AI Model
↓
Generated Token IDs
↓
Detokenization
↓
Text

Once you understand tokenization, it becomes easier to understand concepts such as:

Large Language Models (LLMs)
Context windows
AI API pricing
Prompt length
Token limits
AI coding assistants
Text generation

So the next time you type a question into an AI chatbot, remember:

Your sentence is first transformed into tokens, and those tokens become the building blocks the AI uses to process your request and generate a response. 🤖

💬 What should I explain next?

If you're learning about AI, the next interesting topics are:

🧠 How Neural Networks Work
🔥 What Are Transformers?
🎯 How Attention Works
📝 How LLMs Predict the Next Token
🏋️ How AI Models Are Trained
🎨 How Image Generation Models Work

Let me know which one you'd like to learn next! 🚀

How Does AI Learn and Get Trained? A Simple Guide to Text, Image, and Video AI

Prince Singh — Sun, 02 Aug 2026 00:57:58 +0000

Artificial Intelligence (AI) can write stories, answer questions, create realistic images, generate videos, write code, and much more.

But how does AI actually learn?

Does it think like a human?

Does someone manually teach it every answer?

The short answer is no.

AI learns patterns from huge amounts of data using mathematical algorithms and powerful computers. In this article, we'll explore how AI training works and look at examples of text AI, image generation AI, and video generation AI.

🧠 What Is AI Training?

AI training is the process of teaching a computer model to recognize patterns and make predictions.

Imagine you want to teach a child what a cat looks like.

You might show the child thousands of pictures:

🐱 Cat
🐱 Cat
🐱 Cat
🐱 Cat

After seeing many examples, the child starts learning common characteristics:

Cats have four legs.
Cats have ears.
Cats have fur.
Cats have tails.

AI training works in a somewhat similar way, but instead of a human brain, AI uses mathematical models, algorithms, and neural networks.

The AI sees many examples and adjusts its internal parameters to become better at predicting the correct output.

📚 Step 1: Collecting Data

AI needs data to learn.

The type of data depends on the AI system.

For example:

Text AI

A text AI may be trained using large collections of text, such as:

Books
Articles
Websites
Documentation
Code
Publicly available text

Image AI

An image generation model may learn from large collections of images and their associated descriptions.

For example:

«"A dog running in a park"»

The model learns relationships between the words and visual patterns.

Video AI

Video generation models can learn from combinations of:

Images
Video frames
Text descriptions
Motion patterns
Audio, depending on the model

The goal is to learn how things look and how they change over time.

🧮 Step 2: Training the Model

Once data is collected, the AI model begins training.

A neural network contains many numerical values called parameters.

During training, the model makes predictions.

If the prediction is wrong, the training process calculates an error and adjusts the parameters.

This process happens again and again:

Input → Prediction → Compare with target → Calculate error → Update model → Repeat

After millions or billions of training examples, the model can become very good at recognizing patterns.

A simplified view looks like this:

    Training Data
         │
         ▼
  ┌──────────────┐
  │  AI Model    │
  └──────────────┘
         │
         ▼
    Prediction
         │
         ▼
   Calculate Error
         │
         ▼
  Update Parameters
         │
         └───────► Repeat

🔤 Example 1: How Text AI Learns

Let's imagine we have a simple sentence:

«"The cat is sitting on the ___."»

The AI might predict:

«"table"»

The correct answer could be:

«"mat"»

The model compares its prediction with the expected answer.

The training process adjusts the model so that it becomes better at predicting the next word.

After seeing millions or billions of examples, the model learns many patterns in language.

For example:

The sky is ___
→ blue

2 + 2 = ___
→ 4

The capital of France is ___
→ Paris

Modern large language models work on a much larger and more complex scale. They learn statistical patterns in language and can generate new text based on the context they receive.

When you ask a question, the model generates an answer by predicting a sequence of tokens based on the input and what it learned during training.

🖼️ Example 2: How Image Generation AI Works

Imagine you type:

«"A futuristic city at night with flying cars."»

An image generation AI tries to create an image that matches your description.

The model has learned relationships between language and visual concepts.

It may have learned patterns related to:

"futuristic"
"city"
"night"
"flying cars"
Lighting
Shapes
Perspective
Colors
Objects

A simplified process might look like:

Text Prompt
│
▼
"A futuristic city
at night..."
│
▼
AI understands
the concepts
│
▼
Generates visual
patterns
│
▼
Final Image

For many modern image-generation systems, the generation process involves starting from random noise and progressively transforming it into an image that matches the prompt.

For example:

Random Noise
↓
Less Noise
↓
Basic Shapes
↓
Objects Appear
↓
Details Added
↓
Final Image

The result could be an image of a futuristic city with skyscrapers, neon lights, and flying cars.

🎬 Example 3: How Video Generation AI Works

Video generation is even more challenging than image generation.

Why?

Because the AI needs to generate not just one image, but many frames that work together.

Imagine the prompt:

«"A robot walking through a futuristic city."»

The AI needs to understand:

What the robot looks like.
What the city looks like.
How the robot moves.
How the camera moves.
How lighting changes.
How objects remain consistent between frames.

A simplified process could look like:

Text Prompt
│
▼
Understand Scene
│
▼
Generate Video Frames
│
▼
Maintain Motion
and Consistency
│
▼
Create Final Video

For example:

Frame 1 → Robot starts walking
Frame 2 → Robot moves forward
Frame 3 → Robot continues walking
Frame 4 → Camera follows robot
Frame 5 → Robot reaches the building

The AI tries to make these frames look like parts of the same continuous scene.

This is one of the major challenges in generative video AI: maintaining consistency and realistic motion over time.

🧠 Does AI Actually "Understand" Like Humans?

This is an important question.

AI can produce impressive results, but it does not necessarily understand the world in the same way humans do.

For example, a language model can explain what a bicycle is.

It may know that:

«A bicycle has two wheels, pedals, handlebars, and is used for transportation.»

But that doesn't mean the AI has physically experienced riding a bicycle.

AI models are very good at learning patterns from data and using those patterns to generate predictions and outputs.

Human intelligence involves many other things, including:

Physical experience
Emotions
Consciousness
Personal memories
Common-sense reasoning
Social understanding

AI is different.

🏋️ Training vs. Using AI

There is an important difference between training and using an AI model.

Training

Training happens when the model learns from large amounts of data.

Large Dataset
↓
AI Model
↓
Training
↓
Learned Parameters

This can require:

Powerful GPUs or specialized AI hardware
Large datasets
A lot of electricity
Significant time
Complex algorithms

Using the Model

After training, people can use the model.

For example:

User:
"Write a Python function to sort a list."

↓

Trained AI Model

↓

Generated Answer

The model doesn't necessarily retrain itself from scratch every time you ask a question.

It uses what it learned during training to generate an output.

🔄 What Happens When You Ask AI a Question?

Suppose you ask:

«"Explain recursion in Python."»

A simplified process is:

Your Prompt
↓
Tokenization
↓
Model Processes Context
↓
Predicts Next Token
↓
Predicts Next Token
↓
Predicts Next Token
↓
...
↓
Final Response

The model generates the response piece by piece.

For example:

Recursion
→ is
→ a
→ programming
→ technique
→ where
→ a
→ function
→ calls
→ itself...

The actual process is much more complex, but this gives a basic idea.

🌟 Three Types of Generative AI

We can broadly think about generative AI in three categories:

Examples:

Text AI

«Prompt: "Write a Python program that calculates factorial."»

Output:

def factorial(n):
if n == 0:
return 1
return n * factorial(n - 1)

Image AI

«Prompt: "A small robot coding at a computer."»

Output:

🖼️ A generated image matching the description.

Video AI

«Prompt: "A spaceship flying through a colorful galaxy."»

Output:

🎬 A generated video showing a spaceship moving through a galaxy.

🚀 The Future of AI

AI is developing very quickly.

We are moving toward systems that can work across multiple types of information, including:

Text
Images
Video
Audio
Code

These are often called multimodal AI systems.

Imagine asking an AI:

«"Look at this image, explain what is happening, write code based on it, and create a video demonstrating the result."»

The ability to combine different types of information could make AI even more useful for developers, designers, students, researchers, and businesses.

🎯 Final Thoughts

AI doesn't learn exactly like a human.

Instead, AI models are trained on large amounts of data and use mathematical techniques to learn patterns.

A simplified picture is:

         DATA
           │
           ▼
       TRAINING
           │
           ▼
     AI MODEL LEARNS
       PATTERNS
           │
           ▼
      USER PROMPT
           │
           ▼
    GENERATED OUTPUT
           │
    ┌──────┼──────┐
    ▼      ▼      ▼
  TEXT   IMAGE   VIDEO

Text AI learns patterns in language.

Image AI learns patterns related to visual content.

Video AI learns patterns involving both visual content and motion over time.

The technology is complex, but the basic idea is simple:

«AI learns patterns from data and uses those learned patterns to generate predictions and new content.»

As AI continues to improve, understanding how it works will become an increasingly important skill for everyone interested in technology and computer science. 🚀

💬 What do you think?

Which type of AI interests you the most?

📝 Text AI
🖼️ Image Generation
🎬 Video Generation
💻 AI Coding Tools

Share your thoughts in the comments!

How Is ""2" > "10"" "true" in JavaScript?

Prince Singh — Sun, 02 Aug 2026 00:53:24 +0000

What is output of ""2">"10""
true or false

At first glance, this looks completely wrong.

We all know that:

2 > 10

is obviously:

false

Because 2 is smaller than 10.

But what happens when we add quotation marks?

"2" > "10"

The result is:

true

😱 Wait… how can 2 be greater than 10?

The answer is simple: ""2"" and ""10"" are not numbers. They are strings.

🔢 Numbers vs. Strings

In JavaScript, these two values are different:

This is a number.

While:

"2"

This is a string.

The quotation marks tell JavaScript that the value should be treated as text.

So:

2 > 10

compares two numbers:

2 > 10 → false

But:

"2" > "10"

compares two strings.

And that's where things get interesting!

🔤 How Does JavaScript Compare Strings?

When JavaScript compares two strings, it uses lexicographical comparison.

You can think of this as comparing text in a dictionary-like order, based on the character values.

Let's compare:

"2"
"10"

JavaScript looks at the first character of each string:

"2" → first character is 2
"10" → first character is 1

Since the character ""2"" comes after ""1"" in the ordering used for the comparison, JavaScript determines:

"2" > "10"

as:

true

So:

console.log("2" > "10");

outputs:

true

🧪 Let's Compare Both Cases

Case 1: Numbers

console.log(2 > 10);

Output:

false

Because JavaScript compares the actual numerical values:

2 is less than 10

Case 2: Strings

console.log("2" > "10");

Output:

true

Because JavaScript performs a string comparison.

The important thing is:

2 → Number
"2" → String

The quotation marks can completely change how JavaScript interprets the value.

⚠️ What About Mixed Types?

Now look at this:

console.log("2" > 10);

Here, one value is a string and the other is a number.

JavaScript handles this differently. In this case, it converts the string ""2"" into a number for the comparison.

So the comparison effectively becomes:

2 > 10

The result is:

false

This is why understanding data types is extremely important when programming.

💡 The Big Lesson

These three comparisons may look similar:

2 > 10
"2" > "10"
"2" > 10

But JavaScript can handle each one differently because the data types are different.

🚀 Final Takeaway

The computer isn't "wrong" when it says:

"2" > "10"

It's simply following the rules of string comparison instead of mathematical comparison.

So always remember:

«🔢 "2" is a number.
📝 ""2"" is a string.»

And when you're debugging a strange comparison in JavaScript, check the data types first!

That's one of the many reasons why understanding how programming languages handle types and comparisons is so important. 👨‍💻

Have you ever encountered a programming result that looked completely wrong at first? 🤔

Share it in the comments! 👇

javascript #webdevelopment #programming #coding #beginners #computerscience #100daysofcode