Understanding Tokenization: AI Doesn't Actually Read Text the Way We Do

Notes from a full-stack developer exploring Generative AI.

Lately, I've been spending some time learning about Applied AI and Generative AI.

Coming from a full-stack web development background, I initially thought building AI-powered applications wouldn't be that different from integrating any other API.

You send a request, the model processes the input, and you get a response.

And technically, that's true.

But the more I learn about how Large Language Models (LLMs) work, the more I realize there's a lot happening behind the scenes before a model generates an answer.

One of the first concepts that caught my attention was tokenization.

It sounds pretty straightforward at first. But as it turns out, tokenization plays an important role in how language models process text, handle context, and even how much it costs to use them.

AI Doesn't Read Sentences Like Humans Do

When we read a sentence like:

I'm currently learning about Generative AI.

Our brains naturally recognize the words and understand the meaning behind the sentence.

But an LLM doesn't process raw text in quite the same way.

Before the model can work with the text, it goes through a process called tokenization.

In simple terms, tokenization is the process of breaking text into smaller units called tokens.

Here's where it gets interesting: one token doesn't necessarily mean one word.

A token can represent an entire word, part of a word, punctuation, or even individual characters, depending on the tokenizer being used.

For example, a tokenizer might break this sentence down like this:

Input:

I love programming!

Illustrative tokenization result:

["I", " love", " programming", "!"]

Notice how spaces can actually be part of a token.

With some tokenizers, "love" and " love" may even have different token IDs.

So tokenization isn't simply splitting a string by spaces like we might do with string.Split().

There's more going on under the hood.

Why Does AI Need Tokens in the First Place?

One question I had while learning about this was:

Why can't the model just read the text directly?

The answer comes down to how computers process information.

Language models operate on numerical representations, not words as we see them on a screen.

That means text needs to be converted into something the model can process mathematically.

At a high level, the process looks like this:

Text → Tokens → Token IDs → Embeddings → Model

Let's use the same example:

I love programming!

After tokenization, each token gets mapped to a number called a token ID.

Something like this:

Token	Token ID
I	40
love	1842
programming	9265
!	0

These IDs are just examples, not actual values from a specific tokenizer.

The token IDs are then used to look up numerical representations called embeddings.

These embeddings are vectors of numbers that the neural network can work with.

One important distinction here:

A token ID doesn't represent the meaning of a word.

It's simply a numerical identifier.

The richer representation comes from embeddings and how the model processes those representations in relation to the surrounding context.

This was one of those concepts that made me realize how different language processing is from traditional application logic.

Tokenization Also Helps Explain Why AI Sometimes Struggles to Count Letters

Have you ever asked a chatbot something like:

How many times does the letter "r" appear in "strawberry"?

For humans, this seems like a ridiculously simple question.

But language models haven't always been reliable at answering questions like this.

One reason is tokenization.

A model doesn't necessarily receive every individual letter as a separate unit. A word like "strawberry" might be represented by a few tokens, each containing multiple characters.

So the model isn't automatically working with a character-by-character representation.

Of course, tokenization isn't the only reason an AI might get this wrong.

The model's learned patterns, reasoning capabilities, and access to external tools also play a role.

More capable models can often answer these questions correctly, especially when they use additional reasoning or tools to verify the result.

Still, I think this is a great example of something that's easy for humans but not necessarily straightforward for an AI model.

And sometimes, the opposite is also true.

Tokens Aren't Just About Text Processing. They're About Cost, Too.

This is where tokenization becomes especially relevant for developers building applications with LLM APIs.

When working with these APIs, you'll frequently come across terms like:

• Input tokens: The tokens sent to the model, including instructions, user messages, and additional context.
• Output tokens: The tokens generated by the model as part of its response.
• Context window: The amount of context a model can handle within a single processing sequence, subject to model-specific input and output limits.

Why does this matter?

Because LLM API usage is generally priced based on the number of tokens processed, although input, output, and cached tokens may have different rates.

Imagine we're building an AI-powered customer support feature.

Every time a user sends a message, our backend might assemble several pieces of information:

1. A system prompt containing the chatbot's instructions.
2. Previous conversation history.
3. The user's latest question.
4. Additional documents or relevant information.

All of that consumes part of the model's available context.

Now imagine sending the entire conversation history with every request, even when most of it is no longer relevant.

The result?

Potentially higher costs, slower processing, and less room for the information that actually matters.

This is where I started realizing something important:

Prompt engineering isn't just about writing better instructions. It's also about managing the information we send to a model.

And that becomes an engineering problem, not just a writing problem.

The Language We Use Can Affect Token Count

Here's another interesting thing I learned.

Two sentences with the same meaning, written in different languages, don't necessarily produce the same number of tokens.

For example:

I want to learn programming.

and

Saya ingin belajar pemrograman.

Both sentences communicate roughly the same idea, but their token counts may differ depending on the tokenizer.

Why?

Because tokenizers are built around patterns found in the text data used to develop them.

Common character sequences may be represented as single tokens, while less common sequences might need to be broken into smaller pieces.

As a result, token efficiency can vary across languages.

Does that mean we should always use English to save tokens?

Not necessarily.

Token count matters, but so does the quality of the instructions and whether the model responds appropriately to the user.

If token efficiency is a concern, the better approach is to measure it using the actual tokenizer associated with the model we're using.

Rather than assuming one language will always be cheaper.

Why Should a Full-Stack Developer Care About This?

At first, I thought tokenization was something mainly relevant to machine learning engineers.

But the more I explored Applied AI, the more I realized that understanding these fundamentals can help us make better engineering decisions.

We don't need to build our own tokenizer to integrate an LLM into an application.

But knowing how tokens work becomes useful when designing features that involve large amounts of text.

Let's say we're building an application that reads and summarizes documents using AI.

The simplest approach might be to extract all the text from a document and send it directly to the model.

That might work for a short document.

But what happens when the document contains hundreds of pages?

Suddenly, we have several things to think about.

1. Managing the context window

We need to make sure the amount of information we're sending stays within the model's limits.

2. Splitting documents into chunks

Instead of sending everything at once, we can divide documents into smaller sections.

But chunking isn't as simple as splitting text into equal-sized pieces. Important context might be lost when related information ends up in separate chunks.

3. Controlling costs

More input tokens and longer generated responses can mean higher API costs.

For a feature used by thousands of users, small inefficiencies can become significant.

4. Retrieving only relevant information

Not every piece of information needs to be included in every request.

With techniques like Retrieval-Augmented Generation (RAG), we can retrieve relevant parts of a document before passing them to the model.

Tokenization doesn't solve all these problems by itself.

But understanding tokens makes it much easier to see why these problems exist in the first place.

What I Took Away From Learning About Tokenization

Learning about tokenization changed the way I look at LLM APIs.

Previously, I approached them like most backend services:

Send a request, receive a response, and integrate the result into the application.

Now I'm starting to realize that building AI-powered applications also requires understanding how the model receives, represents, and processes information.

Tokenization is only a small part of the overall LLM architecture.

But it's connected to several important concepts: embeddings, context windows, text processing, API costs, and the efficiency of AI-powered systems.

And the more I learn, the more I realize that building with Generative AI isn't just about successfully calling a model.

The real challenge is designing systems that provide the right context, generate useful responses, and remain efficient as they scale.

That's why I think understanding fundamentals like tokenization matters, even if our goal isn't to become an AI researcher.

Sometimes, being a better AI application developer starts with understanding what actually happens before the API returns a response.