When I first heard the word "token" in the context of LLMs, I assumed it meant words. Made sense — "I love cricket" is 3 words, so 3 tokens, right?
Wrong. And that misunderstanding cost me real money before I figured out why.
What is a token — really
Let's start from the beginning.
A model doesn't read text the way you do. Before it processes a single character, it breaks your text into pieces called tokens. These pieces are not words. They are not characters. They are something in between — and the exact split depends on a vocabulary the model was trained with.
Let me show you with a real example. Take this sentence:
"I love cricket"
You might expect: 3 tokens (one per word). Here's what actually happens:
| Text | Tokens |
|---|---|
I |
1 token |
love |
1 token (note: the space is part of the token) |
cricket |
1 token |
Okay, so 3 tokens here. Your instinct was right this time. But try this:
"Microservices architecture"
| Text | Tokens |
|---|---|
Micro |
1 token |
services |
1 token |
architecture |
1 token |
That's 3 tokens for 2 words. The model split "Microservices" into two pieces because it learned that Micro and services are more common building blocks than Microservices as a whole.
Now try a number:
"2024"
That's 1 token. But:
"20241215"
That might be 3-4 tokens depending on the model. Long numbers get split unpredictably.
And emojis? One emoji can be 2-4 tokens. A single 🚀 costs more tokens than the word "rocket."
How tokenisation actually works
Every LLM uses something called a tokenizer — a fixed vocabulary of text pieces, built during training. GPT models use a tokenizer called BPE (Byte Pair Encoding). Gemini has its own. Claude has its own.
Here's how it works conceptually:
The tokenizer has a vocabulary of ~50,000 to ~100,000 text pieces. Common words like "the", "is", "and" are single tokens. Less common words get split into smaller pieces. Very rare words or made-up words get broken down to almost character level.
This means:
- Common English words → usually 1 token
- Long or technical words → 2 or more tokens
- Code → often more tokens than equivalent English (special characters, indentation)
- Non-English languages → often 2-3x more tokens than English for the same meaning
- Emojis, special characters → unpredictable, often expensive
Is it different for GPT vs Gemini vs Claude?
Yes. Each model has its own tokenizer with its own vocabulary. The same text will produce a different token count on GPT vs Gemini vs Claude. Not dramatically different — but different. If you're comparing costs across models, you can't just copy one model's token count to another.
Why this matters — the context window
Now that you know what a token is, here's why it matters.
Every LLM has a context window — a hard limit on how many tokens it can process in a single call. Think of it like RAM. The model can only "see" and work with whatever fits inside that window at once.
When you make an API call, you're not just sending the user's message. You're sending:
- Your system prompt ("You are a helpful assistant that only answers about...")
- The conversation history (every message back and forth so far)
- Any documents you want the model to read and use
- The user's current message
All of that together must fit within the context window. In tokens.
Let's put real numbers on this:
| What you're sending | Approximate tokens |
|---|---|
| System prompt | 300 – 800 |
| Last 10 messages of conversation | 1,500 – 3,000 |
| 3 retrieved document chunks (RAG) | 1,500 – 3,000 |
| User's current message | 20 – 100 |
| Total | ~3,300 – 6,900 |
Gemini Flash has a 1 million token context window. So 6,900 tokens seems fine — and it is, for one call. But you pay for every token. And if you're not careful about what you send, those numbers grow fast.
What happens when you exceed the context window
The model doesn't silently ignore the extra text. It throws an error. Your API call fails.
But here's the sneaky part — you often don't hit the hard limit. Instead, you approach it gradually, and the model's quality degrades before you ever get an error.
Before we get into that, let's define something you'll hear constantly when building AI systems: chunks.
Imagine you have a 50-page product manual as a text file. You can't send all 50 pages to the model every time a user asks a question — that's too many tokens, too slow, too expensive. So instead, you split that document into smaller pieces. Each piece is a chunk. Maybe 200-300 words each. You store all those chunks, and when a user asks something, you find the 3-5 chunks most relevant to their question and send only those to the model.
That's what a chunk is — a small slice of a larger document, sized to fit comfortably inside the context window alongside everything else you're sending.
Now, back to the problem.
Researchers found something called the "lost in the middle" problem. Picture this: you send the model a long prompt — system instructions at the top, then 20 document chunks in the middle, then the user's question at the bottom. The model reads all of it. But it turns out models pay more attention to what's at the very beginning and the very end of the input. The stuff buried deep in the middle? It gets less attention.
Think of it like reading a very long email. You remember the opening line and the closing ask. The three paragraphs in the middle? Fuzzy.
So if you send 20 chunks hoping the model finds the right answer somewhere in them — it probably won't. The answer sitting in chunk 11 might as well not exist.
More context is not always better. 3 highly relevant chunks will give you a better answer than 20 loosely relevant ones. Quality of what you send matters more than quantity.
Code costs more tokens than English
This one surprises a lot of backend engineers.
If you're building a system that processes code — code review, documentation generation, code explanation — be aware that code is significantly more expensive in tokens than regular English.
Why? Because code has a lot of characters that aren't common in English — braces, semicolons, indentation spaces, underscores, camelCase names. The tokenizer wasn't primarily trained on code, so it breaks these down into smaller pieces.
A rough comparison:
| Content type | Words | Approximate tokens |
|---|---|---|
| Plain English | 100 | ~75 |
| Java code | 100 "words" | ~150-200 |
| JSON payload | 100 "words" | ~120-160 |
| SQL query | 100 "words" | ~100-130 |
So if you're sending a 500-line Java file to the model for review, you're looking at significantly more tokens than you'd expect from the line count alone. Factor this in when designing systems that handle code.
Non-English languages cost more too
If you're building for users who write in Hindi, Tamil, Arabic, Chinese, or most non-English languages — tokens will cost more.
The reason is the same: the tokenizer vocabulary was built primarily from English text. English words map efficiently to tokens. Non-English scripts — especially those with their own character sets — break down into many smaller pieces.
Hindi text can cost 2-3x more tokens than the equivalent English meaning. This matters if you're building a multilingual product and estimating API costs.
How to actually count tokens in your code
Don't guess. Measure.
Most SDKs give you a way to count tokens before making the call. In Spring AI with Gemini, you can log token usage from the response:
ChatResponse response = chatClient.prompt()
.user(userMessage)
.call()
.chatResponse();
// Token usage is in the metadata
Usage usage = response.getMetadata().getUsage();
log.info("Input tokens: {}, Output tokens: {}",
usage.getPromptTokens(),
usage.getGenerationTokens());
Log this for every call in development. You'll immediately see where your tokens are going. It's the same as profiling slow SQL — you can't optimise what you haven't measured.
One number to remember
If you remember nothing else from this article:
1,000 tokens ≈ 750 words ≈ a page and a half of text.
Every time you construct a prompt, think in pages. If your prompt is 5,000 tokens, you're handing the model 7-8 pages of text to read before it writes a single word back to you. Is all of that necessary? That's the question to ask.
What's next
We've covered what a token is and how the context window works. Next: what does the full payload you send to the model actually look like? System prompt, conversation history, user message — how are these structured, and how does the model use them? That's what the next article covers.
Still confused about how tokenisation works for a specific case? Drop it in the comments — happy to break it down.
Sham Prakash K — Backend Engineer, 4+ years in Java, Spring Boot, and distributed systems. Building AI backend infrastructure. Writing about what I actually learned, mistakes included.
Top comments (0)