You've typed something into Claude Code, ChatGPT, or Gemini more times today than you'd probably admit. You've also, at some point, stared at a bill that came in higher than expected, or watched one prompt cost noticeably more than a seemingly similar one, and just shrugged rather than work out why. The honest reason is almost always the same: you're being charged per token, and most people have never actually had tokens explained properly.
The LLM doesn't 'understand' you
Underneath, the Large Language Model (LLM) powering the answer to your question can't actually read your text. It doesn't know the difference between a rabbit and a cup of tea. It needs your words broken down into tokens first because behind the scenes the models compute on numbers. Natural Language Processing is a big space in the AI world, and I won't cover it all here, but tokenization specifically is worth understanding, not least because it's the actual unit you're billed in.
Tokenization for toddlers
It helps to start with an analogy. Imagine a child learning a language at school: first the alphabet, then simple words, then gradually more complex vocabulary and full sentences as they get older. Now imagine that child is an LLM, except it can only ever work with numbers, never text. So the first step is a tokenizer, whose job is to translate characters into numerical IDs. a might become 1, b might become 2, and so on. Once that base layer exists, the tokenizer starts noticing which strings of characters keep showing up together. It might notice that "an" appears as a standalone word, and also inside common words like "and," "gran," and "ban," and assign that pattern its own ID, say 100. Repeat that process across a huge amount of text, and the tokenizer ends up with common words compressed into a single token each, while rarer words get split across several tokens, since it's cheaper to build them from pieces already in the vocabulary than to give every possible word its own ID. Worth noting: The tokenizer has already learned what words match what ID over a vast training set before the model was even deployed, so at runtime the tokenizer is simply a translation layer from word to ID number. Also every model does this slightly differently, so GPT-5 and Claude Sonnet 5 won't necessarily tokenize the same sentence into the same tokens, even though the underlying method is similar.
An example makes this concrete:
"Show me how many tokens this requires" demanded the hatter. comes out to 14 tokens in OpenAI's GPT-5 tokenizer. Almost every word gets one token, plus one token per piece of punctuation. The exception is "hatter": it's a real English word, yet it costs two tokens instead of one. It's simply not common enough to have earned its own ID, so the tokenizer falls back to building it from "hat" and "ter," both of which are common enough to already have one. You can try this yourself at platform.openai.com/tokenizer. Worth playing with a capital "H" on "Hatter" too, and seeing where the split moves.
What that actually costs you
With that model in place, the pricing tables providers publish actually mean something. Current (as of 15/08/26) Claude Code pricing, per million tokens:
| Model | Input Tokens | Output Tokens | Ability |
|---|---|---|---|
| Claude Fable 5 | $10 / MTok | $50 / MTok | Ultra |
| Claude Opus 5 | $5 / MTok | $25 / MTok | High |
| Claude Sonnet 5 | $2 / MTok | $10 / MTok | Medium |
| Claude Haiku 4.5 | $1 / MTok | $5 / MTok | Low |
Run Claude Haiku 4.5 for long enough to use 1 million input tokens and get 1 million output tokens back, and that's $6. The jump between models is steep: Opus 5 costs 5x that of Haiku for the same volume, and Fable 5 costs 10x. That same $6 becomes $30 on Opus, and $60 on Fable. Model choice alone moves your bill by an order of magnitude before you've changed anything else about how you work, which is exactly why it's worth understanding how tokens actually get used, not just what they cost.
So, what is a token?
Strip away the tokenizer mechanics and the pricing table, and a token is just the actual unit of text an LLM operates on, and the unit you're billed by. Not a word, not a character, whatever fragment of text the tokenizer decided was common enough to earn its own numerical ID. Common words get one. Rarer ones get built from pieces. Every model draws that line differently.
That's also the whole reason this is worth knowing rather than just accepting the bill each month. Tokens are the atomic unit everything else in this series builds on, how much context a session accumulates, why one model costs five times another, why a vague prompt burns through more of them than a precise one. Understanding the unit is what makes the rest of it legible.
Up next: Looking Through Claude's Context Window.
Top comments (0)