AI is moving fast, and it feels like there's a new concept to learn every week. In an effort to actually understand this whole new world instead of just skimming past it, I started writing these ELI5 articles breaking down the concepts that show up constantly in AI conversations but rarely get explained simply. First up: tokens, and what's actually happening when you send a prompt.
Tokens, tokens everywhere
Every time you type into ChatGPT, Claude, or Copilot, your text doesn't go into the model as words. It gets chopped up first, into pieces called tokens. Sometimes this is a whole word, sometimes half a word, sometimes just a punctuation mark. The model never sees your sentence. It sees a sequence of numbers standing in for those pieces.
This is one of those things that's easy to skip over as a detail, but once you understand it, a bunch of AI behavior suddenly makes sense: why long prompts cost more, why some words seem to trip models up, why "count the letters in this word" is weirdly hard for an LLM.
Step one: splitting the text
Take the word "accessibility." A tokenizer doesn't treat it as one unit/word. It's common enough to be one token, but a lot of real-world text splits into recognizable chunks. Common short words like "the" or "cat" are usually a single token. Longer, less frequent, or more technical words often get broken into two or more pieces.
Step two: turning pieces into numbers
Each token maps to a number, based on a fixed vocabulary the model was trained with. So "access" might become 5426, and "ibility" might become 9821. The model doesn't process letters or meaning directly. It processes a list of numbers and predicts what number is statistically likely to come next, then translates that number back into text for you to read.
This is why an LLM can struggle with something like reversing a word or counting how many "r"s are in "strawberry". That's because it's not looking at s-t-r-a-w-b-e-r-r-y character by character. It's looking at a couple of opaque token chunks and has no direct access to the letters inside them.
Fun facts of tokanization
Whitespace and punctuation count as tokens too. A sentence full of short words and spaces can end up costing more tokens than a shorter sentence built from fewer, longer words. Token count isn't the same as word count or character count.
Non-English text often tokenizes worse. Most models are trained primarily on English text, so the same sentence in another language can break into noticeably more tokens to represent the same meaning.
Token limits define the "context window." This is the part that matters most if you're building products on top of these models, not just using them. Every model has a maximum number of tokens it can hold in memory at once — your prompt, the conversation history, and the response all share that budget. If you go over this limit the context will get condensed: older messages get dropped, input gets truncated, or you hit an error.
Why this matters if you're a frontend engineer
Tokenization isn't just a backend or ML-team concern anymore. Most products nowaday build something that streams AI output, that means you're designing around token budgets whether you realize it or not:
- Streaming UX: responses arrive token by token, not word by word, which is why text sometimes appears to render in odd fragments before normalizes.
- Truncation handling: what does your UI do when a conversation hits the context limit?
- Cost and latency: token count drives both API cost and response time, so a feature that quietly sends more context than it needs is a real product cost, not just a technical detail.
Understanding tokens won't make you a machine learning engineer. But it will make you a better engineer of the products sitting on top of these models, which, increasingly, is most of us.
Resources
- OpenAI Tokenizer — paste any text and see it split into tokens live: platform.openai.com/tokenizer
- Anthropic's token-counting docs — how context windows and token limits work across Claude models: docs.claude.com
- tiktoken — the open-source library OpenAI uses for tokenization, if you want to count tokens programmatically: github.com/openai/tiktoken
- Hugging Face Tokenizers — for exploring tokenization across different open models, not just one vendor: huggingface.co/docs/tokenizers
Top comments (0)