Models and pricing count tokens, but chunking libraries usually count characters — and the two don't map cleanly. The rough rule is ~4 characters per token for English prose, but it varies with content, code, and language.
Character counts mislead because a 600-character chunk isn't a fixed number of tokens. To budget context and cost accurately, you need the token count, not the character count.
Why tokens, not characters
Everything that matters downstream is measured in tokens: the context window the model can hold, the embedding cost, the generation cost. But the chunking step usually operates on characters, because that's what's easy to split on. This mismatch is a quiet source of surprises — you set a character size, and the token reality turns out different from what you assumed.
The rough rule, and where it breaks
For English prose, a token is roughly four characters — so ~600 characters is ~150 tokens, give or take. It's a useful rule of thumb, but it breaks down in exactly the cases you care about. Code tokenizes differently from prose. Numbers, punctuation, and rare words split into more tokens. Other languages diverge from the English ratio entirely. The rule is a starting estimate, not a guarantee.
~4 characters per token — until it isn't.
Why the mismatch costs you
- Context budget — if you assume 600 characters is fewer tokens than it is, you can overflow the context window you planned.
- Cost estimates — embedding and generation are priced per token, so a character-based estimate can be off by a wide margin.
- Retrieval tuning — 'top-k = 5 chunks' means very different token loads depending on real chunk token sizes.
Free RAG Chunk Visualizer — see your chunks, token counts, and quality flags in the browser. Try it free.
Estimating tokens without the exact tokenizer
The precise answer comes from the exact tokenizer your model uses. But for planning, a good subword estimate — one that accounts for word length, punctuation, and numbers rather than just dividing characters by four — tracks real tokenizers closely enough to budget confidently. The point is to get away from raw character counts, which are the least accurate signal.
See tokens per chunk
The RAG Chunk Visualizer estimates tokens for every chunk and the whole document, using a subword heuristic rather than a crude character divide. You can see immediately whether your chunks land near your token target and how the total maps to embedding and generation cost — the character-to-token guesswork removed.
Go further: the Full Edition adds overlap control, cost-model presets, top-k modelling, strategy comparison, JSON export, and vector-DB record preview. Get the Full Edition. Built as a companion to RAG: The Complete Guide.
FAQ
How many tokens is a RAG chunk?
For English prose, roughly one token per four characters — so a 600-character chunk is about 150 tokens. But it varies: code, numbers, punctuation, and other languages tokenize differently, so character counts are only a rough estimate.
Why not just count characters?
Because models and pricing run on tokens, and characters don't map cleanly to tokens. A character-based estimate can overflow your context budget or throw off cost estimates, especially for code, numbers, or non-English text.
How do I estimate tokens without the model's tokenizer?
Use a subword heuristic that accounts for word length, punctuation, and numbers rather than dividing characters by four. It tracks real tokenizers closely enough for budgeting context and cost.
How many characters per token?
About four characters per token for English prose, as a rule of thumb. It breaks down for code (which tokenizes densely), numbers, punctuation, and other languages, so treat it as a starting estimate, not a fixed rate.
How can I see tokens per chunk?
A chunk visualizer can estimate tokens for each chunk and the whole document using a subword heuristic, showing whether chunks land near your token target and how the total maps to cost — without the character-to-token guesswork.
Top comments (0)