I Tokenized 10,000 Real Prompts: The "4 Characters 1 Token" Rule Is Off by More Than 20% on 4 Out of 10 of Them

For three months I priced every LLM feature with one number: 4 characters ≈ 1 token.

It is everywhere. It is in blog posts, in pricing calculators, in the back of my head when somebody asks "roughly how much will this cost". It feels safe because it is conservative: it rounds up the token count, so your estimate lands above the real bill.

Last week I stopped trusting it and measured it. 10,000 real samples, real tokenizer, no synthetic text.

The rule is bad per prompt. It is fine for the monthly total. Knowing which of those two you are doing turns out to be the whole game.

How I measured it

I did not want synthetic text, because synthetic text is where benchmarks go to lie. The corpus is 521 real posts pulled from the dev.to public API, across ai, webdev, python, programming, javascript and opensource.

Each post was stripped of front matter, then split into:

• prose — paragraphs with code fences removed
• code — the contents of fenced blocks
• structured — blocks that are JSON-ish or log-shaped

That gave a pool I sampled 10,000 from: 7,055 prose, 2,669 code, 275 structured. Every sample was encoded with cl100k_base through tiktoken, one at a time, and I compared the real count against len(text) / 4.

Two honest caveats before the numbers:

1. This is English technical writing. I got exactly one CJK sample in the whole corpus, so I make no claim about Chinese, Japanese, Korean or emoji-heavy text. If your traffic is multilingual, treat every number here as a lower bound on how wrong the rule is.
2. The mix is heavy on prose because that is what dev.to posts are. Do not read the "all" column as "the internet".

What the rule does to a single prompt

Measure	Result
Median error	+11.7% (the rule over-counts)
Median absolute error	16.3%
Samples off by more than 20%	40.5%
of those, over-counted	33.3%
of those, under-counted	7.2%
5th percentile	−23.9%
95th percentile	+46.8%

So on a single prompt, the rule is a coin flip that is usually wrong by more than a fifth. If you have ever sized a context window with chars / 4 and wondered why the request still got truncated, that 7.2% under-count is why. The rule is mostly conservative, but "mostly" is not "always", and the failure mode is the expensive one: you packed too much and the call failed.

The part I did not expect: the monthly bill was basically right

Here is the aggregate over all 10,000 samples:

• 2,715,542 characters
• 646,092 real tokens
• → 4.203 characters per token

Run chars / 4 over the whole pile and you get 678,885 tokens against a true 646,092. That is +5.08%.

Read those two sections together, because they look like they contradict each other. They do not:

• Per prompt: 40% of samples are off by more than 20%.
• In aggregate: the batch is off by 5%.

The errors cancel. Short samples and long samples, prose and code, they all pull the average toward 4.2 from different directions. So the rule is not "wrong" in the way people say it is wrong. It is unusable per call and acceptable per month. If you are budgeting a feature, chars / 4 will get you a number that is about 5% high, which is fine. If you are deciding whether one specific prompt fits in a 128k window, it will get you a number that is wrong one time in three.

Short prompts are where it actually breaks

I bucketed by length and this is the finding I now care about most:

Sample length	n	Median chars/token	Median abs error	Off by >20%
0–200 chars	5,455	4.500	17.5%	43.6%
200–500 chars	3,448	4.537	16.6%	41.4%
500–1500 chars	951	4.186	10.6%	24.6%
1500–4000 chars	146	4.018	4.8%	4.8%

The shorter the text, the worse the rule. Below 200 characters — which is most chat prompts, most system messages, most tool descriptions — it is off by more than 20% on 43.6% of samples. Above 1,500 characters it is nearly fine.

That is the opposite of what intuition suggests. You would think a tiny string is easy to estimate. It is not: a single rare word, an unusual identifier or one emoji can swallow a dozen characters into three tokens, and there is no averaging to save you.

Content type moves the constant

Type	n	Median chars/token	Weighted chars/token	Median error with /4
Prose	7,055	4.577	4.375	+14.4%
Code	2,669	4.156	4.010	+3.9%
Structured (JSON, logs)	275	3.778	3.877	−5.6%

Structured is the one that should worry you. With chars / 4 it under-counts: median error −5.6%, and 21.1% of structured samples were under-counted by more than 20%.

Which means: if your app emits JSON and you set a hard token cap by dividing characters by four, you are setting that cap too low roughly a fifth of the time. You will reject responses you could have afford. That is the mirror image of the truncation bug, and it is the one nobody writes about.

So which number should you use?

I tried a range of constants against the same 10,000 samples:

Constant	Median abs error	Median bias
4.0	16.3%	+11.7%
4.25	13.5%	+5.2%
4.47	11.9%	−0.03%
4.5	11.9%	−0.7%
4.75	12.1%	−5.9%

4.47 is the unbiased one — it splits the difference almost perfectly. 4.5 is indistinguishable in practice and easier to remember. But notice the ceiling: even the best constant still has a median absolute error of 11.9%. No single number fixes this. The distribution is too wide for a constant to win.

My rules now:

1. Budget with 4.5, bill with a real tokenizer. 4.5 for the spreadsheet, actual encode() before anything that costs money or hits a limit.
2. Never set a hard cap with chars / 4 on structured output. That is the one direction where the rule fails cheap instead of safe.
3. Anything under 200 characters, do not estimate. Measure it once, cache the number. These are the samples where the rule is worst and they are also the ones you reuse a million times, so one measurement pays for itself immediately.

Counting it for real

If you want to see where a tokenizer actually splits your text — which is the only way to build intuition for why résumé and resume are not the same price — I built a small counter that shows the split boundaries per word: LLM Token Counter. It runs in the browser, no upload.

If you would rather run it yourself in Python or Node, the walkthrough is here: How to count GPT tokens.

And for the case where characters are the unit you actually care about: Word & character counter.

One question

I can tell you about English technical prose. I cannot tell you about yours.

If you have measured this on your own traffic — especially non-English, chat logs, or agent traces — what constant did you land on? I would like to know whether 4.2 holds outside dev.to, or whether this whole thing is even more corpus-dependent than I think.

The raw data (10,000 rows: kind, chars, tokens, ratio, error) is in the repo with the script that produced it. Rerun it on your own text; that takes five minutes and beats trusting my constant.