The same paragraph sent to two providers produces two different token counts, and the gap is not noise. It comes from three unrelated causes, one of which means you cannot compute Claude’s number without asking Anthropic for it.
One is published, the other is not
OpenAI publishes its tokenizers. tiktoken is an open library, and the encoding used by the GPT-4o family, o200k_base, ships in it. You can count tokens for a million documents on your laptop, offline, for free, in a loop.
Anthropic does not publish an equivalent. There is no library that reproduces Claude’s tokenization locally, and the vocabulary is not distributed. The supported way to get a Claude token count is the count_tokens endpoint, which is a network call to Anthropic.
This asymmetry is the first thing to design around, and it has nothing to do with which count is larger. It means any pipeline that counts tokens per chunk — a RAG chunker, a truncation guard, a cost estimator in a UI — either makes a network call per chunk, or estimates. Most production code estimates: it counts with tiktoken, applies a margin, and verifies against the real number at the boundary where being wrong is expensive.
Do not use a GPT tokenizer count as a Claude count without a margin, and do not assume the margin is stable across content types. It is not a constant; see the measurement procedure below.
Reason one: different vocabularies
Both providers use byte-pair encoding, and BPE is not one algorithm producing one answer. It is a procedure: start from bytes, repeatedly merge the most frequent adjacent pair, stop at a target vocabulary size. The resulting merge table depends entirely on the corpus it was learned from and on where you stopped.
Two consequences follow. First, vocabulary size matters: a larger vocabulary can represent more whole words and common phrases as single tokens, so it generally produces fewer tokens for the same text. The GPT-4 generation moved from a roughly 100,000-token encoding to a roughly 200,000-token one partly for that reason, and the gain was largest on non-English text, where the smaller vocabulary had been falling back to multi-token byte sequences.
Second, corpus composition matters. A tokenizer trained on a corpus heavy in code will have learned merges for self., </div> and four-space indents. One trained on a different mix will not. So the ratio between two providers’ counts is not a property of the tokenizers alone — it is a property of the tokenizers and your text. English prose, Python, minified JSON, Japanese and base64 will each give you a different ratio, and base64 will give you a terrible one on both.
Reason two: what counts as the message
Even with an identical tokenizer, two APIs would disagree, because the thing being counted is not your string. It is the serialised conversation, including whatever delimiters the chat format uses to mark where a turn begins and which role is speaking.
OpenAI documents a per-message overhead on top of the content tokens — a handful of tokens for the role and the message boundaries, plus a fixed amount for the reply priming. Anthropic’s format has its own scaffolding. Neither overhead is large per message, and both are large in aggregate: a thousand-turn conversation history carries a thousand times the per-message overhead, and a chat application that resends its history on every request pays for that scaffolding on every request.
This is why a “count the characters and divide by four” heuristic drifts worst exactly where it matters most: on long multi-turn histories, where the error compounds per turn rather than per document.
The scaffolding is also not something you can inspect. Neither provider exposes the exact serialised form of a conversation, so there is no way to reconstruct the count from first principles even if you had the tokenizer — you would still be guessing at the delimiters. Both providers instead document the overhead approximately and give you an endpoint or a library function that includes it. Use those, and treat any counting code that operates on a bare string as an estimate by construction, however precise the tokenizer underneath it is.
Reason three: tools and system prompts
The largest single surprise in a real integration is usually not the conversation at all. Tool definitions are serialised into the prompt, and they are counted. A dozen tools with thorough descriptions and nested JSON schemas can be several thousand tokens, charged on every single request, before the user has said anything.
Anthropic’s count_tokens endpoint accepts the tools and system fields for exactly this reason. If you call it with only messages, the number you get back is not the number you will be billed, and the gap is the size of your tool definitions. The same is true of image blocks and PDF content — pass the request body you actually intend to send.
Where the tokens in one "small" request actually go:
system prompt 1,240
6 tool definitions 3,880 ← charged on every request
conversation (8) 2,610
user question 47
-----
total input 7,777
The user question is 0.6% of the input bill.
The figures above are an illustrative breakdown, not a measurement. Produce your own by calling count_tokens twice — once with the full body, once with tools removed — and taking the difference. That single subtraction has ended more cost investigations than any amount of prompt shortening.
Measuring the ratio for your own text
There is no useful universal ratio to quote, so measure it on a sample of your actual traffic. Both halves are a few lines.
- Collect a representative sample. Two hundred real documents or conversation histories, not lorem ipsum, and keep the content types separate if you handle several — count prose and code as two samples, not one.
-
Count the GPT side locally.
import tiktoken enc = tiktoken.get_encoding("o200k_base") gpt_tokens = sum(len(enc.encode(doc)) for doc in docs) print(gpt_tokens) -
Count the Claude side over the network, sending each document in the message shape you would really use.
import anthropic client = anthropic.Anthropic() claude_tokens = 0 for doc in docs: r = client.messages.count_tokens( model="claude-sonnet-4-5-20250929", system=SYSTEM_PROMPT, messages=[{"role": "user", "content": doc}], ) claude_tokens += r.input_tokens print(claude_tokens) Divide, and keep the spread. The mean ratio is less useful than the worst case. If you are sizing a truncation guard, the document with the highest ratio is the one that will blow the context window, not the average one.
Re-measure when you change models or content types. A new model generation can bring a new encoding, and a new customer segment can bring a new language.
A caution on the sample. Two hundred documents drawn from last week’s traffic is representative of last week; it is not representative of the customer who onboards next month with a corpus in a different language, or of the day somebody starts pasting spreadsheets into your chat box. The ratio is a property of the text, so it moves when the text does, and the guard you sized against the old ratio fails on the new one. Keep the measurement script rather than the number.
The reason to bother is that a price per million tokens is only comparable between providers if a token means the same thing on both sides, and it does not. Two models quoted at the same rate can differ in real cost for your text by a margin big enough to reverse the ranking — and the direction depends on what you send.
If you route the same workload across several providers, the accounting problem is the one above at scale: each provider counts in its own tokens, so a single number for “tokens used this month” across all of them is not a meaningful quantity. Multigrid records each request against the provider’s own reported usage and converts to cost per request, which is the unit that survives the comparison.
Top comments (0)