tiktoken vs count_tokens: My Claude Budget Was 17% Off

My budget guard said the prompt was 171,000 tokens. Haiku 4.5's context window is 200,000. Plenty of room. The API returned a 400 anyway: prompt too long.

That was run 1,102 of a pipeline I'd been babysitting for three weeks, and it was the first time I seriously questioned the little function at the top of my code that counted tokens with tiktoken. So I logged both numbers — my local estimate and the usage the API actually reported — for every request for the rest of the month. This is the tiktoken vs count_tokens comparison across 4,200 calls, and how far off the cheap local estimate really was.

TL;DR

• tiktoken is OpenAI's tokenizer. It does not know Claude's vocabulary, so every number it gives you for a Claude prompt is a guess.
• Across 4,200 real requests my tiktoken estimate ran a median 17.4% below the input tokens the API billed. Worst bucket (JSON tool results) was 38% low.
• Most of my error wasn't even the tokenizer. I tokenized messages and forgot that tool definitions and the system prompt are part of every request — 1,318 + 900 tokens I counted as zero.
• The fix is POST /v1/messages/count_tokens with the same body you're about to send. It matched usage exactly on every call I checked. Cost: one extra round trip, median 240ms.
• Do not compare your estimate against usage.input_tokens alone. With prompt caching on, the bulk of your input shows up in cache_read_input_tokens and your comparison will look insane.

What was I actually running?

A cron pipeline that triages GitHub issues across my repos. Haiku 4.5 does the cheap first pass (read the issue, the linked file, the last 20 commits touching it, classify and summarize), Opus 5 drafts a patch for anything classified as a real bug. About 200 calls a day, 9 tools, a system prompt with a generated repo map glued to the end.

The whole thing had a guard in front of it:

import tiktoken
enc = tiktoken.get_encoding("cl100k_base")

def estimate(messages) -> int:
    return sum(len(enc.encode(json.dumps(m))) for m in messages)

if estimate(messages) > LIMIT:
    messages = drop_oldest(messages)

Looks responsible. It's fiction with a dependency.

Why does tiktoken undercount Claude tokens?

Because it's a different tokenizer with a different vocabulary. cl100k_base is OpenAI's BPE merge table. Claude has its own, and the merges don't line up, so the same string splits into a different number of pieces. There is no conversion ratio you can multiply by — the drift depends entirely on what's in the text.

That last part is the trap. If the error were a flat 15%, you'd add a fudge factor and go home. It isn't flat. On my traffic the gap tracked content type hard:

Content bucket	Median undercount
Plain English issue bodies	9%
Python and TS diffs	24%
Serialized JSON tool results	38%
Stack traces and log dumps	29%

So a fudge factor tuned on prose blows up the moment the agent pastes a 400-line diff, which is exactly when the prompt is big enough to matter. My run 1,102 failure was a request that was 94% code by volume.

How wrong was tiktoken vs count_tokens across 4,200 calls?

Median 17.4% low, p95 34% low, best case 6% low. It was never once high. An estimator that only errs in the dangerous direction is worse than no estimator, because the guard is the thing that made me stop thinking about it.

Concrete damage over 21 days:

• 19 runs died on context-limit 400s that my guard had waved through.
• Input spend ran 21% above what the estimator's forecast implied for the month. Tokens are the unit you're billed in, so an undercount is a bill surprise with extra steps.
• My truncation logic fired late. drop_oldest triggered off the same wrong number, so the pipeline kept whole issue threads it should have dropped and dropped them only after a failure.

The counting bug and the retry bill compound: every 400 was followed by a truncate-and-retry, so the worst prompts got paid for twice.

What does count_tokens count that your estimator doesn't?

Everything the model actually reads. This was the bigger half of my error and it has nothing to do with tokenizers.

A request isn't messages. The API renders tools, then system, then messages. My nine tool schemas came to 1,318 tokens, on every single call, sitting in a variable my estimator never saw. The system prompt with its generated repo map added another ~900. That's 2,218 tokens of pure blind spot before the first message.

The endpoint takes the same body shape as messages.create, which is the entire point:

from anthropic import Anthropic

client = Anthropic()

resp = client.messages.count_tokens(
    model="claude-haiku-4-5",   # counts are model-specific
    system=SYSTEM_PROMPT,
    tools=TOOLS,
    messages=messages,
)
print(resp.input_tokens)       # exact, for this model, for this body

Two things worth internalizing. Counts are model-specific — pass the model you're actually going to call, not whichever ID you had in a constant. And if you build the body in one place and count it in another, you will eventually count a body you don't send. That was my only remaining mismatch after the switch, and it was my bug, not the API's.

Why did my first comparison look completely broken?

Because of prompt caching, and this cost me an evening. My first pass compared estimate() against response.usage.input_tokens and got results like estimate 41,000 versus actual 812. I assumed my logging was wrong.

It wasn't. With caching on, input_tokens is only the uncached portion. The rest is reported separately in cache_read_input_tokens and cache_creation_input_tokens. If you want the number to compare a prompt-size estimate against, you have to add them:

u = response.usage
billed_input = (
    u.input_tokens
    + (u.cache_read_input_tokens or 0)
    + (u.cache_creation_input_tokens or 0)
)

Useful side effect: once you're summing those three, a cache_read_input_tokens of zero across repeated requests is a loud signal that something in your prefix is changing between calls and your cache is silently never hitting.

Does one extra count_tokens call per request slow you down?

On my pipeline, no. Median 240ms, p95 610ms, against calls that already take multiple seconds. 200 counts a day is noise.

In a tight per-user loop it's a different story, so two things I'd actually do:

Count the static prefix once. Tools plus system don't change between runs. Count them at startup, cache the number, and only count the volatile messages per request. One caveat I'd rather you hear from me than discover: token counts are not perfectly additive. Counting the whole body and summing the parts differed by 0 to 4 tokens on my requests. Tiny, real, and enough that I pad the budget by 1% instead of pretending the arithmetic is exact.

Don't count in a fan-out. I briefly counted every candidate chunk in a retrieval step and caught a 429 — the endpoint has its own limits and it is not free of rate limiting just because it's cheap. Count the assembled prompt, once.

The honest limitation: count_tokens tells you the input exactly and the output not at all. Output is unknowable until it's generated. I still reserve headroom equal to max_tokens on top of the counted input, which is the only part of my original guard that survived.

So should you use tiktoken to count Claude tokens?

No. tiktoken is OpenAI's tokenizer, and on 4,200 real Claude calls it undercounted my prompts by a median of 17.4% — never once erring high, and drifting worst (38%) on the JSON and code payloads that make prompts big in the first place. Half my error came from the wrong tokenizer and half from counting only messages while tool schemas and the system prompt quietly added 2,218 tokens to every request. Use client.messages.count_tokens() with the same model, system, tools, and messages you're about to send; it returns the exact input count for one extra round trip of a few hundred milliseconds. And when you compare it against reality, sum input_tokens with the two cache_* fields, or prompt caching will make your own logs look like they're lying to you.

---

Written by the developer behind Preterview, an interview prep platform.