cl100k_base and o200k_base: Why GPT-4o's Token Counts Changed

#ai #machinelearning #llm #programming

Your token estimator was accurate on gpt-4 and started disagreeing with the usage object on gpt-4o. Nothing is broken: GPT-4o uses a different tokenizer, with roughly twice the vocabulary, and the same string does not decompose the same way.

Two encodings, one library

OpenAI publishes its byte-pair encodings in tiktoken, and each model id is mapped to exactly one of them. The two that matter today:

cl100k_base — GPT-4, GPT-4 Turbo, GPT-3.5 Turbo, and the text-embedding-ada-002 and text-embedding-3 models. tiktoken reports a vocabulary of 100,277 tokens.
o200k_base — the GPT-4o family, including gpt-4o-mini, and the o-series reasoning models. tiktoken reports 200,019 tokens.

The names encode the size: roughly 100k and roughly 200k entries. That doubling is the entire story of why counts changed.

The mapping from model id to encoding lives in tiktoken and is updated when new models ship. Read it with encoding_for_model rather than hard-coding a name, and upgrade the library when you adopt a new model — an old tiktoken does not know about a model released after it and will raise KeyError rather than guess.

Running a string through both

The comparison is four lines. Run it on your own prompts rather than taking anyone else’s numbers, because the ratio depends entirely on what your text is made of:

import tiktoken

cl = tiktoken.get_encoding("cl100k_base")     # GPT-4, GPT-3.5
o2 = tiktoken.get_encoding("o200k_base")      # GPT-4o, o-series

print(cl.n_vocab, o2.n_vocab)                 # 100277 200019

for s in [
    "The quick brown fox jumps over the lazy dog.",
    "def normalise(xs):\n    return [x / sum(xs) for x in xs]",
    "こんにちは、世界。今日はいい天気ですね。",
]:
    a, b = len(cl.encode(s)), len(o2.encode(s))
    print(f"{a:>4}  {b:>4}  {b/a:5.2f}x  {s[:32]!r}")

And the safer way to do it in application code, which never lets you pair a model with the wrong encoding:

enc = tiktoken.encoding_for_model("gpt-4o")    # -> o200k_base
enc = tiktoken.encoding_for_model("gpt-4")     # -> cl100k_base

One caveat that makes local counts disagree with the bill even when the encoding is right: encode counts the text, and the API counts the assembled chat request, which adds a handful of tokens per message for role and delimiter structure. That overhead is the subject of counting tokens across a multi-turn conversation.

Why a bigger vocabulary means fewer tokens

A BPE tokenizer starts from bytes and repeatedly merges the most frequent adjacent pair into a single new token, until the vocabulary is full. The vocabulary size is therefore a budget: every entry spent on one multi-character sequence is an entry not spent on another. With 100k entries, the common English words, the frequent code fragments and the punctuation get their own tokens, and most of everything else is assembled from fragments. With 200k, the budget stretches far enough to give whole words in other scripts, longer identifiers and more multi-byte sequences their own single token.

Fewer tokens for the same string is not free. Each of those tokens now selects from a distribution over twice as many candidates, so the model’s embedding and output layers are larger. The trade is real: a larger vocabulary buys shorter sequences, and shorter sequences are what you pay for and what fills the context window. Since attention cost grows with the square of sequence length, halving a sequence’s length is worth considerably more than a proportional saving.

Where the difference is largest

For ordinary English prose the two encodings land close together — the English vocabulary was already well covered at 100k, so there is little left to win. The gains are concentrated in text that cl100k had to spell out byte by byte. When OpenAI announced GPT-4o on 13 May 2024, it published a table of tokenizer improvements by language, reporting reductions of roughly 2.9x for Hindi, 2.0x for Arabic and 1.4x for Chinese on its sample text, against around 1.1x for the western European languages.

Two practical readings of that. If your workload is English, re-costing it under o200k changes your numbers by a few percent and you should still do it, because a few percent of a large bill is a real number. If your workload is Hindi, Arabic, Japanese or Korean, the change is large enough to alter which model is cheapest for you, since price per token multiplied by tokens per document is the only figure that matters and the second factor just moved.

Code and structured text are the other place the two encodings diverge noticeably, and in a direction that is easy to reason about: both encodings dedicate entries to runs of whitespace and to common identifier fragments, and the encoding with twice the budget covers more of them. Deeply indented source, JSON with long key names, and logs with repeated field labels are all compressible in ways plain prose is not. If your workload is mostly code, measure it rather than assuming the language table applies to you — it was built from natural language samples.

Special tokens, and the exception encode raises

Both encodings reserve a small set of tokens that are not text. In cl100k these include the chat delimiters — the markers that separate one message from the next in the format the model was trained on — plus an end-of-text token. They exist above the ordinary vocabulary and are how a chat request becomes one flat sequence the model can read.

Their practical consequence is a surprising exception. By default, encode refuses to tokenize a string containing the literal text of a special token:

enc.encode("<|endoftext|>")
# ValueError: Encountered text corresponding to disallowed special token
# '<|endoftext|>'. If you want this text to be encoded as a special token,
# pass allowed_special={"<|endoftext|>", ...}.

This looks like a nuisance and is a safety rail. If user-supplied text were silently encoded as a real delimiter, that text could impersonate the boundary between messages — a prompt injection carried out at the tokenizer rather than in the prose. Refusing by default means you have to decide explicitly. For counting tokens in untrusted input, the correct call treats the sequence as ordinary text:

n = len(enc.encode(user_text, disallowed_special=()))

That counts the literal characters and never mints a control token. The opposite setting exists for building a training corpus by hand, where you genuinely do want the delimiter, and almost never for anything serving user traffic.

What in your code is still on cl100k

The encoding is usually chosen once, early, and then inherited by everything. The places it hides:

The chunker in your retrieval pipeline. If chunks are sized in tokens under cl100k and then embedded and sent to a GPT-4o context, every downstream budget is off. Worth noting that this one is genuinely subtle, because text-embedding-3-small and -large really do use cl100k_base — so the chunker may be right for the embedding step and wrong for the generation step at the same time.
The pre-flight length check. The guard that refuses an over-long request before sending it. Under the wrong encoding it either rejects requests that would have fit or lets through requests that will 400 on the context window.
Cost estimates in dashboards. These are usually written once against whatever model was current and never revisited. Compare an estimate to the usage object on real traffic; that object is ground truth and it is free.
Anything using a non-OpenAI tokenizer as a stand-in. A Llama or Claude token is not an OpenAI token and never was. Model families do not share vocabularies, which is why per-model token accounting is the only kind that works.