Thursday AI Tool/Workflow. This week's tool: tiktoken, OpenAI's open-source BPE tokenizer. It turns text into the same token IDs that OpenAI models see, so you can count prompt tokens before you send the request. Use it to check that the prompt fits the context window, cut documents by tokens instead of characters, and estimate what a call will cost.
Most "context length exceeded" errors and surprise bills come from guessing "about 4 characters per token." On my samples that guess was right for none of them. English came out at 6.0 characters per token, while a Python snippet, a JSON payload and a Hindi sentence came out at 2.5-2.8.
Here's the workflow, with a script I ran on the box, the real output, and the gotchas I hit.
What it is
tiktoken is a fast byte pair encoding (BPE) tokenizer for OpenAI's models. The core is written in Rust and wrapped in Python. Per the README, BPE is reversible and lossless, works on any text (even text not in the tokenizer's training data), and compresses text, with each token covering "about 4 bytes" on average in practice.
The thing to get right is the encoding. Different models use different vocabularies:
-
o200k_base: the newer one.encoding_for_model()mapsgpt-4o,gpt-4o-mini,gpt-4.1,gpt-5and theo1/o3/o4-minifamilies to it. -
cl100k_base:gpt-4,gpt-3.5-turboand thetext-embedding-3-*/ada-002embedding models. -
p50k_baseandr50k_base: older Codex and GPT-3 models. The OpenAI Cookbook table lists them all.
1. Install
python -m venv venv && source venv/bin/activate
pip install tiktoken
I used tiktoken 0.14.0, the current release on PyPI and GitHub (published August 17, 2026, MIT license, Python 3.9+). The changelog for 0.14.0 says it builds wheels for Python 3.15 and supports looking up more GPT-5 series models. My box is an 8-vCPU Intel Xeon with no GPU. Tokenizing doesn't need one.
2. The workflow, in code
The script below does seven things:
- Picks the encoding with
get_encoding()andencoding_for_model(). - Counts tokens for four kinds of text under both encodings.
- Checks the encode/decode round-trip and looks at a few token pieces.
- Shows how special tokens are handled.
- Truncates and chunks text by tokens.
- Budgets a chat prompt and estimates its cost.
- Times encoding of about 2 MB of text.
# demo.py - count tokens with tiktoken before you call the LLM
import os, time, json, tempfile
# Keep the encoding files in a folder we control (default is <tmp>/data-gym-cache)
CACHE = os.path.join(os.path.dirname(os.path.abspath(__file__)), "tiktoken-cache")
os.environ["TIKTOKEN_CACHE_DIR"] = CACHE
import tiktoken
print("tiktoken", tiktoken.__version__, "| cache dir:", CACHE)
print("cached files before loading:", len(os.listdir(CACHE)) if os.path.isdir(CACHE) else 0)
# 1. Pick the encoding. First load downloads the BPE file; later loads read the cache.
for name in ("o200k_base", "cl100k_base"):
t = time.perf_counter(); enc = tiktoken.get_encoding(name)
print(f"load {name}: {(time.perf_counter()-t)*1000:.0f} ms, vocab size n_vocab={enc.n_vocab}")
o200k = tiktoken.get_encoding("o200k_base")
cl100k = tiktoken.get_encoding("cl100k_base")
print("\nencoding_for_model():")
for m in ("gpt-4o", "gpt-4o-mini", "gpt-4.1", "gpt-5", "gpt-4", "gpt-3.5-turbo", "text-embedding-3-small"):
print(f" {m:<24} -> {tiktoken.encoding_for_model(m).name}")
for m in ("claude-sonnet-4", "llama-3.1-8b"):
try:
tiktoken.encoding_for_model(m)
except KeyError as e:
print(f" {m:<24} -> KeyError: {str(e)[:70]}...")
# 2. Count tokens for different kinds of text
samples = {
"English": "Please summarize this support ticket in three bullet points.",
"Hindi (same meaning)": "कृपया इस सपोर्ट टिकट का तीन बिंदुओं में सारांश दें।",
"Python": "def add(a: int, b: int) -> int:\n return a + b\n\nprint(add(2, 3))\n",
"JSON": json.dumps({"user_id": 4821, "plan": "annual", "seats": 12, "tags": ["billing", "refund"]}),
}
print(f"\n{'sample':<22}{'chars':>6}{'utf8 bytes':>12}{'o200k':>7}{'cl100k':>8}{'chars/tok (o200k)':>19}")
for label, text in samples.items():
a, b = len(o200k.encode(text)), len(cl100k.encode(text))
print(f"{label:<22}{len(text):>6}{len(text.encode()):>12}{a:>7}{b:>8}{len(text)/a:>19.2f}")
# 3. Round-trip and token pieces
text = samples["English"]
ids = o200k.encode(text)
print("\nround-trip ok:", o200k.decode(ids) == text)
print("first 8 ids:", ids[:8])
print("first 8 pieces:", [o200k.decode_single_token_bytes(i) for i in ids[:8]])
hi = o200k.encode(samples["Hindi (same meaning)"])
print("Hindi first 6 pieces (bytes):", [o200k.decode_single_token_bytes(i) for i in hi[:6]])
print("decode of a single Hindi token:", repr(o200k.decode(hi[1:2])))
# 4. Special tokens: encode() refuses them unless you opt in
try:
o200k.encode("user text with <|endoftext|> inside")
except ValueError as e:
print("\nencode() with special token ->", type(e).__name__, str(e)[:60] + "...")
print("encode_ordinary() ->", len(o200k.encode_ordinary("user text with <|endoftext|> inside")), "tokens (treated as plain text)")
# 5. Truncate and chunk by tokens, not characters
def truncate_to_tokens(text, n, enc=o200k):
ids = enc.encode(text)
return text if len(ids) <= n else enc.decode(ids[:n])
def chunk_by_tokens(text, size, overlap=0, enc=o200k):
ids = enc.encode(text)
step = size - overlap
return [enc.decode(ids[i:i + size]) for i in range(0, max(len(ids) - overlap, 1), step)]
doc = " ".join([samples["English"], samples["Hindi (same meaning)"], samples["Python"], samples["JSON"]]) * 3
print(f"\ndoc: {len(doc)} chars, {len(o200k.encode(doc))} o200k tokens")
short = truncate_to_tokens(doc, 20)
print("truncate_to_tokens(doc, 20) ->", len(o200k.encode(short)), "tokens:", repr(short))
chunks = chunk_by_tokens(doc, 64, overlap=8)
print("chunk_by_tokens(doc, 64, overlap=8) ->", len(chunks), "chunks, token sizes:", [len(o200k.encode(c)) for c in chunks])
by_chars = [doc[i:i + 256] for i in range(0, len(doc), 256)]
print("same doc cut every 256 chars -> token sizes:", [len(o200k.encode(c)) for c in by_chars])
# 6. Budget the context window and estimate cost (formula only, EXAMPLE price)
def chat_tokens(messages, enc=o200k, per_message=3, reply_priming=3):
# Cookbook heuristic for gpt-4o-style chat: ~3 tokens/message + 3 for the reply. An estimate.
return sum(per_message + sum(len(enc.encode(v)) for v in m.values()) for m in messages) + reply_priming
messages = [
{"role": "system", "content": "You are a concise support assistant."},
{"role": "user", "content": samples["English"] + "\n\n" + doc},
]
raw = sum(len(o200k.encode(m["content"])) for m in messages)
est = chat_tokens(messages)
print(f"\nchat prompt: content tokens={raw}, with message overhead estimate={est} (+{est-raw})")
CONTEXT, MAX_OUTPUT = 8_000, 1_000 # example budget numbers for the demo
print(f"fits in a {CONTEXT}-token window with {MAX_OUTPUT} reserved for output:", est + MAX_OUTPUT <= CONTEXT)
EXAMPLE_PRICE_PER_MTOK = 1.00 # EXAMPLE ONLY, not a real quoted price
print(f"cost = tokens/1e6 * price -> {est}/1e6 * ${EXAMPLE_PRICE_PER_MTOK:.2f} = ${est/1e6*EXAMPLE_PRICE_PER_MTOK:.6f} per call (example price)")
# 7. Speed: encode a long text
long_text = (doc + "\n") * 2000
for name, enc in (("o200k_base", o200k), ("cl100k_base", cl100k)):
enc.encode(long_text[:1000]) # warm-up
times = []
for _ in range(5):
t = time.perf_counter(); n = len(enc.encode(long_text)); times.append(time.perf_counter() - t)
times.sort()
mb = len(long_text.encode()) / 1e6
print(f"encode {mb:.2f} MB ({len(long_text):,} chars) with {name}: {n:,} tokens, median {times[2]*1000:.0f} ms ({mb/times[2]:.1f} MB/s)")
lines = long_text.splitlines()
t = time.perf_counter(); o200k.encode_batch(lines, num_threads=8); print(f"encode_batch over {len(lines):,} lines (8 threads): {(time.perf_counter()-t)*1000:.0f} ms")
3. Real output
This is from my second run, with the encoding files already cached. The first run's differences are covered in the gotchas below.
tiktoken 0.14.0 | cache dir: /workspace/thursday-2026-11-19-pm/tiktoken-cache
cached files before loading: 2
load o200k_base: 199 ms, vocab size n_vocab=200019
load cl100k_base: 80 ms, vocab size n_vocab=100277
encoding_for_model():
gpt-4o -> o200k_base
gpt-4o-mini -> o200k_base
gpt-4.1 -> o200k_base
gpt-5 -> o200k_base
gpt-4 -> cl100k_base
gpt-3.5-turbo -> cl100k_base
text-embedding-3-small -> cl100k_base
claude-sonnet-4 -> KeyError: 'Could not automatically map claude-sonnet-4 to a tokeniser. Please us...
llama-3.1-8b -> KeyError: 'Could not automatically map llama-3.1-8b to a tokeniser. Please use `...
sample chars utf8 bytes o200k cl100k chars/tok (o200k)
English 60 60 10 10 6.00
Hindi (same meaning) 51 135 18 54 2.83
Python 67 67 27 27 2.48
JSON 79 79 31 30 2.55
round-trip ok: True
first 8 ids: [8256, 104833, 495, 2498, 16681, 306, 3407, 31926]
first 8 pieces: [b'Please', b' summarize', b' this', b' support', b' ticket', b' in', b' three', b' bullet']
Hindi first 6 pieces (bytes): [b'\xe0\xa4\x95', b'\xe0\xa5\x83', b'\xe0\xa4\xaa', b'\xe0\xa4\xaf\xe0\xa4\xbe', b' \xe0\xa4\x87\xe0\xa4\xb8', b' \xe0\xa4\xb8\xe0\xa4\xaa']
decode of a single Hindi token: 'ृ'
encode() with special token -> ValueError Encountered text corresponding to disallowed special token '...
encode_ordinary() -> 11 tokens (treated as plain text)
doc: 780 chars, 255 o200k tokens
truncate_to_tokens(doc, 20) -> 20 tokens: 'Please summarize this support ticket in three bullet points. कृपया इस सपोर्ट टिकट का तीन ब'
chunk_by_tokens(doc, 64, overlap=8) -> 5 chunks, token sizes: [64, 64, 64, 64, 31]
same doc cut every 256 chars -> token sizes: [84, 86, 85, 5]
chat prompt: content tokens=272, with message overhead estimate=283 (+11)
fits in a 8000-token window with 1000 reserved for output: True
cost = tokens/1e6 * price -> 283/1e6 * $1.00 = $0.000283 per call (example price)
encode 2.07 MB (1,562,000 chars) with o200k_base: 508,000 tokens, median 87 ms (23.7 MB/s)
encode 2.07 MB (1,562,000 chars) with cl100k_base: 726,000 tokens, median 153 ms (13.5 MB/s)
encode_batch over 26,000 lines (8 threads): 679 ms
How to read it:
-
Same meaning, very different counts. The English sentence is 10 tokens under both encodings. The Hindi translation is 18 tokens with
o200k_baseand 54 withcl100k_base, so the same request costs 1.8x the English count on a gpt-4o-class model and 5.4x on a cl100k model. Devanagari takes 3 UTF-8 bytes per character (135 bytes for 51 characters), and the older vocabulary has far fewer merges for it. - Code and JSON are dense. The Python snippet came out at 2.48 characters per token and the JSON at 2.55, against 6.00 for the English sentence. Indentation, brackets, quotes and short identifiers all cost tokens. If your prompt is mostly tool output or JSON, a chars/4 estimate will undercount.
-
Round-trip is exact.
decode(encode(text)) == textreturnedTrue. The token pieces show how BPE splits text. Common English words are single tokens with the leading space attached (b' summarize'). The Hindi tokens are much smaller, and one of them decoded to just the vowel sign'ृ'. -
Chunking by tokens gives predictable sizes.
chunk_by_tokens(doc, 64, overlap=8)gave chunks of exactly 64 tokens (plus a 31-token tail). Cutting the same text every 256 characters gave chunks of 84, 86 and 85 tokens. On text with more code or non-English content, that drift gets bigger. - Message overhead is small but real. The two message contents were 272 tokens. With the Cookbook's heuristic for gpt-4o-style chat (3 tokens per message, plus 3 to prime the reply) the estimate is 283.
-
It's fast. Encoding 2.07 MB in one call took a median 87 ms with
o200k_base(508,000 tokens) and 153 ms withcl100k_base(726,000 tokens). The same text took 30% fewer tokens with o200k because the text mixes in Hindi. Note that the long text is the same 780-character block repeated 2,000 times, so treat the speed as a rough figure.
How to budget a request
The check before every call is simple:
prompt_tokens (incl. message overhead) + max_output_tokens <= model context window
- Count the prompt with the encoding for your model (
encoding_for_model(model)). - Add a margin for chat formatting. The Cookbook says to treat its message formula as "an estimate, not a timeless guarantee." Tool/function definitions add extra tokens on top.
- Reserve room for the output (
max_output_tokens). The demo uses an example 8,000-token window with 1,000 tokens reserved for output. - If it doesn't fit, shrink the biggest variable part first (retrieved chunks, chat history) with
truncate_to_tokensor fewer chunks. Don't cut the system prompt.
Cost uses the same count:
cost = input_tokens / 1e6 * input_price_per_MTok + output_tokens / 1e6 * output_price_per_MTok
In the demo I used an example price of $1.00 per million input tokens. That is not a quoted price. Look up your model on OpenAI's pricing page and plug in the current numbers. With that example price, a 283-token prompt costs $0.000283. The useful part is the ratio: if Hindi users send 1.8x the tokens of English users for the same request, your per-user cost is 1.8x too.
Gotchas I hit while testing
-
The first call downloads the encoding file. With an empty cache, loading
o200k_basetook 6,877 ms andcl100k_base4,711 ms, because tiktoken fetched the BPE files over the network. Once cached, loading took 199 ms and 80 ms. By default the cache lives in your temp directory (<tmp>/data-gym-cache), which containers and serverless functions often wipe. SetTIKTOKEN_CACHE_DIRto a persistent path and pre-load the encodings when you build your image, or you'll pay the download on a user request. Offline or firewalled environments will fail on that first load. -
encoding_for_model()only knows OpenAI models.claude-sonnet-4andllama-3.1-8bboth raisedKeyError: Could not automatically map ... to a tokeniser. Other providers use their own tokenizers.o200k_basecounts for a Claude or Llama prompt are an approximation at best, so use that provider's token-counting method if you need exact numbers. In your own code, catch theKeyErrorand fall back deliberately. The Cookbook falls back too200k_basewith a warning. -
Chat overhead is approximate. tiktoken counts text, not API requests. The Cookbook shows its
num_tokens_from_messages()matching the API on its example (124 prompt tokens for gpt-4o and gpt-4o-mini, 129 for gpt-4), but warns that the formatting can change between models. To check your own setup, logusage.prompt_tokensfrom real responses and compare it with your estimate. - Non-English text and code inflate counts. See the Hindi row: 54 tokens under cl100k for a 10-token English sentence. If your users write in Indian languages, budget per language, not with a single chars-per-token constant. Picking the right encoding matters a lot here.
-
Special tokens raise by default. User text containing
<|endoftext|>madeencode()raiseValueError: Encountered text corresponding to disallowed special token. That is a safe default. For untrusted input, useencode_ordinary()(it treated the string as 11 plain tokens) instead of settingallowed_special="all". -
Truncation can cut mid-word.
truncate_to_tokens(doc, 20)ended on'... तीन ब', halfway through a Hindi word. The Cookbook also warns that decoding a single token can be lossy if it isn't on a UTF-8 boundary. For prompts, cut on token boundaries and then back off to the last sentence or newline if it matters. -
encode_batchisn't automatically faster. Splitting the 2 MB text into 26,000 short lines and runningencode_batch(..., num_threads=8)took 679 ms, against 87 ms for a singleencode()of the whole text. With many tiny strings, the per-call overhead dominates. Benchmark your real inputs.
Checklist before you ship it
- Pin
tiktokenand look up the encoding withencoding_for_model(model)instead of hard-coding it. - Set
TIKTOKEN_CACHE_DIRto a persistent path and pre-downloado200k_base(andcl100k_baseif you use embeddings) at build time. - Count prompt tokens plus message overhead plus
max_output_tokensbefore every call, and fail early or trim if it won't fit. - Chunk RAG documents by tokens (with overlap), not by characters.
- Use
encode_ordinary()for untrusted user text. - Use each provider's own tokenizer or token-count endpoint for non-OpenAI models.
- Log
usage.prompt_tokensand compare it with your estimate. Alert if the gap grows. - Compute cost from current official prices, and track tokens per request per language.
References
- tiktoken on GitHub (README, BPE explanation, MIT license): https://github.com/openai/tiktoken
- tiktoken changelog: https://github.com/openai/tiktoken/blob/main/CHANGELOG.md
- tiktoken on PyPI: https://pypi.org/project/tiktoken/
- OpenAI Cookbook, How to count tokens with tiktoken: https://github.com/openai/openai-cookbook/blob/main/examples/How_to_count_tokens_with_tiktoken.ipynb
- Model-to-encoding mapping in the source: https://github.com/openai/tiktoken/blob/main/tiktoken/model.py
- Cache directory logic (
TIKTOKEN_CACHE_DIR): https://github.com/openai/tiktoken/blob/main/tiktoken/load.py
Do you count tokens before the call, or find out from the error message?
Top comments (0)