Counting LLM Tokens in Python: tiktoken vs a Dependency-Free Heuristic

Every LLM feature I build eventually needs to answer one question: how many tokens is this? Context windows are measured in tokens, cost is measured in tokens, and truncation happens in tokens. Characters and words are close enough to lie to you and not close enough to trust.

There are two ways to count, and I have shipped both. In this tutorial I want to show you when each is the right tool, and then how to stop choosing between them by hiding both behind a single interface with a graceful fallback. I will ground it in a small project of mine called ctxlens, but the technique is what matters, not the project.

Option one: exact counts with tiktoken

tiktoken is the byte-pair-encoding tokenizer used by several OpenAI models. It gives you the real token count, the same number the model sees. If your job is billing, hard context-limit enforcement, or trimming a prompt to fit exactly, you want exact.

import tiktoken

enc = tiktoken.get_encoding("cl100k_base")
tokens = enc.encode("Counting tokens is deceptively subtle.")
print(len(tokens))  # exact token count

Two things to know. First, the encoding matters: cl100k_base matches one family of models, and other models use other vocabularies, so pick the one your target model actually uses. Second, encode can choke on text containing special tokens unless you tell it not to treat them specially. In production I always pass disallowed_special=() so arbitrary user text never raises:

def count(text: str) -> int:
    if not text:
        return 0
    return len(enc.encode(text, disallowed_special=()))

The cost of exactness is a dependency. tiktoken is a compiled package, it downloads vocabulary files, and it is not something you always want in a lightweight library, a CI runner with no network, or a constrained environment.

Option two: a dependency-free heuristic

Sometimes I do not need the exact number. When I am profiling which part of a prompt dominates the context, or watching how usage grows across a conversation, I care about relative size, not the precise integer. For that I use a pure-Python heuristic with zero dependencies and no network.

The idea: real BPE tokenizers roughly pack a handful of characters into each token and break long words into several pieces. So I split text into word, number, and symbol runs, and charge longer runs more than one token.

import re

# words (including unicode letters), numbers, or any single non-space symbol
_TOKEN_RE = re.compile(r"[^\W\d_]+|\d+|[^\s\w]", re.UNICODE)
_CHARS_PER_PIECE = 4


def heuristic_count(text: str) -> int:
    if not text:
        return 0
    total = 0
    for match in _TOKEN_RE.finditer(text):
        piece = match.group(0)
        n = len(piece)
        if n <= _CHARS_PER_PIECE:
            total += 1
        else:
            # long runs get broken into sub-tokens, rounding up
            total += (n + _CHARS_PER_PIECE - 1) // _CHARS_PER_PIECE
        # whitespace is folded into adjacent tokens, not counted
    return total

This runs anywhere Python runs. It is deterministic, so tests and CI produce stable numbers with no heavy install.

The honest caveat: this is an approximation, full stop. It does not reproduce any real BPE vocabulary, so its number will differ from the model's actual count, sometimes meaningfully, especially on code, non-English text, or unusual symbol-heavy input. Never bill a customer or enforce a hard context limit on a heuristic. Use it for relative comparison and offline profiling, and use tiktoken when the exact number carries consequences.

Stop choosing: one interface, graceful fallback

The real technique is not picking one counter. It is making the rest of your code not care which one it got. Define an abstract interface both implementations satisfy:

from abc import ABC, abstractmethod


class Tokenizer(ABC):
    name: str = "base"

    @abstractmethod
    def count(self, text: str) -> int:
        raise NotImplementedError

    def count_all(self, texts: list[str]) -> list[int]:
        return [self.count(t) for t in texts]

Both counters become subclasses. The heuristic one always works. The tiktoken one imports lazily inside __init__, so it only fails if you actually try to build it without the dependency installed:

class TiktokenTokenizer(Tokenizer):
    def __init__(self, encoding: str = "cl100k_base") -> None:
        try:
            import tiktoken
        except ImportError as exc:
            raise ImportError(
                "tiktoken is not installed; install it to get exact counts"
            ) from exc
        self._enc = tiktoken.get_encoding(encoding)
        self.name = f"tiktoken:{encoding}"

    def count(self, text: str) -> int:
        if not text:
            return 0
        return len(self._enc.encode(text, disallowed_special=()))

Notice the import lives inside the constructor, not at the top of the module. That is what makes the fallback clean: importing the module never requires the dependency, and the ImportError surfaces only at the exact moment a caller asks for something it cannot have.

Now a small factory ties it together. auto prefers exactness and silently degrades to the heuristic:

def _try_tiktoken(encoding: str = "cl100k_base"):
    try:
        return TiktokenTokenizer(encoding)
    except ImportError:
        return None


def get_tokenizer(name: str = "auto") -> Tokenizer:
    name = (name or "auto").lower()
    if name == "heuristic":
        return HeuristicTokenizer()
    if name == "tiktoken":
        tok = _try_tiktoken()
        if tok is None:
            raise ImportError("tiktoken requested but not installed")
        return tok
    if name == "auto":
        return _try_tiktoken() or HeuristicTokenizer()
    raise ValueError(f"unknown tokenizer: {name!r}")

The design choices here are deliberate. auto gives you the best answer available without a crash. heuristic guarantees the dependency-free path, which is what your tests should pin so they never depend on an optional install. tiktoken is explicit and loud: if you asked for exact and cannot have it, you get an error rather than a silent approximation. That last distinction matters, because silently swapping exact for approximate is exactly the bug that shows up as a surprise overage bill.

The payoff

The rest of your application never touches tiktoken or the regex again. It calls tokenizer.count(text). Swapping counters is a one-line change at the edge, tests run offline against the heuristic, and production gets exact counts wherever the dependency is present. You keep the name field around so reports can say which counter produced a number, because "1,240 tokens (heuristic)" and "1,240 tokens (tiktoken)" mean different things to the reader.

That is the whole pattern: an abstract count, a lazy optional dependency, and an auto mode that prefers exact and degrades gracefully. If you want to see it wired into a real tool that profiles where the tokens in your prompts actually go, this is lifted straight from my project ctxlens, and the full tokenizer package is at github.com/AgentPostmortem/ctxlens. Borrow the interface, keep the caveat, and count with confidence.