I Added a GPT Token Counter, and Learned Tiktoken's Own npm Package Ships the Vocab Files Twice

I use OpenAI's API often enough that I'm regularly guessing at token counts before I hit send — "is this prompt going to blow the context window" is a dumb thing to find out from an error response. So I added a GPT Token Counter to Bracketly, my free client-side dev tools site, and it turned into a smaller, more interesting problem than I expected.

The tokenizer, not an approximation

The obvious wrong answer is "estimate ~4 characters per token." The obvious right answer is js-tiktoken, OpenAI's own pure-JS port of tiktoken — same byte-pair-encoding vocabulary the API actually bills against, not a guess. GPT-5.5, GPT-5.4, and GPT-4o all still tokenize with the o200k_base encoding (confirmed against OpenAI's own tiktoken repo — nothing newer has shipped a different one yet), while GPT-4 and GPT-3.5 Turbo use the older cl100k_base. Two vocab files cover essentially every model anyone's actually calling right now.

The part that made me look twice

Those vocab files aren't small — o200k_base alone is about 2.3MB unminified, because it's a literal table of ~200,000 byte-pair merge ranks, not code. Bundling that into every page on the site would tax everyone visiting a JSON formatter to benefit nobody. So the tool imports js-tiktoken/lite (just the encoder logic, no vocab) and dynamically import()s only the one rank file the selected model actually needs — Vite splits each into its own chunk automatically, so it only downloads the first time you type something, cached from then on.

I didn't trust "it compiled" as proof it worked. I ran the actual built-and-bundled tool in a real headless browser against known tiktoken outputs — "Hello, world!" is 4 tokens in both encodings, "The quick brown fox jumps over the lazy dog." is 10 — and only shipped it once the live page matched exactly.

What it doesn't do

It doesn't cover Claude or Gemini — their tokenizers aren't tiktoken-compatible, and I'd rather leave a model out than fake a number for it. It also doesn't add the small fixed per-message overhead a chat-format API request tacks on for role/name framing — it counts raw text tokens only, which is the number you actually want when you're sizing a prompt before you've decided how to structure the request.

Free, no signup, runs entirely in your browser: bracketly.pages.dev/tools/token-counter.