Why LLMs Can't Count the R's in Strawberry: A Look Inside BPE Tokenization

A model never sees letters. Every prompt gets chopped into tokens before a single parameter runs, and a surprising share of "the model can't do basic things" bug reports trace straight back to where those cuts land. I built an actual tokenizer against OpenAI's published vocabularies to measure this instead of guessing, and wrote up the full run in more detail on DevToolLab - here's the short version.

The classic example: ask a model to count the r's in "strawberry" and it often gets it wrong. Run through OpenAI's cl100k_base vocabulary, "strawberry" isn't a word to the model at all, it's three pieces: str, aw, berry. It was never shown ten letters.

What a Token Actually Is

A token is a byte sequence common enough in training data to earn its own slot in a fixed vocabulary. That vocabulary comes from byte pair encoding: start from raw bytes, then keep merging whichever adjacent pair shows up most often until the table is full.

Two things fall out of that. First, tokens are byte sequences, not characters, so one token can end in the middle of a multi-byte character. Second, the vocabulary is frozen the moment training ends, so anything that resembles the training distribution compresses tightly and anything that doesn't gets shredded into fragments.

A Tokenizer in About 60 Lines

OpenAI publishes the raw vocabulary files, so you don't need a library to see this happen, just a fetch call and a merge loop:

// minimal BPE against OpenAI's public vocab files - Node 18+, zero deps
const VOCAB_URLS = {
  cl100k_base: "https://openaipublic.blob.core.windows.net/encodings/cl100k_base.tiktoken",
  o200k_base: "https://openaipublic.blob.core.windows.net/encodings/o200k_base.tiktoken",
}

async function loadVocab(name) {
  const raw = await fetch(VOCAB_URLS[name]).then((r) => r.text())
  const rankOf = new Map()
  for (const line of raw.split("\n")) {
    if (!line) continue
    const [token, rank] = line.split(" ")
    rankOf.set(Buffer.from(token, "base64").toString("latin1"), Number(rank))
  }
  return rankOf
}

// Repeatedly fuse the lowest-rank adjacent pair until nothing more can merge.
function mergeToTokens(word, rankOf) {
  let pieces = [...word]
  while (pieces.length > 1) {
    let lowestRank = Infinity, mergeAt = -1
    for (let i = 0; i < pieces.length - 1; i++) {
      const r = rankOf.get(pieces[i] + pieces[i + 1])
      if (r !== undefined && r < lowestRank) { lowestRank = r; mergeAt = i }
    }
    if (mergeAt === -1) break
    pieces.splice(mergeAt, 2, pieces[mergeAt] + pieces[mergeAt + 1])
  }
  return pieces
}

Feed it real samples (English, Hindi, a UUID, some JSON) and diff the token counts across cl100k_base and the newer o200k_base, and the reference tiktoken library agrees with it on "hello world" (2 tokens: hello, world), which is the sanity check that matters.

What the Splits Actually Explain

A few results jumped out when I ran the samples:

• strawberry costs 3 tokens (str|aw|berry) but " strawberry" with a leading space costs 1. The model is conditioned on whitespace as part of the token, which is why trailing spaces in a prompt genuinely change the output.
• Numbers split by frequency, not by place value. 2026 becomes 20|26, 1234567 becomes 123|45|67. A model doing arithmetic is working with fragments that don't line up column by column.
• A UUID alone costs 22 tokens under cl100k_base. Paste a hundred of them into a prompt and you've burned over 2,000 tokens on identifiers nobody reads.
• Non-English text pays a real tax. The identical sentence measured 10 tokens in English versus 50 in Hindi under cl100k_base, a 5x cost difference for the same meaning. o200k_base narrows that gap (Hindi drops to 21 tokens) but doesn't close it, and since APIs bill per token this is a direct price difference by language, not a rounding error.

I go through the JSON and Japanese numbers too, plus why streaming APIs occasionally emit a broken character mid-word, in the full writeup.

What to Actually Do About It

• Never estimate non-English cost from a character count. The gap is too large for a rule of thumb.
• Keep raw UUIDs and other identifiers out of prompts you pay for; map them to short indexes instead.
• Don't ask a model to count letters or reverse a string. It isn't looking at characters, so that's a job for code.
• Never end a prompt with a trailing space, it silently changes how the next tokens get cut.
• Re-measure after switching models. A newer vocabulary isn't uniformly cheaper, one JSON sample in my run actually cost more tokens under o200k_base than under cl100k_base.

Conclusion

Tokenization is the layer where a model's input stops being text, and most of what looks like a reasoning failure is really an artifact of that boundary. The vocabularies are public and the algorithm fits in 60 lines, so it costs almost nothing to check your own prompts instead of assuming.

References

• Original article on DevToolLab
• openai/tiktoken on GitHub
• AI Token Counter - count tokens for a prompt across model families without wiring up a tokenizer library yourself.
• LLM Token Cost Calculator - turn a token count into a per-request price before a multilingual feature ships.