LLM Tokens, Measured in C#: Why AI Can't Count the R's in Strawberry

In August 2024, TechCrunch asked a simple question: how many R's are in "strawberry"? According to GPT-4o and Claude at the time, two. The answer is three.

The model isn't bad at counting. It never saw the letters. It saw tokens, and every token is just a number.

Every token count in this article was measured by me on October 30, 2026, with Microsoft's own .NET library, Microsoft.ML.Tokenizers 2.0.1, using the tokenizer for gpt-5 (the same o200k_base encoding GPT-4o uses). The program is at the end, so you can check them.

What the model actually gets

Before your prompt reaches the model, a tokenizer cuts it into pieces, called tokens, and turns each piece into an ID from a fixed list.

How many r's are in strawberry? becomes 8 tokens:

Token	How	many	r	's	are	in	strawberry	?
ID	5299	1991	428	885	553	306	101830	30

Look at strawberry, with the space in front. It's one token: number 101830. The R's are inside that number, and the model can't open it.

Spell the word out with spaces, s t r a w b e r r y, and it becomes 10 tokens. Three of them are r (ID 428). Now there's something to count.

Where tokens come from: byte pair encoding

Before the model is trained, the tokenizer learns from a big pile of text:

1. Start with single bytes.
2. Find the pair of neighbors that shows up most often, and glue it into a new token.
3. Repeat, until the list reaches a fixed size. For GPT-4o's encoding, o200k, that's about 200,000 tokens.

This is byte pair encoding. It started as a compression algorithm, and was adapted for words in Sennrich, Haddow and Birch (2016). OpenAI's tiktoken README lists what you get from it: it's lossless, it works on any text, and on average one token is about 4 bytes.

The result: common text becomes one token, rare text becomes pieces.

• tokenization → token · ization
• Strawberry at the start of a line → Str · aw · berry (3 tokens)
• strawberry in the middle of a sentence → 1 token

Same word, different splits, depending on a capital letter and a space.

The same sentence costs more in other languages

I'm Romanian, so I measured this one first:

• English: "Your order has shipped. You can track it in your account, and it should arrive within three business days." → 22 tokens
• Romanian: "Comanda ta a fost expediată. O poți urmări în contul tău și ar trebui să ajungă în trei zile lucrătoare." → 34 tokens

That's about 55% more for the same meaning. Common English words got their own tokens. Romanian words get cut up: urm·ări, t·ă·u, lucr·ăto·are. The same answer costs more and fills the context window faster.

Numbers get cut too

• 2026 → 202 · 6
• 1234567 → 123 · 456 · 7 (groups of three, from the left)

A 2024 paper, Tokenization counts (Singh and Strouse), notes that GPT-3.5 and GPT-4 have separate tokens for 1-, 2- and 3-digit numbers, and that forcing the digits to group from the right, just by adding commas, made the models much better at addition.

Your code, and the expensive part

This C# method is 234 characters and 59 tokens:

public async Task<List<Order>> GetRecentOrdersAsync(int customerId)
{
    return await _db.Orders
        .Where(o => o.CustomerId == customerId)
        .OrderByDescending(o => o.CreatedAt)
        .Take(10)
        .ToListAsync();
}

Indentation is cheap: a run of spaces is usually one token. Common identifiers like Where and Async are cheap too.

Now a GUID: 3f2b8c1e-9d4a-4e7b-a1c2-5d6e7f809a1b. 36 characters, 33 tokens. Almost one token per character, because random hex has no common pieces to glue together. A log full of GUIDs, hashes or base64 eats your context window much faster than the same length of prose.

Why this matters

• Price is per token, input and output priced separately (Claude pricing).
• The context window is measured in tokens.
• Rate limits are in tokens per minute (Claude rate limits).

Characters and words don't matter. Tokens do.

Every model has its own tokenizer

ToListAsync is one token for GPT-4's tokenizer (cl100k_base) and three for GPT-4o's (To · List · Async).

It changes inside one family too. Anthropic's token counting docs say Claude models from Opus 4.7 on use a newer tokenizer, and the same text comes out about 30% higher than on earlier models. A count from one model is a guess for another. Count with the model you actually call.

Counting tokens in .NET

dotnet add package Microsoft.ML.Tokenizers
dotnet add package Microsoft.ML.Tokenizers.Data.O200kBase

using Microsoft.ML.Tokenizers;

// Create once, reuse it.
Tokenizer tok = TiktokenTokenizer.CreateForModel("gpt-5");

int count = tok.CountTokens(prompt);

It can also cut text to a budget, without guessing with character counts:

int end = tok.GetIndexByTokenCount(
    text, 4000, out string? normalized, out _);
string chunk = (normalized ?? text)[..end];

The library covers Tiktoken (the GPT encodings), Llama, BPE and a few others. For Claude, ask the API: POST /v1/messages/count_tokens counts with the tokenizer of the model you pass. It's free (with rate limits), and the docs call it an estimate that can differ from the real usage by a small amount.

Back to strawberry

Newer models often get the strawberry question right. And you can help any model: ask it to spell the word letter by letter, and each letter becomes its own token. But the tokens didn't go away. Under the hood, the model still reads IDs, not letters.

The measurement program

// dotnet add package Microsoft.ML.Tokenizers
// dotnet add package Microsoft.ML.Tokenizers.Data.O200kBase
using Microsoft.ML.Tokenizers;

var tok = TiktokenTokenizer.CreateForModel("gpt-5");
foreach (var t in tok.EncodeToTokens(" strawberry", out _))
    Console.WriteLine($"[{t.Value}] #{t.Id}");   // [ strawberry] #101830

Console.WriteLine(tok.CountTokens("3f2b8c1e-9d4a-4e7b-a1c2-5d6e7f809a1b")); // 33

What's the most expensive thing you paste into a prompt? Logs, JSON, a whole file? Tell me in the comments.

---

I make Vlad's Stack — short explainers about how the tools you use every day actually work, for people who write code (.NET-flavored). The video version of this article: https://www.youtube.com/@VladsStack