What Is a Token? (And Why AI Thinks 'Strawberry' Has Two Rs)

Picture someone who learned to read at superhuman speed by scanning text in chunks of three or four letters at a time. They absorbed 'university' as 'uni-ver-si-ty' and 'basketball' as 'bas-ket-ball', blazing through novels while everyone else was still sounding out syllables. Incredibly efficient. One small problem: they never actually learned to look at individual letters. So when you ask them, "How many times does the letter 's' appear in 'Mississippi'?" they just blink at you. They genuinely have no idea.

That's exactly what's happening inside ChatGPT, Claude, and every other large language model when you ask them to count letters.

The Chunking Problem

These systems don't read text the way you do. Before any AI sees your sentence, it gets chopped into pieces called tokens. These are the actual units the model processes, usually chunks of a few characters that might be whole words ('the', 'cat') or fragments ('ing', 'tion', 'str').

This is why ChatGPT faceplants when you ask how many Rs are in 'strawberry'. It's not seeing the letters S-T-R-A-W-B-E-R-R-Y laid out in a row. It's scanning something like 'straw-berry' or 'str-aw-ber-ry' as pre-chunked fragments and trying to reconstruct letter counts from memory. Imagine someone who learned 'strawberry' as two bites and you're now asking them to tell you how many individual grains of sugar were sprinkled on top. They're guessing.

The model has no mechanism to examine individual characters because it never learned to see them in the first place.

Why Read This Way At All?

Because scanning chunks instead of individual letters lets you tear through exponentially more material. Our speed reader could finish three books in the time it takes you to finish one, and they'd remember the plots, themes, and character arcs perfectly.

For AI, tokens are an efficiency breakthrough. Instead of tracking millions of possible combinations of individual characters, the model works with a vocabulary of around 50,000 tokens. That compression makes it computationally feasible to process the entire internet's worth of text during training. Gmail's spam filter can scan thousands of emails in milliseconds because it's looking for chunk patterns like 'claim your prize now' and 'click here immediately', not analyzing every character in sequence.

This method absolutely excels at patterns, context, meaning, and flow. The AI grasps metaphors, maintains tone across paragraphs, and catches subtle implications. It understands that 'bank' means different things in 'river bank' versus 'savings bank' because it's learned how surrounding chunks predict meaning.

What gets sacrificed? Precise character-level awareness. You can't be great at chunk-pattern-recognition and simultaneous letter-counting. The architecture made a choice.

The Weird Chunks You Never Notice

Token boundaries aren't logical or consistent. The word 'the' is a single chunk because the model saw it millions of times during training. But 'Tchoupitoulas', a street in New Orleans, might get hacked into five separate pieces: 'Tch-oup-it-oul-as'.

This is based purely on frequency. Common words and letter combinations stay intact. Rare words, technical jargon, and unusual names get chopped mid-syllable into whatever fragments the algorithm recognizes from its training data.

This creates AI's odd strengths and weaknesses in ways you'd never predict. It breezes through 'restaurant' and 'necessary' (common words, single chunks) but stumbles over 'xylophone' or invented brand names (rare combinations, multiple chunks). Each chunk is a separate mental step, so more chunks means slower, less confident processing.

You can actually see this in action. Ask ChatGPT about 'phenomenon' and then ask about 'phenomonon' (a common misspelling). The correct spelling is probably one or two chunks. The misspelling gets shredded into more pieces, and the model has to work harder to figure out what you mean. Our speed reader just encountered a word with letters rearranged, and their chunk-scanning method suddenly fails them.

When The Chunks Betray You

This architecture creates a predictable set of failures. Letter counting fails. Spelling words backwards fails. Precise character manipulation fails. Anything requiring exact positional awareness of individual characters becomes a minefield.

Ask ChatGPT to reverse 'algorithm' letter by letter and watch it flail. But ask it to write a sonnet about algorithms and it nails the meter and rhyme scheme instantly. Poetry is about word-chunk patterns, stress, and sound relationships. Character gymnastics? Not so much.

These aren't bugs waiting to be fixed. This is a fundamental architectural choice. You can't ask our speed reader to suddenly switch to reading letter-by-letter on demand. They trained one way. Their brain wired itself around that method. Retrofitting a completely different skill isn't a simple patch.

The AI compensates with genuinely impressive strengths elsewhere. It'll catch that you used 'their' when you meant 'there' (wrong chunk for the context) even though both are correctly spelled. It understands sarcasm, metaphor, and implication because those emerge from how chunks relate to each other, not from individual character properties.

So What Can YOU Do With This?

Stop asking AI to count letters or manipulate individual characters. When you need to know how many times 'e' appears in a paragraph, use your word processor's find function or write three lines of Python.

Play to chunk strengths: pattern recognition, summarization, rewriting, idea generation, explaining concepts, catching logical inconsistencies. If you're using AI to help write code documentation, it'll excel at explaining what a function does but might stumble if you ask it to count how many times the variable name 'x' appears in your snippet.

Understand that your prompts cost money based on chunks, not words. 'The cat sat' is probably three chunks. 'The feline perched' might be four or five because 'feline' is less common. Longer or unusual words mean more chunks, which means higher API costs if you're using these tools at scale.

Be aware that rare technical jargon, made-up words, or unusual proper nouns will be choppier for the AI to process. If you're discussing 'Kubernetes' or 'Schwarzenegger', the model is working harder than if you're discussing 'computer' or 'Smith'. It'll still do fine, but you might notice slightly less confident or slower responses.

TL;DR

• AI reads text in multi-character chunks called tokens, not individual letters, which makes it blazingly fast at processing meaning but essentially blind to character-level details.
• Common words like 'the' are single chunks, while rare words and names get chopped into multiple pieces, affecting both processing speed and the model's confidence.
• This explains why AI can write elegant prose but fails at seemingly simple tasks like counting letters or spelling words backwards.
• Use AI for understanding patterns and meaning, not for precise character manipulation (use find-and-replace or simple code for that).

And if you ever catch an AI confidently telling you that 'strawberry' has two Rs, just remember: it never actually learned to spell.