Think LLMs "read" like we do? Think again. Here is why your typos (and your code formatting) are costing you real money.
So, there I was, scrolling through Instagram late at night—probably when I should’ve been sleeping—and I saw a weird trivia post.
It asked: "Hello world" is 2 tokens, but "helloworld" is more than 2. Why?
My brain went into "problem-solving mode." I thought, Okay, "Hello world" is just two common words. But "helloworld" isn't a real word, so the AI has to chop it up into smaller pieces. It sounded like a good guess, but "good guesses" aren't enough for me. I wanted to see the actual math.
I jumped onto my computer, opened Cursor, and built a quick tool using gradio and some common AI "tokenizers" (the stuff that chops up words). I wanted to see exactly where the "cuts" happen and—most importantly—how much they cost.
Here is what I found.
Level 1: The "Lego" Rule (Spaces and Caps)
First thing I learned? AI models are obsessed with patterns. If you break the pattern, you pay for it.
For us, "apple" and "aPpLe" mean the same thing. But to an AI? One is a common fruit it knows well. The other is a weird string of letters it has to piece together like a jigsaw puzzle.
Just by changing the capital letters, I tripled the "work" the AI had to do. It’s like trying to read a book where every third letter is capitalized—you can do it, but it’s way slower and more "expensive" for the brain.
Level 2: The "Typo Tax"
This is the part that actually surprised me. I tested a normal word like "environment" against my favorite typo, "envinorment." I always knew typos made me look a bit messy, but I didn't realize they were actually making my AI bill higher.
The word "environment" is so common that the AI sees it as one single unit. But as soon as I swapped two letters, the AI panicked. It couldn't find the whole word in its dictionary, so it had to use four different "bricks" to build it.
The result? A 400% jump in token usage for the exact same meaning. If you’re building an AI app and your users have bad spelling, you’re literally burning money on typos.
Level 3: Shortcuts That Backfire
I also tested how we talk in real life. We use "btw" instead of "by the way" to save time. But does it save money?
- "By the way" = 3 tokens.
- "btw" = 1 token.
Cool, slang works there! But then look at "knowledge" (1 token) vs. "knwldg" (4 tokens). Even though "knwldg" is shorter for us to type, it’s "noisier" for the AI because it’s not a common pattern. It ends up costing more!
The simple rule of thumb: 1 token is usually about 4 letters of normal English. But as soon as you add weird symbols, extra spaces, or code, that rule breaks.
Level 4: The "Senior" Reality Check (Code, Emojis, and Unicode)
As an engineer & backend dev, this is where things get really interesting. If you think a 400% jump is bad, wait until you see what happens when we step outside of standard English or start piping JSON data.
1. The JSON/Code Overhead
We love clean, readable code. But "pretty" JSON is an AI budget killer. Look at the difference:
-
{"key":"value"}— 5 tokens -
{ "key" : "value" }— 9 tokens
By simply adding spaces inside those brackets for "readability," I doubled the cost of the payload. When you're sending thousands of API calls, those spaces aren't just whitespace; they're line items on your invoice.
2. The Unicode Trap (Telugu vs. English)
This is where the bias of modern AI really shows. Most tokenizers are based on UTF-8 but are heavily trained on Latin scripts.
- English: "Hello" — 1 token
- Telugu: "నమస్కారం" — ~8 tokens
Because a single Telugu character often requires multiple bytes to represent in Unicode, the tokenizer has to "sub-divide" the character multiple times. For Indian developers, this "token bloat" means building apps for local languages can be 6-10x more expensive than building for English.
3. The Emoji "Combo"
Think an emoji is just one character? Think again.
- "😀" — 1 token
- "🏳️🌈" — 4 tokens
The Pride flag isn't a single "brick." It’s a Zero Width Joiner (ZWJ) combo—it’s actually a Rainbow emoji + a special invisible character + a Flag emoji. The AI has to process the entire sequence to understand it's one symbol.
It’s fascinating to see how complex emojis "mutate" from simple ones. If you want to see the "DNA" of an emoji for yourself, run this quick Python script. It’s a fun way to see exactly how many hidden characters are hiding inside a single icon:
s = "🏳️🌈"
for c in s:
print(c, hex(ord(c)))
When you run this, you'll see the Rainbow, the variation selector, the joiner, and the flag all listed out separately. To the AI, that's not one "vibe"—that's a whole sentence of data!
So, what did I learn?
Building this tool showed me that AI doesn't "read" words like we do. It looks for the easiest way to chop things up.
If you want to save money and get better results:
- Clean your text. A simple spell-check before you send text to an AI can save you 400%.
- Minify your JSON. If the AI is the only one reading the data, remove the spaces.
- Be mindful of Unicode. If you're building for Indic languages, factor in the "token tax" during your budget planning.
I’m going to keep testing my tool to see what else I can break.
If you found this breakdown useful, feel free to stalk my profile for more deep dives into the weird world of AI engineering.
For the full picture of what I’m building, check out my portfolio at bikkina.vercel.app. Catch you in the next one! 🚀
Top comments (0)