You costed the support assistant
on a spreadsheet.
Average conversation,
about fifteen hundred tokens.
Price per million.
Times the expected chats.
It came in under budget,
and in English, it stayed there.
Then you launched it
in India,
in Greece
and in Ethiopia.
Same questions.
Same answers.
Same length on the screen.
The bill for those markets
came in at three to five times
the English rate
per conversation.
And it got worse in ways
nobody had put on the spreadsheet.
The assistant kept
the last four thousand tokens
of history.
In English, that was a dozen turns.
In Amharic,
it was barely three,
so it forgot the order number
the customer had typed
two minutes earlier.
Answers took longer to arrive.
Your rate limit,
counted in tokens,
started refusing Greek customers
at lunchtime.
Here is the mechanism.
A model does not read words.
It reads tokens,
pieces from a fixed vocabulary
that was built
from its training text.
That text is mostly English.
So common English words
earned a token each.
A word in a script
the vocabulary saw less of
gets cut into many pieces,
sometimes into single bytes.
Researchers measured it.
The same sentence,
translated,
can cost several times the tokens,
and in some languages
more than ten.
Newer tokenizers
have narrowed the gap.
They have not closed it.
Nothing in the product shows it.
A character limit on the input box
looks fair to everyone.
The cost underneath it
is not.
So measure in the unit you pay in.
Take real conversations
in every language you launch,
run them through the model's own tokenizer,
and price each market separately.
Size the history window
in turns,
not a fixed number of tokens,
or set the budget per language.
Truncate by tokens
at the edge,
and watch the per-language spend
from the first day.
A token is not a word.
How many tokens a word costs
depends on whose language it is.
– Serguey Asael Shinder
Top comments (0)