DEV Community

iLostCount
iLostCount

Posted on

How to count the tokens in an LLM prompt (and why the number matters)

#ai #beginners #webdev #llm

If a prompt gets rejected as too long, or an API bill looks bigger than expected, the number that matters is tokens, not words. Here is the short version of how to check one.

What a token is

Models do not read characters or words. They read tokens: chunks of text that a tokenizer splits your input into. For ordinary English prose, a useful rule of thumb is:

  • 1 token is roughly 4 characters
  • 1 token is roughly 0.75 words
  • 1,000 words is roughly 1,300 tokens

Those ratios shift. Code, JSON and long URLs tokenize worse than prose, because punctuation and odd substrings break into more pieces. Non-Latin scripts such as Arabic, Chinese and Japanese can cost several tokens per character in some tokenizers. Rules of thumb are fine for estimating and useless for a hard limit.

Counting in code

For OpenAI models, tiktoken gives an exact count:

import tiktoken

enc = tiktoken.get_encoding("o200k_base")
print(len(enc.encode("your prompt here")))
Enter fullscreen mode Exit fullscreen mode

Anthropic and Google both expose token-counting endpoints in their APIs, so you can measure a prompt against the exact model you are calling instead of an approximation.

Counting without writing code

Most of the time you are not writing a script. You pasted something into a chat window and you want to know whether it fits. That is what we built iLostCount for: paste text, read the token, word and character counts as you type. No signup, and nothing is uploaded, since it runs in the page.

Why the number matters

  • Context window. The window has to hold your system prompt, the conversation so far, any retrieved documents, and the answer. If the input fills the window, there is no room left for the output.
  • Cost. Input and output are both billed per token, so a prompt that re-sends a large document on every turn adds up quietly.
  • Truncation. Some tools silently drop the oldest turns when you run over. That looks like the model forgetting, not like an error.

A cheap habit: count a big document before you paste it into a prompt. If a 40-page PDF turns into 30,000 tokens, you know to chunk or summarise it first.

Disclosure: this post is from the iLostCount project. The tool is free, and the source is public at github.com/ahmad-almazeedi/token-counter.

Top comments (0)