DEV Community

ThatToolSite
ThatToolSite

Posted on

Count Your Tokens Before You Paste: Practical LLM Token Math

#ai #llm #openai #beginners

You paste a long doc into ChatGPT, Claude or Gemini and one of three things happens: the answer ignores the end of it, you hit a context limit, or the API bill is bigger than you expected. All three come down to the same number: tokens.

Here's the practical version of token maths: what a token is, quick estimates you can do in your head, how to count exactly in a few lines of code, and how to work out what a prompt will cost.

What a token actually is

Models don't read characters or words. A tokenizer splits text into chunks from a fixed vocabulary, and the model sees a list of chunk IDs.

  • Common English words are usually one token: the, house, token.
  • Longer or rarer words get split: tokenization → token + ization.
  • Spaces and punctuation count too. A leading space is often part of the next token (world).
  • Numbers, code, URLs and non-Latin scripts use more tokens per character than plain English.

That last point is the one that surprises people. The same sentence in Hindi, Japanese or Arabic can take two or three times the tokens of its English version, and minified JSON or a stack trace is far denser than prose.

Rules of thumb (good enough for a quick check)

For English prose with OpenAI's current tokenizers:

  • 1 token ≈ 4 characters
  • 1 token ≈ ¾ of a word, so 100 words ≈ 130 tokens
  • A dense A4 page of text ≈ 600–800 tokens

Other providers publish their own rough ratios, and they differ a little because each model family has its own tokenizer. Treat these as estimates; for anything that matters, count.

Count exactly in code

OpenAI's tokenizers are open source. In JavaScript, js-tiktoken runs anywhere, including the browser:

import { Tiktoken } from "js-tiktoken/lite";
import o200k_base from "js-tiktoken/ranks/o200k_base";

const enc = new Tiktoken(o200k_base);
const text = "Count your tokens before you paste.";
console.log(enc.encode(text).length); // number of tokens
Enter fullscreen mode Exit fullscreen mode

o200k_base is the encoding used by GPT-4o and newer OpenAI models; older ones like GPT-4 and GPT-3.5 use cl100k_base. In Python it's the tiktoken package:

import tiktoken

enc = tiktoken.get_encoding("o200k_base")
print(len(enc.encode("Count your tokens before you paste.")))
Enter fullscreen mode Exit fullscreen mode

For Claude and Gemini the tokenizers aren't published, but both APIs have a token-counting endpoint (Anthropic's count_tokens for messages, Gemini's countTokens) if you need an exact number before sending a request.

What a prompt will cost

Every API price list is per million tokens, and input and output are priced separately:

cost = input_tokens  / 1,000,000 × input_price
     + output_tokens / 1,000,000 × output_price
Enter fullscreen mode Exit fullscreen mode

Worked example with made-up round prices, so the maths is easy to follow: say input costs $2 per million and output $8 per million. A 3,000-token prompt with a 500-token answer costs

3,000 / 1,000,000 × $2 = $0.006
  500 / 1,000,000 × $8 = $0.004
                 total = $0.010 per request
Enter fullscreen mode Exit fullscreen mode

One cent sounds like nothing until it runs 50,000 times a day. Two things people forget:

  1. Chat history is input every time. In a long conversation, every new message resends the whole thread, so input tokens grow with each turn.
  2. Output is usually the expensive side. Asking for "brief" answers or a fixed format often saves more than trimming the prompt.

Five ways to cut tokens without losing quality

  1. Strip HTML and boilerplate before you paste a web page. Navigation, footers and tags are pure token waste.
  2. Minify or trim JSON: drop fields the model doesn't need instead of sending the whole API response.
  3. Summarise old context. Replace the first 20 messages of a long thread with a five-line summary.
  4. Don't repeat instructions in every message; put them once in the system prompt.
  5. Check non-English and code-heavy inputs, because they're where token counts surprise you most.

A quick way to check without writing code

If you just want to paste something and see the count, I made a free token counter that runs entirely in your browser, so nothing you paste is uploaded. It gives exact counts for OpenAI's tokenizers, clearly labelled estimates for Claude and Gemini, and a cost calculator with a dated, editable price table.

What's the biggest token surprise you've run into: a huge chat history, a non-English document, or a JSON blob that ate the whole context window?

Top comments (0)