A customer asks if your assistant can read their 400 contracts, one page each. You say yes. The first request fails with a "too long" error, and after you send a smaller slice, the bill comes back ten times your quote. That is not a bug in your code. It is the two ideas most teams skip when they first build with language models: tokens, and the context window.
A model does not read words. It reads tokens, small pieces of text cut from a fixed list. A common word like "the" is one token, a rare word is built from several smaller pieces, numbers split into chunks of digits, and every punctuation mark counts. As a rule of thumb for English, one token is about four characters, so a 500-word page is roughly 700 tokens.
The cutting is not clever. Before training, builders count which groups of letters show up most often in a huge pile of text, and the common groups go on a list of pieces, usually tens of thousands of them. To cut your text, the tokenizer walks along it and takes the longest piece from the list that fits. A common method for building that list is byte pair encoding: start with single characters, merge the most frequent neighbouring pair into a new piece, and repeat thousands of times. The exact list differs for each model, so the same text has a different token count in different models.
Here is the part that quietly decides your costs. The piece list is built mostly from English prose, so plain English gets big, cheap pieces. Code, JSON, numbers, and many non-English scripts commonly cost two to five times the tokens for the same meaning, because the text falls back to smaller fragments, sometimes single characters. If your users write in other languages, or your documents are mostly code, budget for that. And a repeated word is charged again every time it appears. Nothing is skipped because it is a repeat.
The context window is a desk
The context window is the most tokens a model can hold for one request. Picture it as a small desk. Everything for one request has to fit on it at once: the system prompt, the chat so far, your documents, your question, and room for the answer. Too much, and the request fails with a length error.
Why is there a limit at all? The model compares every token with the others as it reads. More tokens means more memory and more work, so builders choose a size they can run. Bigger windows exist, but they cost more to use.
Where the picture breaks down: a real desk keeps your papers overnight. The model keeps nothing between requests, so your app must fill the desk again every time. Nobody hands you a bill that says which sheet used how much, so builders track each category on its own.
Think of a 32,000-token desk. A typical split might be 1,500 tokens of system prompt, 6,000 of chat history, 12,000 of retrieved documents, 300 for the question, and 2,000 reserved for the answer: 21,800 in total, which fits with 10,200 to spare. But push the documents to 25,000 and the bar goes over budget. Your question is only about 1 percent of the desk. Documents and history fill it fast, so cut the biggest part first.
Why long chats cost more
Now the surprise bill. A chat sends its whole history again with each new message, so message 50 costs about 19 times message 1. Work it out with made-up but realistic numbers. The system prompt is 500 tokens, and each turn adds 200: a 40-token message plus a 160-token reply. Message 1 sends 500 plus 40, or 540 tokens. Before message 50 there are 49 old turns: 49 times 200 is 9,800 tokens. Message 50 sends 500 plus 9,800 plus 40, or 10,340 tokens, and 10,340 divided by 540 is about 19. The customer types about the same amount each time. The bar grows because the history is sent again.
Add up every request in that 50-message chat and the customer sends about 272,000 input tokens in total. At GPT-4o prices as of November 2026, $2.50 per million input tokens, that is about $0.68 of input for one long chat. The naive estimate, 50 times the first message, gives about $0.07. The gap between those two numbers is the surprise bill. On the cheaper GPT-4o mini, $0.15 per million input tokens as of November 2026, the same chat costs only a few cents, but the 19-times gap is exactly the same. Replies often cost more per token than input, so check the price list of the model you actually use.
The 400 contracts
Back to the customer with 400 contracts. Each contract is one page, about 700 tokens. 400 times 700 is 280,000 tokens. The window is 32,000 tokens: 280,000 divided by 32,000 is 8.75, so about nine desks. It can never fit in one request.
The fix is retrieval: search first, then paste only the three best paragraphs. A bigger window would not fix it either, because cost and wait grow with it. Pasting everything in "just in case" puts more sheets on the desk than the question needs. You pay per token on every request, so a big history is paid for again each turn, and more input means more work before the first word comes back. Search for only the relevant passages, and replace old history with a short summary.
That gives a simple rule for trimming. Do not wait for the "too long" error. When the history passes about half the window, replace the old turns with a short summary and keep only the recent turns word for word. When the summary itself grows long, summarize the summary.
The full interactive chapter, with a tokenizer you can try and a context window you can spend yourself, is free at https://deployedlab.dev/lessons/07-tokens-and-context. 11 chapters are free with no signup, and the full course is $5/month or $39/year at deployedlab.dev.
Top comments (0)