What counts as tokens, why context length matters, and how to choose models for your workload.
A context window is the maximum amount of text an LLM can process in a single request, measured in tokens. It is the hard boundary between what a model can see and what it cannot. For a developer or product manager sizing workloads, the context window determines what fits in a single API call, how much history a chatbot can retain, and ultimately, whether a $0.01 problem becomes a $1.00 problem. Understanding context windows means understanding tokens, why limits exist, what trade-offs long-context models introduce, and when they are worth the cost.
Why this matters now
In 2026, context windows are no longer a constraint but a design choice. Claude 3.5 Sonnet ships with a 200k token window. GPT-4o supports 128k. Open-source models like LLaMA 3.1 offer windows up to 128k. Ten years ago, a 2k context window was standard. Five years ago, 4k felt luxurious. Today, "long-context" has become table stakes in the model market. Yet abundance has created a new problem: developers now struggle to decide whether they need it.
This shift matters because context window size directly affects cost, latency, and application architecture. A retrieval-augmented generation (RAG) system using semantic search is fundamentally different from one that stuffs 100k tokens of raw documents into a prompt. The choice between them is not purely technical; it is economic and architectural. It shapes whether you build a simple, stateless API or a complex system with database lookups, ranking pipelines, and state management. Getting this decision wrong wastes engineering time and money.
What tokens actually are
Photo by Leeloo The First on Pexels.
A token is the unit of text that an LLM processes. It is not a word. It is not a character. It is a subword unit chosen by the model's training process.
Most modern LLMs use byte-pair encoding (BPE) or similar subword tokenization schemes. Here is what that means in practice: the English sentence "Hello, world!" tokenizes into roughly 3 to 4 tokens, not 2. The comma might be its own token. The space before "world" might be bundled with "world" as a single token. A word like "international" might split into "inter" and "national", or "international" might stay whole, depending on how frequently it appeared in the training corpus. Common words and punctuation tend to be single tokens. Uncommon words, numbers, and code often split into multiple tokens.
Why this design choice? Tokenization is a trade-off between vocabulary size and sequence length. If an LLM had a vocabulary of single characters, a 1000-word document would require 5000 or more tokens. If the vocabulary included every possible word, it would balloon to millions of entries, making the model harder to train. Subword tokenization finds a middle ground: a vocabulary of around 50k to 100k tokens can represent almost any text, keeping sequence length manageable while maintaining coverage.
For your purposes, remember this rule of thumb: one token is roughly 4 characters, or about 0.75 English words. That estimate holds for English prose. For code, numbers, or non-English text, the ratio is worse (fewer characters per token). Always validate with the actual tokenizer for your model before you submit a workload.
How context windows constrain what you can do
A context window is a hard limit. If your input (prompt plus all prior conversation history, documents, examples) exceeds the window, the model either refuses the request or you must truncate content. There is no sneaking around it.
Consider a practical example: a developer building a multi-turn chatbot on GPT-4o (128k context window) might retain all conversation history. With a 4k window model, they would need to summarize old messages or discard them. For a document processing task, a 32k window means you can fit roughly 8,000 words at a time. A 200k window can fit a whole book, or eight books, or a few books plus additional instructions and examples.
Context length also affects latency. Processing a 100k token prompt takes noticeably longer than processing a 10k token prompt on the same hardware. The time scales roughly with the square of the context size due to the self-attention mechanism in transformer architectures. In production, a system that routinely hits the top 80% of available context will feel slower and tie up GPU resources longer than one that stays under 50%.
Finally, context windows interact with pricing. Most LLM APIs charge per token, with separate rates for input and output tokens. A longer context window means more input tokens billed per request. At OpenAI's GPT-4o pricing (roughly $3 per million input tokens as of early 2026), a 100k token input costs about $0.30. The same content at a 10k context would cost $0.03. This compounds across millions of requests.
Long-context models trade throughput and latency for flexibility
Photo by Plato Terentev on Pexels.
The appeal of a 200k context window is obvious: fit more into one request, simplify orchestration, reduce state management. The hidden cost is less obvious.
First, latency increases. A 200k token prompt takes 4 to 10 times longer to process than a 10k prompt on the same model family. This is not a small constant overhead; it scales with the complexity of attention computation. If your application needs sub-second response times, a long-context model may not be an option, even if the model is otherwise superior.
Second, throughput per dollar falls. A GPU batch-processing shorter requests completes more requests per hour than one processing long-context requests. If you run inference on your own hardware, long-context requests reduce your effective throughput and increase your cost per completed task. With cloud APIs, you pay directly for tokens, so the math is more transparent, but the economics are identical.
Third, long-context models often exhibit the "lost in the middle" problem. Researchers at Stanford and MIT have documented that many LLMs perform worse on information buried in the middle of a long context window. When asked to retrieve a fact from position 50k in an input, these models sometimes perform worse than when the fact is at position 5k or 95k. This is not universal; newer models like Claude 3.5 mitigate it significantly. But it remains a real consideration. If you plan to throw raw data at a long-context model and expect it to find needles in a haystack, test it first.
Fourth, long-context capabilities may come with trade-offs in other areas. A model trained to handle 200k tokens may have been optimized at the expense of short-context reasoning, instruction-following, or output quality on narrower tasks. The model that is best at everything does not exist. Trade-offs are real.
When to use long-context windows, and when not to
Long-context models are best suited for document processing, code understanding, and conversation history. A few concrete use cases:
Summarizing a 20k-word research paper in a single request rather than splitting it into chunks.
Analyzing a full codebase file (say, 10k to 50k tokens of code) without losing context across functions and classes.
Maintaining a full conversation thread of 20+ turns without summarizing or truncating history.
Processing a legal contract, book chapter, or technical manual in one request.
Running a few-shot learning scenario with 20+ examples plus a query, all in one prompt.
Long-context windows are not necessary, and often counterproductive, for:
Single-turn question answering ("What is the capital of France?").
Classification tasks with consistent, short inputs.
Retrieval-augmented generation where semantic search already filters the corpus to the most relevant snippets.
Real-time applications where latency must be under 500ms.
Workloads where you need high throughput and cost efficiency, rather than convenience.
The honest answer: if you are unsure whether you need long context, you probably do not. Start with a 4k or 8k context model. Add longer context only when you hit a concrete limitation. In many cases, a hybrid approach where you use semantic search to pre-filter documents, then pass the filtered results to the LLM, outperforms blind context-stuffing both in quality and cost.
Tokenization quirks that surprise engineers
Tokenization is not universal. Different models tokenize the same text differently. OpenAI's tiktoken tokenizer, Anthropic's tokenizer, and open-source models each have their own vocabulary. The same 1000-word essay might be 1300 tokens in one model and 1500 in another. This is not a bug; it is a consequence of different training corpora and design choices.
A second surprise: whitespace and formatting matter. A string with many newlines, tabs, or inconsistent spacing will tokenize differently from the same string formatted tightly. Leading and trailing spaces around words can add tokens. This matters when you are trying to fit content precisely into a context window.
Code and non-English text tokenize less efficiently than English prose. A line of Python or JSON can require 1.5 to 2 times as many tokens as the equivalent amount of English prose. If you are building a system that processes code, budget generously for context window usage.
Numbers and proper nouns are inefficient. The number "123456789" might tokenize into 4 to 5 tokens, not 1. Long names, URLs, and hex strings likewise break into many tokens. If your workload includes lots of these, your effective context window is smaller than the nominal limit.
Common pitfalls when building on context windows
Underestimating tokenization costs is the most frequent mistake. Developers estimate content size in characters or words, then run an API call and find themselves $50 over budget. Use a tokenizer before you deploy.
Assuming context window size equals quality is a second pitfall. A 200k context is not automatically better than a 64k context. The model architecture, training data, and instruction-tuning matter more than raw size. A smaller model with superior instruction-following can outperform a larger model with more raw context.
The third pitfall: forgetting that context is not free storage. Some developers treat a long-context window as a way to avoid a proper database or retrieval system. They stuff historical data, cached results, and log files into prompts. This works until you hit latency timeouts or usage caps. A database with semantic search is almost always cheaper and faster for retrieval than scanning through a long prompt.
A fourth pitfall: neglecting to test for the lost-in-the-middle problem. If your application puts critical information in the middle of a long context, test how well the model retrieves it. Do not assume a 200k window means the model is equally attentive across all 200k tokens.
Finally, ignoring the cost of token-efficient alternatives. A system that uses RAG with a 4k context window might process the same documents as one using a 128k window, but at a fraction of the cost and latency. The engineering overhead is real, but so are the benefits. Evaluate the trade-off.
Practical next steps
If you are building on LLMs and sizing context windows, start here: use a tokenizer to measure your actual input sizes. OpenAI's tiktoken is free and accurate for GPT models. Anthropic publishes a token counter. Count the characters in your prompts, examples, and documents, then convert to tokens. This takes an hour and saves weeks of confusion later.
Second, model your costs. A 100k token input at $3 per million tokens costs $0.30 per request. If you process 1000 requests a day, that is $90 per day, or $32,850 per year, for input alone. Calculate the same for a 10k context approach. The gap often surprises people.
Third, if you are working with large documents or knowledge bases, test RAG first. Build a semantic search retrieval system that returns the top 5 to 10 most relevant passages, then pass only those to the LLM. Measure latency and quality. Compare it to the brute-force long-context approach. In most real-world scenarios, RAG wins.
Finally, do not chase the latest 200k context window just because it exists. Context windows are a tool, not a goal. Use what solves your problem at the lowest cost and latency. If that is a 4k window with RAG, great. If you genuinely need 128k because you are analyzing full codebases or long research papers, use it without guilt. Measure, test, and optimize based on your actual constraints, not industry hype.
This article was originally published on AI Glimpse.
Top comments (0)