How LLMs Actually Work: Tokens, Context, and Temperature
Every chunk size you pick, every API bill you pay, and every weird agent bug you debug traces back to three things: tokens, context, and temperature. Tutorials treat them as vocabulary. For an AI engineer they are the budget, the working memory, and the randomness dial of every system you ship. This part opens the black box with real numbers you can verify yourself.
Part 3 of the AI Engineer Series. Part 1 mapped the full roadmap from 390 real job postings, and Part 2 covered the Python that matters for AI work: asyncio, API patterns, and production habits. This part goes inside the model itself. Part 4 builds RAG on top of what you learn here.
Tokens are not words, and they are not characters
A token is a piece of text the model's tokenizer learned to treat as one unit. Modern tokenizers use byte-pair encoding (BPE): they start from single bytes and repeatedly merge the most frequent pairs until they reach a fixed vocabulary. The result is a lookup table where common chunks get their own token and rare ones get split apart.
I measured this with tiktoken, OpenAI's open-source tokenizer, so these are real numbers, not estimates:
"tokenization" splits into ["token", "ization"], 2 tokens. The split follows frequency, not grammar.
"embeddings" splits into ["emb", "eddings"], 2 tokens. No morpheme survives.
"The vector database returns the five most similar chunks for the query." (71 characters) becomes 13 tokens.
Its German translation, "Die Vektordatenbank gibt die fünf ähnlichsten Textabschnitte für die Anfrage zurück." (84 characters), becomes 20 tokens. Same meaning, roughly 50% more tokens.
"Donaudampfschifffahrt" shatters into 8 fragments. Long compound words are expensive.
Vocabulary sizes are public: the o200k_base tokenizer used by the GPT-4o and GPT-5 families holds 200,019 tokens, while the older cl100k_base holds 100,277. Different models tokenize the same text differently, so token counts from one provider never transfer exactly to another.
The rough rule of thumb still helps for back-of-envelope math: about 4 characters or 0.75 English words per token. But treat it as a guess. Count for real with code:
import tiktoken
enc = tiktoken.get_encoding("o200k_base") # GPT-4o / GPT-5 family
text = "The vector database returns the five most similar chunks."
print(len(enc.encode(text))) # 13, not len(text) // 4
Why tokenisation decides your bill and your chunk size
Providers bill per token, not per character or per word. That German sentence costs about half as much again as the English one to process, because every API price list is denominated in tokens. When you estimate the cost of a feature, token counts are the only honest unit.
Tokenisation also decides how your data gets sliced. RAG chunkers that cut at "500 characters" produce wildly different token counts for English prose, German compounds, and Python code. Chunk in tokens, using the tokenizer of the model you will actually call. Our hands-on RAG chunking guide tests exactly this: which chunk sizes retrieve best, measured rather than guessed.
The context window is one shared budget
The context window is the maximum number of tokens the model reads and writes in a single request. Beginners imagine it as "how much I can paste in." It is really a shared budget that everything draws from at once:
your system prompt
the conversation history you send back each turn
retrieved documents and tool definitions
the model's hidden reasoning tokens, on reasoning models
the output itself, which is capped separately and lower
This is why "long context" features disappoint in practice. A 1M-token window does not mean 1M tokens of your documents. Your 2,000-token system prompt, the tool schemas, and the running conversation all eat the same budget, and they do it on every single call, because the API is stateless. The model remembers nothing between calls. You rebuild its entire working memory from scratch each time.
1M tokens is the headline, not the usable number
As of October 2026 the frontier has converged near one million tokens: OpenAI's GPT-6 family lists 1.05M, current Claude models list 1M, Google's Gemini 4 Argon and 3.8 Flash accept 1,048,576, and Kimi K3, DeepSeek, Qwen, and GLM-5.3 sit near 1M as well. But three asterisks matter more than the headline:
Output is capped much lower. Claude and GPT-6 allow up to 128K output tokens per request; Gemini defaults to 65,536. Input and output are different budgets.
Apps and harnesses cap you below the API maximum. Consumer chat apps and agent frameworks routinely set their own, lower defaults.
Long prompts can cost more. OpenAI's own pricing page applies a long-context surcharge above 272K input tokens on GPT-6 Astra.
Worse, the advertised number overstates what models actually use well. The RULER benchmark (NVIDIA researchers, 2024) tested 17 models claiming at least 32K context on retrieval, multi-hop tracing, and aggregation tasks. Only about half maintained satisfactory performance at 32K tokens. A 2025 follow-up, NoLiMa, removed lexical shortcuts and found 11 of 13 models fell below half their short-context score at 32K. The field now distinguishes claimed context from effective context, and effective is routinely far shorter.
Two more measured effects complete the picture. "Lost in the middle" (Liu et al., 2023) showed models recall best at the start and end of long prompts and worst in the middle, so burying key instructions mid-prompt is a real failure mode. And long context has a hardware cost: the KV cache that lets transformers attend to past tokens grows with every token. For Llama 3.1 8B at 128K tokens in fp16, the cache alone is 2 x 32 layers x 8 KV heads x 128 dimensions x 2 bytes x 131,072 tokens, about 16 GiB, before the model weights. That is why long prompts are slower as well as pricier.
The engineering takeaway: retrieve instead of stuffing. Send the model the passages it needs, not the whole archive. Our hybrid search guide shows how BM25 plus vectors beats dumping documents into context, and Part 4 of this series builds the full RAG pipeline.
Temperature: one division that changes everything
At each step, the model scores every possible next token (logits) and converts scores to probabilities with softmax. Temperature T divides the scores first:
P(i) = exp(zi / T) / sum(exp(zj / T))
That single division is the whole mechanism. At T = 1 the distribution is untouched. Below 1 it sharpens toward the most likely tokens. Above 1 it flattens, giving unlikely tokens a real chance. As T approaches 0, sampling collapses to greedy decoding: always the top token.
The ranges differ by provider: OpenAI and Gemini accept 0 to 2, Anthropic's Claude accepts 0 to 1, and all default to 1.0. Note the consequence: a 0.1 change on Claude's 0 to 1 scale moves the dial twice as far, proportionally, as the same change on OpenAI's 0 to 2 scale. Compare settings across providers with care.
from openai import OpenAI
client = OpenAI()
# Extraction: deterministic, low temperature
r = client.responses.create(model="gpt-5-mini",
input="Extract the invoice total as JSON.",
temperature=0.1, max_output_tokens=200)
# Brainstorming: higher temperature, more variety
r = client.responses.create(model="gpt-5-mini",
input="Suggest five product names.",
temperature=0.9, max_output_tokens=200)
What tutorials skip about temperature
Tune temperature or top-p, not both. This is the official guidance from OpenAI and Anthropic. Top-p (nucleus sampling) cuts the candidate pool to the smallest set whose probabilities sum to p; changing both knobs at once makes behaviour hard to diagnose. Leave top-p at default and tune temperature first.
Temperature 0 is not fully deterministic. GPU floating-point math, request batching, and model routing introduce small variations. One researcher ran 1,000 identical prompts at temperature 0 and got 80 unique responses. For reproducibility, combine temperature 0 with a seed parameter, and treat even that as best-effort, never a guarantee.
Reasoning models may lock the parameter. OpenAI's o-series and Anthropic's extended-thinking models manage sampling internally and reject or ignore non-default temperature and top-p. Check the model's docs before relying on the knob.
Temperature does not make the model smarter. It widens the sampling distribution; it never improves the model's understanding of your prompt. A vague prompt at temperature 0.9 gives you varied confusion, not insight.
Use temperature 0 for anything you grade. Evals, benchmarks, regression tests, and automated pipelines should run at temperature 0 with a fixed seed so a score change means the system changed, not the dice. Our RAG evals guide follows exactly this discipline.
A practical starting table: 0 to 0.2 for extraction, classification, and code; 0.3 to 0.7 for balanced chat; 0.7 to 1.0 for brainstorming and copy. Then adjust one step at a time and watch real outputs, because the right value depends on your prompt as much as your task.
The AI engineer's mental model
Put the three together and you get the working model every later part of this series assumes:
Count tokens with the right tokenizer before you send. Budget them like money, because they are money. See our measured LLM cost guide for what actually moves the bill.
Treat context as a shared, stateless budget. Nothing persists between calls. Your system prompt bills on every call, history grows every turn, and output has its own lower cap.
Respect effective context, not the headline. Keep prompts lean, retrieve rather than stuff, and put critical instructions at the start or end, never buried in the middle.
Set temperature deliberately. Near 0 with a seed for anything graded or automated; higher only when you genuinely want variety. Never tune temperature and top-p together.
What is Next in This Series
Part 1: AI Engineer Roadmap 2026: What 390 Job Postings Actually Require
Part 2: Python for AI Engineers: Asyncio, APIs, and What Tutorials Skip
Part 3: How LLMs Actually Work: Tokens, Context, and Temperature (this article)
Part 4: RAG from Zero: Embeddings, Vector DBs, and Retrieval
Part 5: Evals: The Skill 56% of Jobs Want
Part 6: Building Your First AI Agent with LangGraph
Part 7: MCP: Connecting Agents to Real Tools Safely
Part 8: LLMOps: Shipping AI to Production
Part 4 builds the RAG pipeline in working code: embeddings, a vector database, and retrieval that respects everything you just learned about token budgets and effective context.
Originally published on LogicStack. Run the code examples right in your browser with our free interactive Python and SQL playgrounds — no setup, no sign-up.
Top comments (0)