Everything an AI agent does — writing code, answering support tickets, browsing the web — bottoms out in a single operation: a language model predicting the next token. Understand that one mechanism, and its limits, and the entire "AI agent" stack stops being magic and starts being engineering.
Let's build it up from nothing.
1. Text becomes tokens
A model never sees letters. The first thing that happens to your prompt is tokenization: the text is chopped into tokens — sub-word chunks produced by an algorithm called Byte-Pair Encoding (BPE). A token is roughly ¾ of an English word (about 4 characters). Common words like "the" are a single token; "unbelievable" might be three (un / believ / able). Each token maps to an integer ID in a fixed vocabulary of tens of thousands of entries.
Why should you care? Because everything downstream is counted and billed in tokens, not words. Context limits, latency, and cost are all token-denominated. When someone says a model has a "200K context window," they mean tokens.
2. The transformer reads everything at once — through attention
Token IDs become embedding vectors (lists of numbers), which flow through a stack of transformer layers. The engine of each layer is self-attention, and it's the idea that made modern AI work.
Self-attention lets every token look at every other token to build context — for example, linking the pronoun "it" back to the noun it refers to. Mechanically, each token is projected into three vectors:
- a Query ("what am I looking for?"),
- a Key ("what do I offer?"),
- a Value ("what do I contribute if attended to?").
A token's attention to another is the dot product of its Query with the other's Key, squashed through a softmax so the weights are positive and sum to 1. The output for each token is a weighted blend of the Value vectors. Run several of these in parallel — the original Transformer used eight attention heads — and concatenate the results. Because attention has no built-in sense of order, a positional encoding is added so the model knows word order.
Here's the catch that shapes the whole industry: attention compares every token to every other token, so n tokens create n² relationships. That quadratic cost is the root reason context windows are finite, expensive, and degrade as they fill.
3. It generates one token at a time
An LLM is autoregressive. The final layer produces a logit vector — a raw score for every token in the vocabulary — which softmax turns into a probability distribution over "the next token." The model picks one, appends it to the input, and repeats. That loop is text generation.
So how does it "pick"? Through sampling knobs:
- Greedy decoding always takes the single most probable token. Deterministic, but repetitive.
- Temperature sharpens or flattens the distribution. Low temperature → focused and repetitive; high temperature → diverse but risks incoherence.
-
Top-p (nucleus) sampling draws only from the smallest set of tokens whose probabilities sum to
p. It adapts to the model's confidence at each step, which is why it's usually preferred over top-k. - Repetition penalty (~1.2–1.5) down-weights tokens it has already produced, to break loops.
4. Why models hallucinate
This is the part people find unsettling: the model's objective is plausible next tokens, not truth. It has no built-in database and no grounding step. When it lacks a fact, it still produces the most likely-sounding continuation — confidently.
Hallucination isn't a bug waiting for a patch; it's a property of the mechanism. And it's precisely why the rest of the stack exists. Retrieval (RAG), tool use, and grounding all do the same job: feed the model real facts inside its context so that "most plausible" and "correct" line up.
5. The context window is the model's entire memory
The single most important thing to internalize: an LLM is a stateless function. Tokens in, next-token distribution out. It remembers nothing between calls. Its only working memory is the context window — the tokens you pass in this call (system prompt + conversation history + tools + retrieved documents).
Two consequences shape all of agent engineering:
Every token has an opportunity cost. Recall accuracy degrades as the window fills — more tokens means worse retrieval from within them. This is sometimes called "context rot," and the n² attention cost is why.
Lost in the middle. A well-known 2023 study ("Lost in the Middle") showed that models attend best to information at the start and end of the context and worst to the middle, producing a U-shaped accuracy curve — even in models explicitly built for long context. Simply moving a relevant passage to the middle can degrade the answer without changing a single word of content.
The one idea to carry forward
An LLM is a stateless, next-token predictor with a fixed attention budget. Memory, tools, agents, RAG, and caching are all machinery bolted around that stateless core — to give it state, facts, actions, and efficiency.
Once you see it this way, "agents" stop being a mysterious new species. They're a control loop wrapped around a very good autocomplete. In the next article, we'll build that loop.
Top comments (0)