TL;DR — A glossary to actually understand the terms you hit when reading about LLMs: token, embedding, attention, KV cache, GQA, MoE, quantization and the rest. But not alphabetical — in dependency order: every entry uses only concepts already explained above, so if you read it start to finish, by the time you reach "Mixture of Experts" you already have all the pieces to get it. The analogies come from the world of the people who hang around this blog: networks, caches, PID controllers, tuning. No math prerequisite beyond "I know what an array is."
Alphabetical LLM glossaries have a structural flaw: the "attention" entry sends you to "softmax", which is further down, which in turn uses "logits", which is earlier but assumes "vocabulary". You end up hopping around like a Wikipedia page at two in the morning.
This document is built the other way around: like a chain. It's the glossary I wish I'd had, and the analogies are the ones that made the concepts click for me — someone coming from sysadmin, networks and controllers, not a PhD in ML. If you too think in terms of caches, error signals and control loops, you're in the right place.
A note of honesty, before we start. This glossary doesn't come from knowledge I already possess. It comes from the questions I asked an LLM — "explain the KV cache to me", "why that √d?" — and from the explanations it wrote for me. Some sank in, others I'm still chewing on: I don't fully master everything you read here, and I won't pretend to. What I needed was a single place to come back to — a notebook to reopen when I run into one of these terms while reading something on the topic, to reread it slowly and with the analogies that pin it down for me. I've left it here in case it's useful to you too; but it is, first and foremost, mine.
To keep our feet on the ground I'll use as a throughline the numbers of a small but modern-architecture model: 180 million parameters, a 32,768-token vocabulary, internal size 640, 20 layers. Small enough to train on human hardware, modern enough to contain every concept that matters in 2026.
Level 0 — What an LLM actually does
Language modeling
The whole game is ONE thing: given a sequence of text, predict the next piece. "The cat is on the" → probably "roof" or "couch", improbably "carburetor". An LLM is a giant function that takes text in and returns a probability distribution over the next piece. Everything else — chat, reasoning, code — emerges from this task repeated billions of times over human text. When it "generates", the model predicts a piece, appends it to the input, predicts the next, and so on. Like autocomplete taken to an absurd extreme.
Token
The "piece" above. Not a word nor a character: something in between, a frequent fragment. "computing" might be a single token; "quadcopter" might split into "quad"+"cop"+"ter". The tokenizer is the component that converts text ↔ sequences of integers (each token has an ID). Think of a lookup table: token 4521 = " cat". The vocabulary is the set of all known tokens — in our reference model, 32,768.
Analogy: it's a form of dictionary compression, conceptually a cousin of LZ — frequent sequences become single symbols.
BPE (Byte-Pair Encoding)
The algorithm that builds that dictionary: in the modern byte-level variant, you start from single bytes, find the most frequent adjacent pair in the corpus, merge it into a new symbol, and repeat until you reach the desired vocabulary size. Purely statistical, no linguistic magic.
Level 1 — The mathematical building blocks (the bare minimum)
Vector, matrix, tensor
A vector is an array of floats. A matrix is a 2D array. A tensor is the generalization to N dimensions — in PyTorch code you see shapes like (B, T, D) = (batch, positions in the sequence, dimensions per position). Nothing mystical: multidimensional arrays with vectorized operations, like NumPy.
Matrix-vector multiplication = transformation
The fundamental operation of ALL deep learning: y = W @ x. Take a vector x, multiply it by a matrix W of weights, get a new vector y. Each element of y is a weighted combination of all the elements of x. A matrix = one learned transformation from one space to another. An LLM is, brutally, hundreds of these multiplications in a row with a bit of non-linearity in between. When you read "linear layer" or nn.Linear: this is it. When you read GEMM: General Matrix Multiply, the kernel GPUs grind on.
Parameters (or weights)
The numbers inside those matrices. "180M-parameter model" = the sum of all elements of all matrices is 180 million floats. It's the parameters that get modified during training — the model's "knowing" lives entirely there. The architecture is the printed circuit board; the parameters are the component values that training solders on.
Embedding
The bridge between tokens (integers) and math (vectors). A table: row 4521 = the 640-float vector representing " cat". The deep part: during training these vectors organize themselves geometrically by meaning — "cat" and "feline" end up close in this 640-dimensional space, "cat" and "lathe" far apart. The direction itself encodes semantic relations. If you've ever used pgvector or a vector database for semantic search, it's the exact same concept — except here the embeddings are internal to the model and learned along with everything else.
d_model
The model's "width": how many floats represent each token as it flows through the network. In our reference, 640. Each token enters as a 640-vector, exits every layer as a 640-vector (enriched with context), all the way to the end. It's the system's data bus.
Activation function (non-linearity)
After a matrix multiplication you apply a non-linear function element by element (ReLU, SiLU, GELU — variants on the same theme: squash or let through). Why it's needed: a chain of purely linear operations mathematically collapses into ONE linear operation — you'd learn nothing complex. The non-linearity is what lets the network represent arbitrary functions. When you see F.silu in code: this is it.
Softmax
A function that takes a vector of arbitrary numbers ("scores" or logits) and turns it into a probability distribution: all positive, summing to 1, the high scores dominating. It's the last step before picking the next token, and it also appears inside attention. Sigmoid is its cousin for the single case: it squashes one number into (0,1) independently of the others.
Level 2 — How it learns: training
Forward pass
Running the input through the model to the output. Text → tokens → embeddings → 20 layers of transformations → probabilities over the next token. Just computation, no learning.
Loss (cost function)
The number that says "how wrong you were". For LLMs it's cross-entropy: take the token that actually followed in the training text, look at what probability the model assigned it, and the loss is -log of that probability. If the model gave the right token probability 0.9 → low loss. If it gave it 0.001 → high loss. It's the only learning signal: all of training is "lower this number".
A sanity check practitioners use at the first step: a freshly initialized model fires at random, uniform probability 1/32,768 over every token, so the initial loss must be -log(1/32768) ≈ 10.4. If it starts there, initialization is healthy. The perplexity you find in evaluations is just e^loss — "among how many tokens the model is effectively hesitating".
Gradient and backpropagation
The gradient answers: "if I move this parameter by an epsilon, does the loss go up or down, and by how much?". Backpropagation is the algorithm (a systematic application of the chain rule from differential calculus) that computes this derivative for ALL 180 million parameters in one shot, propagating backward from the final error. In PyTorch it's the line loss.backward(): automatic, you never implement it by hand.
Gradient descent and the optimizer
Once you know which direction to move each parameter, you move them all a tiny step in that direction. Repeat millions of times. The learning rate (LR) is the step size — the single most important hyperparameter. Too high: the loss explodes or oscillates. Too low: you learn at a snail's pace.
An analogy for anyone who's touched a PID loop: the gradient is the error signal, the learning rate is the gain. Gain too high = oscillations and crash, too low = mushy response.
The optimizer is the strategy for using the gradients: the basic one (SGD) just applies them; AdamW keeps moving averages of both the gradient (1st moment) and its square (2nd moment) — it's the latter that adapts the step per-parameter; Muon (the 2025 newcomer) applies a geometric "cleanup" to matrix gradients before using them. No need to understand the how right now — just to know that the optimizer is the how you descend, and that some descend faster.
Batch
You don't process one example at a time: you pack N sequences together (the batch) and average the gradient. Reason 1: GPUs live on parallelism. Reason 2: a gradient averaged over half a million tokens is far less noisy than one from a single sequence. Gradient accumulation (grad_accum in code) is a trick to simulate huge batches when VRAM isn't enough: you accumulate the gradients of N micro-batches before taking the step.
Epoch, step, checkpoint
Step = one parameter update. Epoch = one full pass over the dataset (in LLM pre-training, historically you often didn't even complete a single epoch — the dataset was bigger than the compute budget; today, with curated, high-quality data, seeing it more than once is common again). Checkpoint = a dump of the parameters to disk, your state save.
Overfitting and held-out
If the model memorizes the training set instead of generalizing, the training loss drops but on unseen data it doesn't. That's why you keep a held-out set (data never shown) as an honest measure. There's also a reverse use of the phenomenon, a classic sanity check: overfit a single batch on purpose. If the model can't even memorize 32 sequences, there's a bug in the code.
Hyperparameters
Everything you choose and isn't learned: learning rate, dimensions, number of layers, batch size... Hyperparameter tuning is the equivalent of tuning Betaflight on a drone: there's theory, there are sensible starting values, and then there's empirical experience.
Level 3 — The Transformer architecture
Why attention is needed: the context problem
A token on its own is ambiguous ("bank": a river's edge, a place for money, a plane banking?). Meaning depends on context. The architectural problem is: how does the representation of the token at position 500 incorporate information from the preceding tokens? The pre-2017 answer was recurrent networks (RNNs), which read in sequence and compress everything into a state — a bottleneck and no parallelism. The Transformer's answer (2017, "Attention Is All You Need") is attention.
Self-attention: the intuition
Each token makes a "query to the database" of the other tokens. Mechanically, from each token you derive three vectors (three matrix multiplications, the classic wq, wk, wv):
- Query (Q): "what I'm looking for" — e.g. the token "on" looks for a subject to attach to
- Key (K): "what I offer as a search key" — e.g. "cat" advertises itself as an animal-subject
- Value (V): "the content I deliver if I'm selected"
For each token: take the dot product of its Q with the K of all preceding tokens (→ affinity scores), rescale by √(head dimension) (here √64 = 8) — without it, with large vectors the scores grow, the softmax saturates and the gradients die — then softmax over the scores (→ weights summing to 1), and the token's new representation is the weighted average of the V. Result: each token "absorbs" information from the relevant tokens, with weights that are learned and content-dependent.
Analogy: a lookup on a hash table, but fuzzy and differentiable — instead of an exact match on the key, you get a similarity score with all the keys and draw from all of them in proportion.
Causal mask
In language modeling the token at position t can look only at positions ≤ t — otherwise during training it would "peek" at the answer. The causal mask zeroes out attention toward the future. In PyTorch code it's the is_causal=True flag.
Multi-head
Instead of ONE attention with large vectors, you do N attentions in parallel with small vectors ("heads"), then concatenate. Each head learns to look for different things: one tracks syntactic dependencies, another coreferences, and so on. In our reference model: 10 heads of 64 dimensions each.
FFN (Feed-Forward Network)
The second component of every layer: two (or three, with SwiGLU) matrix multiplications with a non-linearity in between, applied to each token independently. If attention is where tokens talk to each other, the FFN is where each token processes on its own what it gathered. It's also where much of the model's "factual knowledge" is thought to reside — and it's the part that MoE multiplies into experts (we'll get there).
Layer (block) and residual stream
A transformer block = attention + FFN, each preceded by normalization. The model is a stack of identical blocks (20 in our reference). Crucial detail: the residual connections — the output of each component is added to the input, it doesn't replace it (x = x + attn(...)). Picture a "conveyor belt" (the residual stream) carrying the token's representation through the layers, where each block reads it and adds its contributions. Without this, gradients couldn't flow backward through 20 layers (vanishing gradient) and deep networks wouldn't train.
Normalization (LayerNorm → RMSNorm)
Before each component you rescale the token's vector to a standard "magnitude". Reason: numerical stability — without it, the magnitudes of the values drift layer after layer until they explode or vanish. Analogy: AGC (automatic gain control) in a radio chain, or leveling signals between stages. RMSNorm is the minimal version used today (scale only, no centering). "Pre-norm" = you normalize at the input of each component.
Positional encoding and RoPE
Problem: attention by itself doesn't know WHERE the tokens are — it's an operation on sets, "the cat bites the dog" and "the dog bites the cat" would give the same scores. You need to inject position. The modern method is RoPE: rotate the Q and K vectors by an angle proportional to the token's position (pairs of dimensions = planes of rotation, different frequencies per pair). The elegant consequence: the dot product between Q and K ends up depending on the relative distance between tokens, not on absolute positions — exactly what you want for language. Honest analogy: phase encoding — the position information lives in the signal's phase, and the phase difference gives you the distance.
Logits and sampling
At the end of the stack, the last layer (the language modeling head, lm_head) projects the token's 640-dim vector to 32,768: one score per vocabulary token (the logits). Softmax → probabilities → the next token is drawn. Temperature divides the logits before the softmax: <1 makes the distribution more concentrated (conservative), >1 flatter (creative). Top-p / nucleus sampling cuts the tail: it samples only from the most probable tokens that cumulatively reach probability p.
KV cache
During generation, at each new token the attention computation reuses the K and V of all previous tokens — which don't change. Recomputing them every time would be insane: you keep them in memory. This is the KV cache, and its size is THE inference bottleneck (it grows with context × layers × KV heads × head_dim × 2, for K and V). When you read that GQA, sliding window and MLA exist "to compress the KV cache", now you know what it is: the working memory of generation. Perfect analogy: it's memoization, pure caching of immutable intermediate results.
Context window
The maximum number of tokens the model can keep in view at once: the prompt plus everything it has already generated. It has two distinct limits. A learned one — the RoPE frequencies seen in training: beyond that length the model can no longer place positions and quality collapses. A physical one — the KV cache: double the context, double the memory. Extending it after training is possible (RoPE scaling, YaRN: you rescale the frequencies to "stretch" the window beyond the training length), but it's never entirely free in quality. When you read "128k context", this is it — and there's almost always a trade-off behind it.
Level 4 — The "modern" terms, now decipherable
GQA (Grouped-Query Attention)
In classic multi-head each head has its own K and V → huge cache. GQA: many Query heads share a few K/V heads (e.g. 10 query heads, 2 KV heads). Cache reduced 5×, quality almost identical. Cache deduplication, in practice.
Sliding window attention
Most layers limit attention to the last 512 tokens (a sliding window) instead of the whole context; only some layers see everything (one in 4 in our reference — the ratio varies from architecture to architecture). Distant information still travels: layers compose (layer 2's window sees tokens that had already absorbed context in layer 1). Less compute, less cache.
MLA (Multi-head Latent Attention)
The third way to tame the KV cache, after GQA and sliding window. Instead of caching K and V for each head, MLA compresses them into a single low-rank latent vector and caches only that; the per-head K and V are reconstructed on the fly with a multiplication when needed. Much smaller cache than GQA at comparable quality, in exchange for a bit more compute. It's DeepSeek-V2/V3's choice. Same conceptual family as GQA and sliding window: all three trade a sliver of compute or quality to fit more context in memory.
MoE (Mixture of Experts) — now you have the pieces
Remember: the FFN is each block's "individual-processing" component, and it holds much of the parameters. MoE's idea: instead of ONE FFN per block, you put 32 (the experts) and a router — a small linear layer that, for each token, picks the 4 most suitable experts. Only those 4 compute. Result: the model has a capacity comparable (not identical) to 32 FFNs — capacity follows total parameters — but costs as much as 4 (compute = active parameters). Specialization emerges on its own from training: no one assigns topics to the experts.
Analogy: a content-aware load balancer in front of a pool of specialized workers — with the twist that routing and specializations co-evolve during training.
And the terms that orbit it:
- Fine-grained: 32 small experts beat 8 large ones — more possible combinations, finer specialization.
- Shared expert: one always-active expert for generic knowledge, so the others don't have to duplicate it.
- Routing collapse: the pathology — the router converges on sending everything to 2-3 experts (which therefore improve, therefore get even more traffic: positive feedback). The other experts stay untrained: dead parameters.
- Aux-loss-free balancing: the cure made famous by DeepSeek. A per-expert bias added to the scores for selection only: overloaded expert → bias down → chosen less. It's literally an integral controller on load — a control loop outside training. If you've got a head for PID, you'll get it better than the average ML engineer.
bf16 / mixed precision
16-bit floating point formats instead of 32: half the memory, double the throughput on tensor cores. bf16 keeps the same range as fp32 (sacrificing mantissa), so no overflow to manage. "Mixed" because the delicate operations (norms, loss sums) stay in fp32.
torch.compile, FlashAttention, MFU
torch.compile: a JIT that fuses operations into optimized GPU kernels. FlashAttention: an attention implementation that never materializes the T×T score matrix (it computes it in blocks in SRAM) — same math, memory from O(T²) to O(T). MFU (Model FLOPs Utilization): what fraction of the GPU's theoretical FLOPs you're actually using — your pipeline-efficiency indicator.
Pre-training / SFT / DPO / RLHF-RLVR — the pipeline
- Pre-training: pure language modeling on billions of web tokens. Produces a well-read "completer" but not an assistant.
- SFT (Supervised Fine-Tuning): fine-tuning on examples of well-formed conversations. Teaches the question→answer format.
- DPO/RLHF: refinement on preferences — good/bad response pairs. Teaches style and alignment.
- RLVR/GRPO: reinforcement learning on verifiable-answer tasks (math, code). This is where reasoning models are born. The key difference from SFT: the model learns from its own attempts rewarded/punished, not from given examples.
LoRA / QLoRA (PEFT)
Fine-tuning all the parameters (our model's 180M, or a real one's 7 billion) means keeping them in VRAM with their gradients and optimizer states: prohibitive on human hardware. LoRA works around it: it freezes the base weights and trains only small low-rank add-on matrices (A·B, a few thousand parameters) alongside some layers. The bulk stays put, you touch 1% and get most of the result. QLoRA adds the stroke of genius: you keep the base weights quantized to 4 bits (frozen) and put the LoRA adapters on top — so you fine-tune a 7B on a single consumer GPU. PEFT (Parameter-Efficient Fine-Tuning) is the family that contains them.
Reasoning and test-time compute
The thread that starts from RLVR. The idea: let the model generate a long chain of reasoning (chain-of-thought) before answering, and train it — via RLVR — to do it well. The consequence is a paradigm shift: you don't spend compute only in training, you spend it in inference — more "thinking" tokens = better answers on hard problems. It's test-time compute: same architecture, but one that "stops to think". Reasoning models (o1 / R1 style) are born this way. And note the link to everything else: those reasoning tokens fill the context window and inflate the KV cache. Thinking, for an LLM, is paid for in memory and latency.
Scaling laws / Chinchilla
Empirical relationships between compute, parameters and tokens: for a given compute budget there's an optimal ratio (~20 tokens per parameter, "Chinchilla"). "Overtraining" = going well beyond on purpose, because a small model trained for a long time costs less at inference forever. It's a CAPEX/OPEX trade-off, and in 2026 inference OPEX dominates.
Level 5 — Running the model (inference and serving)
So far: how the model learns. Now: how you serve it. It's the level missing from almost every glossary, and the one you touch hands-on the day you try to run a model on your own hardware.
Quantization
The bf16 from before is training stuff. At inference you can go lower still: weights at 8 or 4 bits (sometimes less). A 7B in fp16 wants ~14 GB of VRAM; at 4 bits it wants ~4 — often the difference between "it runs" and "it doesn't" on your card. The names you'll meet: GPTQ and AWQ (they quantize the weights only, calibrating on the corpus to lose less quality), GGUF (llama.cpp's format, with its mixed-precision k-quants). The trade-off is always the same: fewer bits, less memory, a sliver less quality — and below 4 bits the sliver becomes a rope.
Prefill vs decode
Inference has two phases with opposite profiles. Prefill processes the whole prompt at once, in parallel: it's compute-bound (lots of multiplications; the first token comes out). Decode generates one token at a time, and for each it must reread the entire KV cache: it's memory-bandwidth-bound — you're not waiting on FLOPs, you're waiting on memory. It's why two models with the same FLOPs can generate at wildly different speeds, and why optimizing inference is 90% optimizing memory movement. Everything else in this level follows from here.
PagedAttention and continuous batching
The two ideas that made modern serving (vLLM) efficient — and for a sysadmin they're home turf. PagedAttention: instead of allocating each request's KV cache as a contiguous block (which fragments and wastes memory, like a naive allocator would), it splits it into non-contiguous pages with a translation table. It's virtual memory applied to the KV cache, full stop. Continuous batching: instead of waiting for all the requests in a batch to finish before accepting new ones, you swap them in hot — as soon as one sequence ends, another takes its place and the GPU stays full. Together they're worth an order of magnitude of throughput over serving one request at a time.
Speculative decoding
A trick to speed up decode without changing the output. A small, fast draft model proposes the next N tokens; the big model verifies them all in a single forward pass (verifying is parallel, generating is sequential), accepts the correct prefix and discards from the first error on. If the draft guesses often — and on predictable text it does — you go 2-3× faster with exactly the same distribution as the big model. Zero quality trade-off, just less latency.
Inference metrics
The numbers you judge a serving setup by. TTFT (time to first token): how long until it starts answering — dominated by prefill. Generation speed (tokens/s per request): dominated by decode. Throughput (aggregate tokens/s across all requests): what matters if you serve many. And the key trade-off, throughput vs latency: big batches = more throughput but more wait for the single user; small batches = the opposite. The same dilemma as any queueing system.
Special tokens and chat template
A base model completes text; an instruct model expects a structure. Special tokens are vocabulary tokens that aren't words but markers: begin/end of sequence, and the role delimiters (<|im_start|>user, <|im_start|>assistant…). The chat template is the schema — usually a Jinja string that ships with the model — that packs your messages into exactly the form seen during SFT. Getting the template wrong, or forgetting it, is the number-one cause of local models that "answer weird": it's not the model being dumb, it's that you're speaking to it in a dialect it never learned.
And now?
If you got this far reading straight through, you have the map. But a map isn't the territory, and the real leap isn't made by any glossary: it's made by Andrej Karpathy's "Neural Networks: Zero to Hero" video series (YouTube, free). In particular the first two videos (micrograd and makemore) build backpropagation and a tiny language model by hand, and "Let's build GPT" builds attention line by line. They're ~15 hours total, and worth more than anything written — this document included.
One last thing, in the spirit of honesty: the "it resonates but doesn't quite land" feeling is the normal and correct state after a first read. These concepts aren't understood by reading — they're understood the third time your training diverges and you find out why. The glossary only exists so that, when it happens, you know where to look.
Top comments (0)