DEV Community: MojaLab

My coding-agent buddies live on a €5 VPS, and I watch them from the web

MojaLab — Thu, 06 Aug 2026 13:09:58 +0000

The scene: I'm at home, Claude Code is running on my MacBook, forty minutes into a refactor and nowhere near done, and — whoops, it's late. I have to leave, and life does not respect long-running agents. On a laptop-bound setup that's a dead session: SSH drops, tmux is on the other machine, the agent is orphaned mid-task.

If instead I were working on my VPS, I'd close the laptop, walk out, and twenty minutes later pull out my phone, open mterm.mylab.example.com, pass Authelia (password + TOTP), and find myself in front of the same terminal — the agent still chewing through files, me scrolling its output with a thumb.

You can rent a cloud IDE and get something similar with three clicks and a subscription. It's a perfectly fine answer for most people. But if you've been around MojaLab for a while you know the rule of the house: if a problem can be solved with off-the-shelf SaaS or with a weekend, a terminal and a few open-source tools — we take the weekend. Not because the SaaS is bad — it isn't — but because we like to understand by doing.

So I built mojalab-vps-stack: a Docker Compose stack that turns any cheap VPS into a browser-first workbench for coding agents. Ten containers (one a one-shot initializer), one network, one Caddyfile. MIT-licensed, on GitHub.

One gate (Authelia) in front of every door; the workspace shared between terminal and file manager; the plumbing talks to Docker only through a read-only proxy.

A note on what this post is. The full step-by-step setup — DNS records, Authelia secrets, the guided installer, TOTP enrollment, troubleshooting — lives in the README on the repository. It's the canonical reference and it gets updated when the stack does. If you want the how-to, go there (or just run ./scripts/install.sh and answer the questions). This post is the why: the design decisions, the non-obvious failure modes, and what I learned running coding agents on this thing for real.

Why browser-first

The usual way to work on a remote dev box is SSH plus tmux, and it works — for you, from your laptop, with your keys. It stops working the moment the client isn't your laptop: a phone, a tablet, a borrowed machine, a locked-down corporate desktop. And "install Termius and copy your private key to your phone" is a sentence that should make you flinch.

The browser is the one client that exists everywhere. So the design goal was: every useful thing reachable from a URL, every URL behind one real authentication gate. Not a VPN (nothing to install), not per-app passwords (one gate, one session), not security-by-obscurity on a high port (scanners don't care).

The gate is Authelia: single sign-on with TOTP second factor, enforced by Caddy's forward_auth on every subdomain. Behind it:

term.* — Zellij's native web client: full multiplexer, panes, tabs, sessions that survive disconnects
mterm.* — a mobile-friendly terminal (wetty) that SSHes into the same Zellij container and can join the same session
files.* — Filestash, a web file manager over the shared workspace
stats.* — Glances, so you can watch the agent eat CPU in real time
home.* — a landing page with links, because muscle memory needs a home

Plus the plumbing you don't see: Watchtower (monitor-only — it tells me about updates via Telegram, it never surprise-updates anything), a Telegram bot for /stats and proactive alerts, and a read-only Docker socket proxy so neither of those can ever exec into a container even if compromised.

The workbench: a container the agents can trash

The heart of the stack is the Zellij container, and it's shaped by one observation: coding agents are messy tenants. They install packages, compile things, run test suites, spawn processes. You want to give them room to do that — and walls they can't take down.

A preemptive confession about Zellij: at first it's a little annoying. The shortcuts aren't tmux's, the keybinding bar looks like noise, and your first ten minutes go into closing panes you opened by accident. Then it clicks — panes, tabs, the session manager, the detach that just works — and you grow fond of it. Give it an evening before you judge it.

The room: Debian trixie with Node 22, Python 3.13, build-essential, git, gh, uv, ripgrep, and — this one matters more than it looks — a real /usr/local/bin/fd symlink, not a shell alias. Aliases only exist in interactive shells; when Claude Code runs fd through sh -c, an alias is invisible and the command just fails. If you've ever wondered why an agent claims a tool "isn't installed" when you can clearly run it yourself: check whether it's an alias.

The walls: the container gets a memory limit, a CPU limit and a pids limit (an agent-written test suite fork-bombing your VPS into a reboot is funnier in retrospect than at 11 PM). The limits are set in .env, and the installer suggests values based on the host — total RAM minus about 3 GB for the OS and the rest of the stack. On an 8 GB VPS, give the workbench 5 GB; on the €5, 4 GB box from the title, the installer lands on about 1 GB — enough for the agents themselves, but keep an eye on the heavy builds. Do not lowball this one, and here's the war story: the stack originally shipped with a 512 MB limit on this container, which sounds reasonable until you learn how the OOM killer chooses its victim — it kills the biggest process in the cgroup, which after a while is Zellij itself. Result: one hungry npm install and every session, including the agent you were watching from the beach, dies. Memory limits on a workbench are not a tidiness feature. Size them like you mean it.

State is split with intent. Project files live under /srv, which both the terminal and the file manager see — upload a tarball from the browser, untar it in the shell. But the credentials — ~/.claude, ~/.config, npm globals, API keys — live in a separate bind mount that Filestash cannot see. Your web file manager should never be one misclick away from serving your Anthropic session token.

Updating the agents without the rebuild dance

Agent CLIs release constantly, and the naive approach — bake them into the Docker image — means every update is a rebuild, and every rebuild kills your sessions. The fix is boring and effective: everything installs into ~/.local, which is a persistent bind mount that comes first in PATH. The image ships an update-agents command:

$ update-agents
→ claude    (npm @anthropic-ai/claude-code)
→ codex     (npm @openai/codex)
→ opencode  (npm opencode-ai)
→ kimi      (official installer)
→ agy       (self-updater)
→ kiro      (self-updater)

One command, no rebuild, no downtime, survives image rebuilds. Two details earned their place the hard way. First, npm's infamous ENOTEMPTY rename bug (stale files in the old package dir, common right after a Node major bump): the script now clears the package directory — which holds no user state — and retries once, automatically. Second, Antigravity's agy is deliberately not installed from npm: there is no official package, and what you find under the obvious names on the registry is squatted placeholders. That's not paranoia, that's the supply chain in 2026. The script only drives agy's own self-updater after you've installed it once from Google's channel.

There's also git-ssh-key, born from doing the same chore on two VPSes in one afternoon: it generates a per-forge ed25519 key in persistent storage, prints the public key with the exact settings URL to paste it into (GitHub, GitLab, or your self-hosted forge), and wires ~/.ssh/config so it keeps working after rebuilds. One command, then git clone git@github.com:... just works.

Prerequisites and windows into the lab: DNS, Resend, Filestash and Glances

Before anything works, you need two things the installer can't do for you.

A domain. The stack is browser-first, and the browser wants names, not ports. You need a domain (even a few-euros-a-year one) and a handful of A records pointing at the VPS's IP: auth, term, mterm, files, stats, home. Or a single wildcard *.lab.yourdomain.tld, which is the route I recommend — one record, zero maintenance when you add a service. Caddy handles the certificates itself, but only if the names resolve: if you run the installer before DNS has propagated, the certs fail and it all looks broken when it's really just early.

A way to send email. This is the prerequisite nobody expects: to let you register your TOTP, Authelia emails you an identity-verification link. No SMTP, no 2FA, no lab. I use Resend: the free tier is more than enough for a single-user lab, the SMTP credentials go into .env and you never think about it again. (For completeness: Authelia also has a filesystem notifier that writes the link to a file instead of sending it — it works, but it forces you into a shell on the host at exactly the moment you're setting up access without a shell on the host. Resend is faster.)

Then there are the two windows into the lab, each worth a few lines.

Glances (stats.*) is the easy one: zero configuration, open the page and watch the agent eat CPU in real time. It's the fastest way to answer "is it working, or has it hung?" without touching the terminal.

Filestash (files.*) is enormously useful but not ready out of the box: on first launch you configure it — admin password, storage backend pointed at the local /srv mount. Five minutes, once, but skip it and you land on a setup page and think something exploded. The README has the exact steps.

And here's a trick that earned its spot in this post: the Zellij token lives happily in a file under /srv. The native web client on term.* has its own authentication, separate and independent from Authelia: a token it generates on first boot and asks for every so often — typically when the session expires or you switch machines. And a token, by its nature, isn't something you know by heart. The practical answer: save it in a file like /srv/notes/zellij-token.txt. Filestash sees it, so from any browser you log in through Authelia, open the file, copy, paste, and you're back in. (The mobile route on mterm.* will never ask you for it — there, access to Zellij goes another way, which I get to in the security section.)

Yes, it's a deliberate trade-off: that token is, effectively, a second line of defense behind Authelia, and putting it in a file reachable behind that same Authelia weakens the second line. For my threat model — single user, mandatory TOTP — I happily accept it in exchange for never having to dig into the VPS to fish it out. We'll come back to this when the security model is on the table in full; if your threat model is different, just don't create the file.

The phone part, and the IME rabbit hole

The mobile terminal is where most of the interesting engineering hides, because mobile browsers actively fight you.

Honestly, I'm not satisfied with the result yet — call it 60%. I use it, but it still needs work. Either way, here's how it works:

Wetty gives you xterm.js over WebSocket, which is 80% of the job. The remaining 20% is why the stack builds a custom Caddy with the replace-response module: Caddy injects a small vanilla-JS keyboard overlay into wetty's HTML on the fly — wetty itself stays untouched, so future wetty releases just work. The overlay adds what soft keyboards don't have: arrows (hold to auto-repeat), Esc, Tab, F-keys, one-tap Zellij actions (new pane, session manager, detach), one-tap :wq, and sticky Ctrl/Alt — tap to arm for one keystroke, long-press to lock.

That sticky Ctrl taught me more about mobile browsers than I wanted to know. The first implementation intercepted beforeinput on xterm's hidden textarea, transformed the next letter into its control byte, and called preventDefault(). Worked beautifully — on desktop. On a phone it did nothing: you'd arm Ctrl, type o, and a literal o appeared. The reason is that mobile keyboards don't send you a keydown with a letter in it; they compose text through the IME, the keydown arrives as Unidentified keyCode 229, and the composition's beforeinput is frequently not cancelable — your preventDefault() is silently ignored. The fix: intercept at the document level in the capture phase, which runs before xterm's own listeners ever see the event, and use stopImmediatePropagation() — which, unlike preventDefault(), always works. Swallow the event, transform the character, inject the control byte yourself, clear the composition debris. If you're building anything that needs to intercept typed characters on mobile web: element-level preventDefault is a desktop-only illusion.

Same category of lesson, CSS edition: the overlay bar wraps onto a variable number of rows depending on the key set and the screen width, so any hardcoded "reserve 92px for the bar" guess is wrong on some phone — and the terminal's last line hides under the buttons, which on a terminal means hiding the prompt, the only line you actually care about. The bar now measures its own rendered height and publishes it as a CSS variable the terminal sizes against. And on iOS, where the soft keyboard overlays the page instead of resizing it, the bar rides up on visualViewport so Esc and the arrows stay visible exactly when you're typing.

The security model, honestly

Everything behind the gate trusts the gate. That's the design, and it's the right trade-off for a single-user workbench — and the wrong one for anything multi-tenant, so let's be precise about what holds the line:

Two gates before a shell on term.*: Authelia (password + TOTP) and then Zellij's own token. A remote shell on your VPS should not be one phishing away. The mobile route (mterm.*) skips the second gate by design — wetty authenticates into the Zellij container with an internal SSH key that never leaves the Docker network — so there, Authelia alone holds the line. (And if you adopt the token-in-a-file-on-Filestash trick, this becomes the posture everywhere: one gate, the real one, with TOTP. There's even an argument for it: two subdomains with two different postures are harder to reason about than one coherent system where Authelia holds the line, full stop. Know it and choose.)
Inside the container, the lab user has passwordless sudo — deliberately documented rather than hidden. Past both gates, an attacker owns that container. The blast radius is the workbench, not the host: no Docker socket inside, resource limits on the way out.
Watchtower, Glances and the Telegram bot reach Docker through a read-only socket proxy with all write endpoints denied (the bot only reads container logs, to alert on Authelia logins). A compromised metrics dashboard cannot exec into your terminal.
Watchtower never auto-updates. It reports; you read release notes and pull deliberately. Surprise 3 AM updates on the box your agents live on is a genre of fun I've retired from. (Corollary: locally-built images are invisible to Watchtower — bumping Zellij itself is a one-line .env change and a rebuild, on your schedule.)

What it does not defend against: a compromised password and TOTP seed, kernel-level container escapes, or an attacker who already has shell on the host. One kernel, one trust boundary. Know what you're running.

What this is not

This is a personal homelab published as-is, not a product. It's single-user by design. It doesn't back anything up — pair it with something like CryptoSync pointed at /srv and the lab state directory. It assumes you'll read a Caddyfile before trusting it, and the repo's disclaimer is not boilerplate. The installer deliberately refuses to touch your firewall: UFW and Docker's iptables interact in ways that have locked better people than me out of better servers than mine — use your provider's firewall and read the README's hardening notes.

And if what you actually want is "a dev box with zero setup", GitHub Codespaces exists and is genuinely good. This is for people who want the box to be theirs.

Why the journey is worth it

I could have stopped at "SSH works fine". I didn't, because the point was never just remote access. Somewhere between watching the OOM killer take down a session mid-refactor, tracing why a phone keyboard swallows Ctrl, and teaching an installer to tell bytes from megabytes, the stack stopped being a pile of YAML and became something I can reason about — every port, every mount, every trust boundary placed on purpose. Now, when an agent works for forty minutes on my code, I know exactly what room it's working in, what it can touch, and how to look over its shoulder from anywhere with a browser.

That's the MojaLab habit: take the long way once, so the short way feels different ever after.

The repo is at github.com/doradame/mojalab-vps-stack, MIT-licensed. Clone it on a fresh VPS, run ./scripts/install.sh, answer the questions, and fifteen minutes of DNS propagation later your agents have a home. Read the configs before you trust them — including mine.

If you build something on top of it, or break something interesting, I'd love to hear about it.

Made in MojaLab. Several coding agents were mildly inconvenienced by memory limits during the making of this stack.

From "token" to "MoE": the LLM glossary in dependency order

MojaLab — Mon, 03 Aug 2026 22:21:42 +0000

TL;DR — A glossary to actually understand the terms you hit when reading about LLMs: token, embedding, attention, KV cache, GQA, MoE, quantization and the rest. But not alphabetical — in dependency order: every entry uses only concepts already explained above, so if you read it start to finish, by the time you reach "Mixture of Experts" you already have all the pieces to get it. The analogies come from the world of the people who hang around this blog: networks, caches, PID controllers, tuning. No math prerequisite beyond "I know what an array is."

Alphabetical LLM glossaries have a structural flaw: the "attention" entry sends you to "softmax", which is further down, which in turn uses "logits", which is earlier but assumes "vocabulary". You end up hopping around like a Wikipedia page at two in the morning.

This document is built the other way around: like a chain. It's the glossary I wish I'd had, and the analogies are the ones that made the concepts click for me — someone coming from sysadmin, networks and controllers, not a PhD in ML. If you too think in terms of caches, error signals and control loops, you're in the right place.

A note of honesty, before we start. This glossary doesn't come from knowledge I already possess. It comes from the questions I asked an LLM — "explain the KV cache to me", "why that √d?" — and from the explanations it wrote for me. Some sank in, others I'm still chewing on: I don't fully master everything you read here, and I won't pretend to. What I needed was a single place to come back to — a notebook to reopen when I run into one of these terms while reading something on the topic, to reread it slowly and with the analogies that pin it down for me. I've left it here in case it's useful to you too; but it is, first and foremost, mine.

To keep our feet on the ground I'll use as a throughline the numbers of a small but modern-architecture model: 180 million parameters, a 32,768-token vocabulary, internal size 640, 20 layers. Small enough to train on human hardware, modern enough to contain every concept that matters in 2026.

Level 0 — What an LLM actually does

Language modeling

The whole game is ONE thing: given a sequence of text, predict the next piece. "The cat is on the" → probably "roof" or "couch", improbably "carburetor". An LLM is a giant function that takes text in and returns a probability distribution over the next piece. Everything else — chat, reasoning, code — emerges from this task repeated billions of times over human text. When it "generates", the model predicts a piece, appends it to the input, predicts the next, and so on. Like autocomplete taken to an absurd extreme.

Token

The "piece" above. Not a word nor a character: something in between, a frequent fragment. "computing" might be a single token; "quadcopter" might split into "quad"+"cop"+"ter". The tokenizer is the component that converts text ↔ sequences of integers (each token has an ID). Think of a lookup table: token 4521 = " cat". The vocabulary is the set of all known tokens — in our reference model, 32,768.

Analogy: it's a form of dictionary compression, conceptually a cousin of LZ — frequent sequences become single symbols.

BPE (Byte-Pair Encoding)

The algorithm that builds that dictionary: in the modern byte-level variant, you start from single bytes, find the most frequent adjacent pair in the corpus, merge it into a new symbol, and repeat until you reach the desired vocabulary size. Purely statistical, no linguistic magic.

Level 1 — The mathematical building blocks (the bare minimum)

Vector, matrix, tensor

A vector is an array of floats. A matrix is a 2D array. A tensor is the generalization to N dimensions — in PyTorch code you see shapes like (B, T, D) = (batch, positions in the sequence, dimensions per position). Nothing mystical: multidimensional arrays with vectorized operations, like NumPy.

Matrix-vector multiplication = transformation

The fundamental operation of ALL deep learning: y = W @ x. Take a vector x, multiply it by a matrix W of weights, get a new vector y. Each element of y is a weighted combination of all the elements of x. A matrix = one learned transformation from one space to another. An LLM is, brutally, hundreds of these multiplications in a row with a bit of non-linearity in between. When you read "linear layer" or nn.Linear: this is it. When you read GEMM: General Matrix Multiply, the kernel GPUs grind on.

Parameters (or weights)

The numbers inside those matrices. "180M-parameter model" = the sum of all elements of all matrices is 180 million floats. It's the parameters that get modified during training — the model's "knowing" lives entirely there. The architecture is the printed circuit board; the parameters are the component values that training solders on.

Embedding

The bridge between tokens (integers) and math (vectors). A table: row 4521 = the 640-float vector representing " cat". The deep part: during training these vectors organize themselves geometrically by meaning — "cat" and "feline" end up close in this 640-dimensional space, "cat" and "lathe" far apart. The direction itself encodes semantic relations. If you've ever used pgvector or a vector database for semantic search, it's the exact same concept — except here the embeddings are internal to the model and learned along with everything else.

d_model

The model's "width": how many floats represent each token as it flows through the network. In our reference, 640. Each token enters as a 640-vector, exits every layer as a 640-vector (enriched with context), all the way to the end. It's the system's data bus.

Activation function (non-linearity)

After a matrix multiplication you apply a non-linear function element by element (ReLU, SiLU, GELU — variants on the same theme: squash or let through). Why it's needed: a chain of purely linear operations mathematically collapses into ONE linear operation — you'd learn nothing complex. The non-linearity is what lets the network represent arbitrary functions. When you see F.silu in code: this is it.

Softmax

A function that takes a vector of arbitrary numbers ("scores" or logits) and turns it into a probability distribution: all positive, summing to 1, the high scores dominating. It's the last step before picking the next token, and it also appears inside attention. Sigmoid is its cousin for the single case: it squashes one number into (0,1) independently of the others.

Level 2 — How it learns: training

Forward pass

Running the input through the model to the output. Text → tokens → embeddings → 20 layers of transformations → probabilities over the next token. Just computation, no learning.

Loss (cost function)

The number that says "how wrong you were". For LLMs it's cross-entropy: take the token that actually followed in the training text, look at what probability the model assigned it, and the loss is -log of that probability. If the model gave the right token probability 0.9 → low loss. If it gave it 0.001 → high loss. It's the only learning signal: all of training is "lower this number".

A sanity check practitioners use at the first step: a freshly initialized model fires at random, uniform probability 1/32,768 over every token, so the initial loss must be -log(1/32768) ≈ 10.4. If it starts there, initialization is healthy. The perplexity you find in evaluations is just e^loss — "among how many tokens the model is effectively hesitating".

Gradient and backpropagation

The gradient answers: "if I move this parameter by an epsilon, does the loss go up or down, and by how much?". Backpropagation is the algorithm (a systematic application of the chain rule from differential calculus) that computes this derivative for ALL 180 million parameters in one shot, propagating backward from the final error. In PyTorch it's the line loss.backward(): automatic, you never implement it by hand.

Gradient descent and the optimizer

Once you know which direction to move each parameter, you move them all a tiny step in that direction. Repeat millions of times. The learning rate (LR) is the step size — the single most important hyperparameter. Too high: the loss explodes or oscillates. Too low: you learn at a snail's pace.

An analogy for anyone who's touched a PID loop: the gradient is the error signal, the learning rate is the gain. Gain too high = oscillations and crash, too low = mushy response.

The optimizer is the strategy for using the gradients: the basic one (SGD) just applies them; AdamW keeps moving averages of both the gradient (1st moment) and its square (2nd moment) — it's the latter that adapts the step per-parameter; Muon (the 2025 newcomer) applies a geometric "cleanup" to matrix gradients before using them. No need to understand the how right now — just to know that the optimizer is the how you descend, and that some descend faster.

Batch

You don't process one example at a time: you pack N sequences together (the batch) and average the gradient. Reason 1: GPUs live on parallelism. Reason 2: a gradient averaged over half a million tokens is far less noisy than one from a single sequence. Gradient accumulation (grad_accum in code) is a trick to simulate huge batches when VRAM isn't enough: you accumulate the gradients of N micro-batches before taking the step.

Epoch, step, checkpoint

Step = one parameter update. Epoch = one full pass over the dataset (in LLM pre-training, historically you often didn't even complete a single epoch — the dataset was bigger than the compute budget; today, with curated, high-quality data, seeing it more than once is common again). Checkpoint = a dump of the parameters to disk, your state save.

Overfitting and held-out

If the model memorizes the training set instead of generalizing, the training loss drops but on unseen data it doesn't. That's why you keep a held-out set (data never shown) as an honest measure. There's also a reverse use of the phenomenon, a classic sanity check: overfit a single batch on purpose. If the model can't even memorize 32 sequences, there's a bug in the code.

Hyperparameters

Everything you choose and isn't learned: learning rate, dimensions, number of layers, batch size... Hyperparameter tuning is the equivalent of tuning Betaflight on a drone: there's theory, there are sensible starting values, and then there's empirical experience.

Level 3 — The Transformer architecture

Why attention is needed: the context problem

A token on its own is ambiguous ("bank": a river's edge, a place for money, a plane banking?). Meaning depends on context. The architectural problem is: how does the representation of the token at position 500 incorporate information from the preceding tokens? The pre-2017 answer was recurrent networks (RNNs), which read in sequence and compress everything into a state — a bottleneck and no parallelism. The Transformer's answer (2017, "Attention Is All You Need") is attention.

Self-attention: the intuition

Each token makes a "query to the database" of the other tokens. Mechanically, from each token you derive three vectors (three matrix multiplications, the classic wq, wk, wv):

Query (Q): "what I'm looking for" — e.g. the token "on" looks for a subject to attach to
Key (K): "what I offer as a search key" — e.g. "cat" advertises itself as an animal-subject
Value (V): "the content I deliver if I'm selected"

For each token: take the dot product of its Q with the K of all preceding tokens (→ affinity scores), rescale by √(head dimension) (here √64 = 8) — without it, with large vectors the scores grow, the softmax saturates and the gradients die — then softmax over the scores (→ weights summing to 1), and the token's new representation is the weighted average of the V. Result: each token "absorbs" information from the relevant tokens, with weights that are learned and content-dependent.

Analogy: a lookup on a hash table, but fuzzy and differentiable — instead of an exact match on the key, you get a similarity score with all the keys and draw from all of them in proportion.

Causal mask

In language modeling the token at position t can look only at positions ≤ t — otherwise during training it would "peek" at the answer. The causal mask zeroes out attention toward the future. In PyTorch code it's the is_causal=True flag.

Multi-head

Instead of ONE attention with large vectors, you do N attentions in parallel with small vectors ("heads"), then concatenate. Each head learns to look for different things: one tracks syntactic dependencies, another coreferences, and so on. In our reference model: 10 heads of 64 dimensions each.

FFN (Feed-Forward Network)

The second component of every layer: two (or three, with SwiGLU) matrix multiplications with a non-linearity in between, applied to each token independently. If attention is where tokens talk to each other, the FFN is where each token processes on its own what it gathered. It's also where much of the model's "factual knowledge" is thought to reside — and it's the part that MoE multiplies into experts (we'll get there).

Layer (block) and residual stream

A transformer block = attention + FFN, each preceded by normalization. The model is a stack of identical blocks (20 in our reference). Crucial detail: the residual connections — the output of each component is added to the input, it doesn't replace it (x = x + attn(...)). Picture a "conveyor belt" (the residual stream) carrying the token's representation through the layers, where each block reads it and adds its contributions. Without this, gradients couldn't flow backward through 20 layers (vanishing gradient) and deep networks wouldn't train.

Normalization (LayerNorm → RMSNorm)

Before each component you rescale the token's vector to a standard "magnitude". Reason: numerical stability — without it, the magnitudes of the values drift layer after layer until they explode or vanish. Analogy: AGC (automatic gain control) in a radio chain, or leveling signals between stages. RMSNorm is the minimal version used today (scale only, no centering). "Pre-norm" = you normalize at the input of each component.

Positional encoding and RoPE

Problem: attention by itself doesn't know WHERE the tokens are — it's an operation on sets, "the cat bites the dog" and "the dog bites the cat" would give the same scores. You need to inject position. The modern method is RoPE: rotate the Q and K vectors by an angle proportional to the token's position (pairs of dimensions = planes of rotation, different frequencies per pair). The elegant consequence: the dot product between Q and K ends up depending on the relative distance between tokens, not on absolute positions — exactly what you want for language. Honest analogy: phase encoding — the position information lives in the signal's phase, and the phase difference gives you the distance.

Logits and sampling

At the end of the stack, the last layer (the language modeling head, lm_head) projects the token's 640-dim vector to 32,768: one score per vocabulary token (the logits). Softmax → probabilities → the next token is drawn. Temperature divides the logits before the softmax: <1 makes the distribution more concentrated (conservative), >1 flatter (creative). Top-p / nucleus sampling cuts the tail: it samples only from the most probable tokens that cumulatively reach probability p.

KV cache

During generation, at each new token the attention computation reuses the K and V of all previous tokens — which don't change. Recomputing them every time would be insane: you keep them in memory. This is the KV cache, and its size is THE inference bottleneck (it grows with context × layers × KV heads × head_dim × 2, for K and V). When you read that GQA, sliding window and MLA exist "to compress the KV cache", now you know what it is: the working memory of generation. Perfect analogy: it's memoization, pure caching of immutable intermediate results.

Context window

The maximum number of tokens the model can keep in view at once: the prompt plus everything it has already generated. It has two distinct limits. A learned one — the RoPE frequencies seen in training: beyond that length the model can no longer place positions and quality collapses. A physical one — the KV cache: double the context, double the memory. Extending it after training is possible (RoPE scaling, YaRN: you rescale the frequencies to "stretch" the window beyond the training length), but it's never entirely free in quality. When you read "128k context", this is it — and there's almost always a trade-off behind it.

Level 4 — The "modern" terms, now decipherable

GQA (Grouped-Query Attention)

In classic multi-head each head has its own K and V → huge cache. GQA: many Query heads share a few K/V heads (e.g. 10 query heads, 2 KV heads). Cache reduced 5×, quality almost identical. Cache deduplication, in practice.

Sliding window attention

Most layers limit attention to the last 512 tokens (a sliding window) instead of the whole context; only some layers see everything (one in 4 in our reference — the ratio varies from architecture to architecture). Distant information still travels: layers compose (layer 2's window sees tokens that had already absorbed context in layer 1). Less compute, less cache.

MLA (Multi-head Latent Attention)

The third way to tame the KV cache, after GQA and sliding window. Instead of caching K and V for each head, MLA compresses them into a single low-rank latent vector and caches only that; the per-head K and V are reconstructed on the fly with a multiplication when needed. Much smaller cache than GQA at comparable quality, in exchange for a bit more compute. It's DeepSeek-V2/V3's choice. Same conceptual family as GQA and sliding window: all three trade a sliver of compute or quality to fit more context in memory.

MoE (Mixture of Experts) — now you have the pieces

Remember: the FFN is each block's "individual-processing" component, and it holds much of the parameters. MoE's idea: instead of ONE FFN per block, you put 32 (the experts) and a router — a small linear layer that, for each token, picks the 4 most suitable experts. Only those 4 compute. Result: the model has a capacity comparable (not identical) to 32 FFNs — capacity follows total parameters — but costs as much as 4 (compute = active parameters). Specialization emerges on its own from training: no one assigns topics to the experts.

Analogy: a content-aware load balancer in front of a pool of specialized workers — with the twist that routing and specializations co-evolve during training.

And the terms that orbit it:

Fine-grained: 32 small experts beat 8 large ones — more possible combinations, finer specialization.
Shared expert: one always-active expert for generic knowledge, so the others don't have to duplicate it.
Routing collapse: the pathology — the router converges on sending everything to 2-3 experts (which therefore improve, therefore get even more traffic: positive feedback). The other experts stay untrained: dead parameters.
Aux-loss-free balancing: the cure made famous by DeepSeek. A per-expert bias added to the scores for selection only: overloaded expert → bias down → chosen less. It's literally an integral controller on load — a control loop outside training. If you've got a head for PID, you'll get it better than the average ML engineer.

bf16 / mixed precision

16-bit floating point formats instead of 32: half the memory, double the throughput on tensor cores. bf16 keeps the same range as fp32 (sacrificing mantissa), so no overflow to manage. "Mixed" because the delicate operations (norms, loss sums) stay in fp32.

torch.compile, FlashAttention, MFU

torch.compile: a JIT that fuses operations into optimized GPU kernels. FlashAttention: an attention implementation that never materializes the T×T score matrix (it computes it in blocks in SRAM) — same math, memory from O(T²) to O(T). MFU (Model FLOPs Utilization): what fraction of the GPU's theoretical FLOPs you're actually using — your pipeline-efficiency indicator.

Pre-training / SFT / DPO / RLHF-RLVR — the pipeline

Pre-training: pure language modeling on billions of web tokens. Produces a well-read "completer" but not an assistant.
SFT (Supervised Fine-Tuning): fine-tuning on examples of well-formed conversations. Teaches the question→answer format.
DPO/RLHF: refinement on preferences — good/bad response pairs. Teaches style and alignment.
RLVR/GRPO: reinforcement learning on verifiable-answer tasks (math, code). This is where reasoning models are born. The key difference from SFT: the model learns from its own attempts rewarded/punished, not from given examples.

LoRA / QLoRA (PEFT)

Fine-tuning all the parameters (our model's 180M, or a real one's 7 billion) means keeping them in VRAM with their gradients and optimizer states: prohibitive on human hardware. LoRA works around it: it freezes the base weights and trains only small low-rank add-on matrices (A·B, a few thousand parameters) alongside some layers. The bulk stays put, you touch 1% and get most of the result. QLoRA adds the stroke of genius: you keep the base weights quantized to 4 bits (frozen) and put the LoRA adapters on top — so you fine-tune a 7B on a single consumer GPU. PEFT (Parameter-Efficient Fine-Tuning) is the family that contains them.

Reasoning and test-time compute

The thread that starts from RLVR. The idea: let the model generate a long chain of reasoning (chain-of-thought) before answering, and train it — via RLVR — to do it well. The consequence is a paradigm shift: you don't spend compute only in training, you spend it in inference — more "thinking" tokens = better answers on hard problems. It's test-time compute: same architecture, but one that "stops to think". Reasoning models (o1 / R1 style) are born this way. And note the link to everything else: those reasoning tokens fill the context window and inflate the KV cache. Thinking, for an LLM, is paid for in memory and latency.

Scaling laws / Chinchilla

Empirical relationships between compute, parameters and tokens: for a given compute budget there's an optimal ratio (~20 tokens per parameter, "Chinchilla"). "Overtraining" = going well beyond on purpose, because a small model trained for a long time costs less at inference forever. It's a CAPEX/OPEX trade-off, and in 2026 inference OPEX dominates.

Level 5 — Running the model (inference and serving)

So far: how the model learns. Now: how you serve it. It's the level missing from almost every glossary, and the one you touch hands-on the day you try to run a model on your own hardware.

Quantization

The bf16 from before is training stuff. At inference you can go lower still: weights at 8 or 4 bits (sometimes less). A 7B in fp16 wants ~14 GB of VRAM; at 4 bits it wants ~4 — often the difference between "it runs" and "it doesn't" on your card. The names you'll meet: GPTQ and AWQ (they quantize the weights only, calibrating on the corpus to lose less quality), GGUF (llama.cpp's format, with its mixed-precision k-quants). The trade-off is always the same: fewer bits, less memory, a sliver less quality — and below 4 bits the sliver becomes a rope.

Prefill vs decode

Inference has two phases with opposite profiles. Prefill processes the whole prompt at once, in parallel: it's compute-bound (lots of multiplications; the first token comes out). Decode generates one token at a time, and for each it must reread the entire KV cache: it's memory-bandwidth-bound — you're not waiting on FLOPs, you're waiting on memory. It's why two models with the same FLOPs can generate at wildly different speeds, and why optimizing inference is 90% optimizing memory movement. Everything else in this level follows from here.

PagedAttention and continuous batching

The two ideas that made modern serving (vLLM) efficient — and for a sysadmin they're home turf. PagedAttention: instead of allocating each request's KV cache as a contiguous block (which fragments and wastes memory, like a naive allocator would), it splits it into non-contiguous pages with a translation table. It's virtual memory applied to the KV cache, full stop. Continuous batching: instead of waiting for all the requests in a batch to finish before accepting new ones, you swap them in hot — as soon as one sequence ends, another takes its place and the GPU stays full. Together they're worth an order of magnitude of throughput over serving one request at a time.

Speculative decoding

A trick to speed up decode without changing the output. A small, fast draft model proposes the next N tokens; the big model verifies them all in a single forward pass (verifying is parallel, generating is sequential), accepts the correct prefix and discards from the first error on. If the draft guesses often — and on predictable text it does — you go 2-3× faster with exactly the same distribution as the big model. Zero quality trade-off, just less latency.

Inference metrics

The numbers you judge a serving setup by. TTFT (time to first token): how long until it starts answering — dominated by prefill. Generation speed (tokens/s per request): dominated by decode. Throughput (aggregate tokens/s across all requests): what matters if you serve many. And the key trade-off, throughput vs latency: big batches = more throughput but more wait for the single user; small batches = the opposite. The same dilemma as any queueing system.

Special tokens and chat template

A base model completes text; an instruct model expects a structure. Special tokens are vocabulary tokens that aren't words but markers: begin/end of sequence, and the role delimiters (<|im_start|>user, <|im_start|>assistant…). The chat template is the schema — usually a Jinja string that ships with the model — that packs your messages into exactly the form seen during SFT. Getting the template wrong, or forgetting it, is the number-one cause of local models that "answer weird": it's not the model being dumb, it's that you're speaking to it in a dialect it never learned.

And now?

If you got this far reading straight through, you have the map. But a map isn't the territory, and the real leap isn't made by any glossary: it's made by Andrej Karpathy's "Neural Networks: Zero to Hero" video series (YouTube, free). In particular the first two videos (micrograd and makemore) build backpropagation and a tiny language model by hand, and "Let's build GPT" builds attention line by line. They're ~15 hours total, and worth more than anything written — this document included.

One last thing, in the spirit of honesty: the "it resonates but doesn't quite land" feeling is the normal and correct state after a first read. These concepts aren't understood by reading — they're understood the third time your training diverges and you find out why. The glossary only exists so that, when it happens, you know where to look.