What you'll learn
- How a single token travels through a modern GPT — from the entrance embedding all the way to the next-token guess.
- What each modern refinement actually does: RoPE, RMSNorm, sliding-window (SSSL) attention, value embeddings, RELU²/SwiGLU, residual lambdas, and untied weights with softcapping.
- Why these changes make today's models far more efficient than GPT-2 — without giving up quality.
Introduction
I've been trying to understand modern GPT architectures but because of highly specialized terms, the concepts weren't sticking. So I've decided to learn it using an analogy and creating a story out of it, so that I can visualize what actually is happening under the hood, and this visualization will help make the concept concrete and shape a strong mental model. Writing this article is not only about understanding the concepts deeply myself but also this analogy driven strategy might help someone else to have a strong mental model, and this will serve me as a reference material in the future as well 😉.
Before I start to introduce the analogy it'll be beneficial to understand the original GPT-2 architecture (even though I'll give references to where GPT-2 architecture was improved by the modern refinements), and for that I highly recommend the book "Build a large language model from scratch by Sebastian Raschka". It's a great resource to understand what makes the GPT click and the idea is so simple i.e. generate the next token, but the underlying machinery is sophisticated enough which makes all of this magic happen. Although the analogy perfectly fits the older models as well, understanding the legacy first will fully prepare you and let you appreciate what came next and how all the refinements and efficiency of modern architectures have made the GPT model much more efficient and better. It's like if the GPT-2 was the regular production car then the modern GPT architecture is the F1 formula racing car because of the clever techniques that researchers have discovered and replaced all the parts which were making it slow. It's mostly all about achieving the highest possible efficiency and getting the most out of GPUs.
The key part of this GPT architecture is "Transformer Blocks" which basically gives us the capability to define our contextual truth and GPT model is all about going from "Base Truth" to "Contextual Truth" to "Unified Truth" where:
Base Truth = The static, frozen meaning a word has in isolation (The Master Vocabulary Table blueprint).
Contextual Truth = How that meaning changes based on its neighbors (The Attention Layer handshakes).
Unified Truth = The final, blended prediction of what comes next (The Output Head).
So, let's try to understand transformers by first having a little peek at the history of why the transformers were actually needed.
The Problem
Before 2017, sequence models (RNNs, LSTMs) processed text one token at a time, each step depending on the previous one's hidden state. This caused two problems:
Step 10 cannot be computed until step 9 is done, so training was slow and didn't scale on GPUs where parallelization is the real benefit.
Information about token 1 had to survive 100 sequential updates to influence token 100, so it got diluted and eventually vanished.
Attention was already invented actually as a patch for RNNs (in seq2seq translation); it let the decoder "look back" at the encoder states directly (and that helped a lot) instead of relying on a single squashed hidden vector which lost the important contextual truth, so no matter if we had 10 or 100 words, it was the same fixed size, and long sentences lost information.
The Solution
The 2017 insight (Vaswani et al.) from the paper "Attention Is All You Need" dropped the recurrence entirely and only kept the attention.
If every token can attend to every other token in one layer at the same time, we essentially solved both the problems mentioned above where no degradation over distance and the computation for the whole sequence can be done in parallel. This parallelization is really the unlock — it's what made training on internet-scale data possible.
The Analogy
To visualize the whole journey of a token, we need to look at it from a different perspective and in this case we'll be imagining tokens as Lego Blocks. We'll be analyzing their journey through the story i.e. "The Tower" where these Lego blocks can be considered as its visitors.
Each piece in our "Lego block" analogy (RoPE, sliding-window attention, value embeddings, untied output weights, etc.) is a technique used across many modern models — LLaMA, Mistral, Qwen, and others mix and match these. nanochat is a good example to look at because it's small enough to read end-to-end and combines several of these pieces in one place, and it's what we'll be discussing mostly in our story.
The Click
Before going towards our story let's analyse what made GPT click. GPT takes just the decoder part of the transformer, with masked self attention where each token can only attend to tokens before it and never after — this is the most important rule. Let's visualize this in figure 1 below to make this concept concrete:
Figure 1. brown (pos 3) turns around and can look back at "The", "quick". It can't see anything toward the tail — those words haven't arrived yet.
Stack this mechanism and train it for one simple objective: predict the next token!
This click is essentially a combination of 3 inter-related things:
A self-supervised strategy where expensive labeled data is no more required and the whole internet now becomes the training data.
An architecture that scales smoothly i.e. more layers, more parameters, more data reliably yields better performance (scaling laws).
Dynamic, context-dependent representations i.e. the contextual truth means the token's meaning is recomputed based on everything around it.
Introducing Story — The Tower (Our GPT model)
It's a multi-story building where we'll be visualizing the journey of the visitors which is our Lego blocks. Let's meet our cast of this story:
- The (head, position 1)
- Quick (position 2)
- Brown (position 3)
- Fox (tail, position 4)
Each chapter in this story should be read from top to bottom to understand the architecture coherently: how a block (token) enters, how it gathers context, how it consults stored knowledge, how it keeps from forgetting itself and how the next block is finally chosen.
Chapter 1 — The Entrance (Badges, smoothing, and the twist)
A bare Lego block arrives at the gate of the tower. At the gate, the doorman assigns it a position badge and from its book (the master vocabulary table) hands the block a vector strip which is basically a row of numbers that currently holds the block's plain context-free identity which is its base truth. Throughout this whole journey nothing will change the block, it'll exit with the same raw identity as it entered the tower, on the other hand the vector strip will be enriched with the contextual truth and based on the last block's vector strip the next block will be chosen i.e. the unified truth.
After the vector strip is attached to the block, its rough edges are filed smooth and then the block joins the queue. This smoothing technique is also special in modern architectures. Unlike them, GPT-2's LayerNorm smoothing was expensive because of the two steps involved:
- Re-center: find the average of the numbers and subtract it, so they now sit around zero.
- Re-scale: measure how spread out they are (the standard deviation) and divide by it, so the spread is standardized.
In simple words making the mean 0 and variance 1.
RMSNorm (Root Mean Square Normalization) on the other hand only involves a single step:
- Re-scale: divide the numbers by their root-mean-square. No averaging, no subtracting turns out to be the goldilocks zone and it's all we need i.e. the resizing step.
Before the block proceeds it's important to define the position where it exists in the given sequence, without this information the model won't be able to differentiate between "the cat eats the mice" and "the mice eat the cat", they will be treated as an unordered pile of bricks. In GPT-2 to embed the positional signal of the block a second vector strip was bolted on top but modern architectures use a clever trick called RoPE (Rotary Positional Encoding) where we twist the strip's numbers in 2-D pairs on a certain angle calculated based on position of the block as shown in figure 2 below:
Figure 2. The figure shows the RoPE for d_model = 8 → 4 dial pairs for each token. Position 0 ("The") gets no twist. Reading down a column: the fast dial (pair 0) sweeps 0° → 172°, clearly separating positions, the slow dial (pair 3) barely moves. Many speeds let one strip encode both near and far positions at once.
One important point to take into consideration is that RoPE is not applied directly onto the vector strip (an improvement from GPT-2) but on cards which are derived from the vector strip, known as Q (Question card - what am I looking for) and K (Key or label card - what I've to offer) cards, when the block enters into the Attention hall (more on these halls in chapter 2). These cards are torn apart when the block climbs a floor and derived again from an even richer vector strip on the next floor, where the angle of twist will remain the same. This will happen on all floors of the tower as shown in Figure 3.
Figure 3. The strip travels straight up the dashed spine, never twisted, only gaining notes. At each attention hall it spawns fresh Question/Key cards that get the position twist, are compared, then thrown away. The twist on Hall 2 equals the twist on Hall 1.
Chapter 2 — The Message Halls (Looking back, the window, and the relay (SSSL — Short, Short, Short, Long))
The queue continues to climb the Tower. Every floor has a Message Hall where blocks share notes, under one rule that never bends: look only backward, towards the head of the line.
A block compares its Question card against the Label cards of earlier blocks to decide who to listen to, then copies the chosen content into its vector strip enriching it with context and moving closer towards the "contextual truth".
Reading every earlier block is expensive on long lines, and assuming our window size is 3 (it could be 1024 and above in production models) where most halls are Short: a block may only read its nearest couple of neighbours i.e. current + 2 behind.
SSSL specifically represents the sliding window attention and every fourth hall is a Town hall ("Long"), where the limit lifts and a block can look all the way back to the head. Three short, one long — the S.S.S.L rhythm — repeats up the whole stack.
In simple words, we are trying to get the best of both worlds. Short halls pick up the context through a sliding window with no need to look all the way back (this can also be described as "The Relay"), while town halls give them the opportunity to exchange notes all the way back to the head and enrich their vector strips — keeping the architecture efficient at the same time throughout this whole journey as well.
The Relay can simply be described as follows: a block in a short hall can only see two seats back (as we've selected the window size to be 3 here), yet the news from the head still arrives; handed forward like a bucket brigade. Once a near block absorbs "The", a further block reads that block on the next floor and inherits it secondhand. The window never grows but the carriers do. Depth is basically now substituted for reach. This concept is depicted in figure 4 below:
Figure 4. Tracing the word "The" up four floors. Arrows show each new block grabbing the news from a carrier within its 2-seat window; every arrow the same short length. The frontier advances two seats per short floor, then the town hall delivers it to everyone at once.
Chapter 3 — The Secret Backpack (Re-supplying raw identity (Value Embeddings))
I've mentioned previously that at each hall the block derives two cards from its vector strip i.e. Q and K and then applies the twist based on the calculated angle, in reality the block actually makes three cards i.e. Q (Question), K (Label) and V (Value) where V card is special and the twist will not be applied on this particular card which is basically can be thought of as a real parcel of content that changes hands.
After a dozen floors of rewriting and enriching the vector strip the plain fact, "this block is literally the word fox" can fade out, so on alternating halls a block is handed a backpack which is a packet looked up directly from a second ledger keyed by its word; its raw, original identity, untouched by any floor.
Now here's an interesting part, how much of this identity should be poured is also learned by the model and a volume knob (a sigmoid gate) decides how much of that to pour into the value parcel.
From a compute perspective the effort is negligible because it's just a lookup and in return we get an enormous recall boost for near zero compute!
To visualize this concept have a look at the following figure 5:
Figure 5. Three cards are derived from fox's strip; Q and K are twisted, V is not. A backpack (raw "fox" keyed lookup) is metered by the gate and blended into V. The backpack appears only on alternating halls.
Chapter 4 — The Deep-Thinking Room (Consulting stored knowledge (RELU²))
On every floor, after the block passes the message hall, it enters alone into a private room (the feed-forward layer). No other block is allowed to join it here. The walls in the room can be thought of as giant switchboard of concept-detectors the model learned in training ("is this an animal?", "is a verb likely next?").
The block holds its vector strip up and every switch in the room scores how well it matches with the strip.
RELU² can be thought of as wiring in this room which works on two rules:
- Any switch that doesn't score positively is killed to exactly zero.
- Any switch that does fire has its volume "squared", so confident matches become more prominent and timid ones shrink.
In GPT-2, GELU's soft wiring let a faint murmur through, which was a bit compute intensive because even if a concept has minimum chance of a match it passes through, meaning more metrics manipulation. RELU², on the other hand, is a hard switch for negative ones, which means many switches will be completely off (zeros) while matched ones are amplified. These zeros make the lookup cheap and hence make the activation cheaper than GELU.
These activations can be visualized from the following figure 6 for fox's block:
Figure 6. Top: a switchboard at one input. Negative scores are off; positive ones glow at score². Bottom: the activation curves — RELU² is dead-flat left of zero, then quadratic, so weak matches are suppressed and strong ones dominate.
At the end the lit switches pour their stored fact into the block's vector strip and make it richer on the factual front as well.
The Duality. The Message Hall mixes context from neighbors; the Private Room processes each block alone (knowledge from the model's weights). This happens on every floor, and the strip comes out richer on both axes.
Some modern architectures also use SwiGLU activation which can buy them more quality but at the cost of more compute as it introduces more parameters. It's important that we discuss SwiGLU briefly here as well because it's one of the favorite choices of large models.
SwiGLU changes the wiring itself, it adds a second control that RELU² doesn't have, to understand SwiGLU let's compare it with RELU²:
- RELU² — one path, a switch in it. A detector scores the strip and that score is passed through the switch.
Figure 7. one path — the score decides its own volume.
-
SwiGLU — two paths, one gates the other. Each neuron computes two separate numbers from the strip instead of one:
- A content number — here's the value I'm contributing
- A gate number — how much of that value will pass through
The gate is passed through a soft S-shaped squashing function (called SILU/Swish, a smooth cousin of the sigmoid), turning it into a soft volume knob. Then the two numbers are multiplied to get the final signal:
Final output = content × gate-knob
So a neuron can compute a strong content but its own gate can turn it down or vice-versa. The final output is basically a negotiation between two signals.
Figure 8. two paths — content × gate-knob, computed separately then multiplied.
Let's try to rephrase this comparison based on our analogy, RELU²'s switchboard has one operator per switch who decides on/off-and-volume from the match alone and SwiGLU's switchboard gives each switch two operators, where one proposes the content and the other holds the knob; their outputs are multiplied before the facts are poured into the strip. Same room, same purpose (retrieve the facts), but a more expressive control mechanism on each switch.
Chapter 5 — The Backbone (Never forgetting the original (Residual Lambdas))
At the entry gate the model quietly makes the master copy (we can call it x₀) of the original vector strip and slides that into the pocket of the block which never changes throughout the journey.
This contains the state of the block at floor zero i.e. its context-free meaning with its position. Now, why do we need it? The attached vector strip will get enriched on every floor i.e. it'll be overwritten many times, and the original identity will eventually become a whisper at depth — so having the original strip state from floor zero helps the block maintain this identity between floors.
A common confusion might occur here as I've mentioned in chapter 4 that on alternate floors a backpack is given to the block that contains its raw identity but there's a distinction in what each one carries:
- Backpack: the raw identity of the word, local to attention, looked up from a table (by the way it's not the master vocabulary table, it's another table that exists specifically for this purpose to inject the identity at depth also known as value embeddings) by the block — "this slot is literally fox". It's a fresh vector, born at the lookup time, not something the block arrived with.
- Pocket copy: the block's whole entry strip — The full vector it was handed at the gate, including its position and its context-free meaning. It's the block's own original state, sealed and carried up.
So the real difference: the backpack is "what word am I", the pocket copy is "what was my entire strip when I started the journey", one is a dictionary lookup; the other is a snapshot of this specific block at floor zero.
On every floor there's a mixing desk with two dials that are also learned per floor:
- Dial 1: how much of the current enriched strip to keep
- Dial 2: how much of the sealed original to fold back in fresh
The strip carried upward is a blend of both. A normal skip connection in GPT-2 reaches back one floor whereas this reaches all the way down to floor zero, every floor.
Figure 9. How much of the original survives across 12 floors. Plain addition lets it decay toward nothing; re-mixing x₀ holds it at a plateau set by λ₀ (here = 0.3), deep into the tower (Curve illustrative).
Chapter 6 — The Finish Line (Reading of the guess (Untied Weights + Softcap))
At this point the block's vector strip is enriched with contextual truth and factual knowledge as well, and since fox is at the tail, its strip is the one that'll name the next block.
Fox hands its strip to the Examiner, who scores every word in the vocabulary.
Untied weights: GPT-2 reused the doorman's book at the exit (run backwards) to save space. nanochat gives the Examiner a separate book, trained only for predicting.
Why do we have two books? because they serve different purposes, turning a word into vector strip and reading a strip back into a word are different jobs where the Examiner's book will allow the model to make sharper guesses but at the cost of extra parameters, a tradeoff we have to make here for quality.
Logit softcapping: before the scores become a vote, each passes through a "no shouting" filter i.e. 15·tanh(score/15). Quiet scores pass untouched; loud ones get squashed so none can exceed ±15. No single word can fake 100% certainty and crush the alternatives — the final distribution stays honest.
Figure 10. Two separate books (left). Then the same set of scores as probabilities: with no cap one word grabs nearly everything; softcapped, it stays confident but leaves room for the rest. Bottom: 15·tanh keeps any score within ±15.
Conclusion
Hopefully you've enjoyed reading this article as much as I've enjoyed writing it and it has also enabled me to go deeper into the concepts and explore various other dimensions. I hope you now have a stronger mental model on how to visualize the concepts involved in modern GPTs and this approach will definitely help in understanding what comes next in this space as the analogy will perfectly fit for that as well.
Now it's time to see the actual names of our concepts, and the following table will show you the mapping:
Lego analogy → real architecture
| Lego analogy | Technical term | What it actually is |
|---|---|---|
| THE ENTRANCE | ||
| Lego block | token | One unit of text (a token ID) from the tokenizer's fixed vocabulary. |
| Vector strip / notebook | residual stream | The per-token hidden-state vector that flows through every layer; starts as the token embedding. |
| Doorman's book | input embedding (wte) | The master vocabulary table mapping each token ID to its starting strip. |
| Badge / place in queue | position index | The token's position in the sequence. |
| The twist on the cards | RoPE | Rotary positional encoding — rotates Q and K in 2-D pairs by a position-based angle. |
| Filing edges smooth | RMSNorm | Normalization applied before each sublayer to keep magnitudes stable. |
| THE MESSAGE HALLS (ATTENTION) | ||
| The queue, backward-only | causal self-attention | Each position may attend only to itself and earlier positions (the causal mask). |
| Message Hall | attention sublayer | The block where tokens exchange information. |
| Question card | Query (Q) | What this token is looking for (twisted by RoPE). |
| Label card | Key (K) | A searchable tag others match against (twisted by RoPE). |
| Value parcel | Value (V) | The actual content delivered when attended to (not twisted). |
| Comparing cards → who to read | attention scores | softmax(Q·K) weights over visible tokens. |
| Short hall (2 neighbours) | sliding-window attention | Local attention restricted to a fixed recent window (1,024 in nanochat). |
| Town hall | global / full attention layer | A layer with no window — attends back to the start. |
| S·S·S·L rhythm | SSSL layer pattern | Three sliding-window layers then one global, tiled up the stack. |
| The relay / bucket brigade | effective receptive field | Information spreading across stacked local layers — depth substitutes for window size. |
| The backpack | Value Embeddings (value_embeds) | A separate per-token table injected into the Value path on some layers. |
| Volume knob / gate | learned sigmoid gate | A trained scalar controlling how much raw identity is blended into V. |
| THE DEEP-THINKING ROOM (MLP) | ||
| Private Room | feed-forward / MLP sublayer | Per-token processing; no mixing between tokens. |
| Switchboard of concepts | MLP hidden neurons | Up-projection "keys" detecting patterns (key-value-memory view). |
| Hard-off, squared switch | ReLU² activation | relu(x)² — zeros non-positive inputs, squares positive ones. |
| Stored facts poured in | MLP down-projection | "Values" written back into the strip from fired neurons. |
| THE BACKBONE | ||
| Adding notes (not overwriting) | residual connection | Each sublayer's output is added to the strip (skip connection). |
| Sealed pocket copy (x₀) | initial embedding x₀ | The original input strip, preserved for re-mixing. |
| Mixing desk / two dials | residual lambdas | Learned per-layer scalars blending the current strip with x₀. |
| The Tower / floors | transformer blocks (the stack) | The repeated layers; each = attention + MLP. |
| THE FINISH LINE | ||
| The Examiner | output head | The final stage that turns the top strip into next-word scores. |
| Examiner's separate book | untied lm_head | An independent output projection (not the input embedding reused). |
| Scoring every word | logits | One raw score per vocabulary entry. |
| The no-shouting rule | logit softcapping | 15·tanh(logits/15) — bounds logits to ±15. |
| The final vote → fox's word | softmax + sampling | Logits → probability distribution → the chosen next token. |
Top comments (0)