The Transformer diagram looks simple until you try to follow what actually happens to one token inside it.
You see embeddings, attention, Q, K, V, Add & Norm, FFN, another attention block, logits, and finally some probabilities.
Then you look at GPT and notice something even more confusing:
Where is the Encoder?
The original Transformer has an Encoder and a Decoder.
GPT doesn't.
Yet GPT is built on the Transformer architecture.
So rather than treating these as separate topics, it makes more sense to follow the architecture from the bottom up and see how the pieces connect.
1. The Original Transformer: Encoder + Decoder
The Transformer was introduced in the 2017 paper "Attention Is All You Need."
The original architecture was designed for sequence-to-sequence tasks such as translation.
At a high level:
Input sequence
↓
Encoder
↓
Encoded representation
↓
Decoder
↓
Output sequence
For example:
"I love programming."
↓
Encoder
↓
contextual representation
↓
Decoder
↓
"J'aime programmer."
The Encoder's job is to process the input and build useful contextual representations.
The Decoder's job is to use those representations to generate the output.
This distinction matters because modern Transformer models don't all use both sides.
There are three common configurations:
Encoder-only
Used primarily for understanding or representation.
Example: BERT
Encoder–Decoder
Used for transforming one sequence into another.
Examples: the original Transformer, T5
Decoder-only
Used primarily for autoregressive generation.
Examples: GPT-style models
So "Transformer" describes the architecture family, not one fixed model design.
2. Text Becomes Tokens, Then Vectors
Before the Encoder or Decoder can do anything, text has to become numerical data.
Suppose the input is:
"Translate this sentence into French."
A tokenizer breaks it into tokens:
["Translate", " this", " sentence", " into", " French", "."]
The exact tokenization depends on the tokenizer.
A token can be a complete word, part of a word, punctuation, or another frequently occurring piece of text.
Those tokens are then mapped to integer IDs:
[18472, 351, 9281, 417, 6321, 13]
But the IDs themselves don't contain meaning.
They're simply indexes into the model's vocabulary.
So we need another step:
Token ID
↓
Embedding
↓
Vector
A token such as "cat" might become a vector like:
[0.21, -0.73, 0.42, 0.18, ...]
In a real model, that vector can contain hundreds or thousands of values.
This is the point where the model stops dealing with discrete symbols and starts working with continuous numerical representations.
3. Position Matters
There's another problem.
A Transformer needs to know the order of the tokens.
Consider:
Dog bites man.
and:
Man bites dog.
The same basic words are present, but the meaning is obviously different.
Attention alone doesn't inherently tell the model that one token came before another.
That's why Transformers use positional information.
In the original Transformer, this was done with sinusoidal positional encodings.
Modern architectures often use other approaches, such as Rotary Positional Embeddings (RoPE).
The exact implementation can vary, but the purpose is the same:
Give the model information about where tokens occur and how their positions relate to one another.
Conceptually:
Token embedding
+
Position information
↓
Position-aware representation
Now the sequence is ready to enter the Transformer blocks.
4. Self-Attention
Now we get to the part that made Transformers such a big deal.
Consider:
"The animal didn't cross the road because it was tired."
When the model processes "it", information about "animal" may be useful.
The model needs a way to decide which other tokens matter to the current token.
That's what self-attention does.
A useful mental model is:
Each token can look at the other tokens in the sequence and determine which ones are relevant to its current representation.
And this is where Query, Key, and Value come in.
5. Query, Key, and Value
The names sound more complicated than they really are.
Suppose the input representations are stored in a matrix:
X
The model uses three learned weight matrices:
W_Q
W_K
W_V
to calculate:
Q = XW_Q
K = XW_K
V = XW_V
So Q, K, and V aren't three magical objects attached to every word.
They're learned projections of the current token representations.
A useful intuition is:
Query
What information am I looking for?
Key
What information do I contain?
Value
What information should I provide if I'm relevant?
That gives us a way to compare a token with the rest of the sequence.
6. Attention Scores
Now we need to measure how strongly tokens should interact.
This starts with:
The dot product between a Query and a Key gives us a compatibility score.
A larger score means the two representations are more aligned.
A smaller score means they are less aligned.
You can think of it conceptually as:
Current token
↓
Query
↓
compare with
↓
Keys from other tokens
↓
attention scores
Suppose a token produces scores like:
The → 0.2
animal → 2.4
road → 0.7
tired → 1.1
At this stage, these are just raw scores.
We still need to turn them into useful weights.
7. Scaling Before Softmax
The full scaled dot-product attention equation is:
That division by:
isn't there for decoration.
As the dimensionality of the vectors grows, the dot products can also grow in magnitude.
Very large values can make Softmax extremely sharp, which can make optimization difficult.
Scaling keeps the scores in a more useful numerical range before Softmax is applied.
So the pipeline is:
QKᵀ
↓
Scale
↓
Softmax
↓
Attention weights
Each step exists for a reason.
8. Softmax Turns Scores Into Weights
Softmax converts the attention scores into a normalized distribution.
For example:
The → 0.06
animal → 0.60
road → 0.11
tired → 0.23
The values sum to approximately 1.
Now we can use them as weights when combining the Value vectors.
So the model is effectively giving different amounts of influence to different tokens:
The → small influence
animal → large influence
road → small influence
tired → moderate influence
This is a useful intuition, although we shouldn't take it too literally as the model "thinking" about words.
9. The Attention Output
Once we have the attention weights, we use them to calculate a weighted combination of the Value vectors.
Conceptually:
Attention weights
×
Value vectors
↓
New representation
If "animal" has a large attention weight, its Value contributes more to the resulting representation.
The result is a new representation that contains information gathered from the surrounding tokens.
That is the important idea behind self-attention:
A token can build a new representation using information from other tokens.
10. Multi-Head Attention
One attention mechanism gives the model one learned way to look at relationships.
Transformers use Multi-Head Attention, which runs several attention mechanisms in parallel.
Conceptually:
Input
│
┌───────────┼───────────┐
↓ ↓ ↓
Head 1 Head 2 Head 3 ...
↓ ↓ ↓
└───────────┼───────────┘
↓
Concatenate
↓
Projection
Each head has its own learned projections.
We don't manually tell one head:
"You are responsible for grammar."
and another:
"You are responsible for pronouns."
The model learns useful patterns during training.
Different heads can capture different relationships in the sequence.
Afterward, the outputs are concatenated and passed through a learned output projection.
11. What Does Projection Mean?
You'll see the word projection constantly in Transformer implementations.
In this context, it usually means applying a learned linear transformation.
For example:
X
↓
W_Q
↓
Q
is a projection.
Likewise, after multiple attention heads:
Head outputs
↓
Concatenate
↓
Output projection
↓
Updated representation
And later:
Hidden representation
↓
Vocabulary projection
↓
Logits
So "projection" sounds more exotic than it really is.
A lot of Transformer computation is ultimately learned matrix multiplication.
12. Residual Connections
Attention produces a transformed representation.
But the Transformer doesn't simply throw away the original input.
It uses residual connections.
Mathematically:
The original representation is added back to the transformed representation.
Why?
Because Transformers can contain many layers.
Residual connections give information and gradients a shorter path through the network, which makes deep networks much easier to optimize.
A useful intuition is:
Keep what we already have, then add what this layer learned.
13. Layer Normalization and Add & Norm
You'll often see Transformer diagrams containing:
Add & Norm
This is shorthand for a residual addition together with Layer Normalization.
A simplified flow is:
Input
│
├──→ Attention ──┐
│ ↓
└──────────────→ Add
↓
Norm
Layer Normalization helps keep the numerical activations in a more stable range as the representation passes through many transformations.
Pre-LN vs Post-LN
The exact placement of LayerNorm can differ between Transformer architectures. Two common arrangements are Post-LN and Pre-LN.
Post-LN
In Post-LN, the sublayer runs first, the residual connection is added, and LayerNorm is applied afterward:
x → Attention → Add → LayerNorm
Or more explicitly:
x + Attention(x)
↓
LayerNorm
Pre-LN
In Pre-LN, LayerNorm is applied before the sublayer:
x → LayerNorm → Attention → Add
Or:
x
↓
LayerNorm
↓
Attention
↓
+ x
The same ordering applies to the FFN (Feed-Forward Network) sublayer as well.
The distinction matters because the placement of normalization affects how easily the Transformer can be optimized. Pre-LN is common in many modern Transformer architectures because it tends to make optimization easier, especially for deeper models.
So when you see:
Attention → Add → Norm
versus:
Norm → Attention → Add
you're not looking at two completely different ideas.
They're simply two different ways of arranging the same basic components.
The important part is understanding the roles:
Residual connection → preserve a direct path through the network.
Layer normalization → stabilize the representations.
14. The Feed-Forward Network
Attention isn't the entire Transformer block.
After attention, the representation goes through a Feed-Forward Network (FFN).
A simplified version looks like:
Representation
↓
Linear transformation
↓
Non-linear activation
↓
Linear transformation
↓
New representation
Mathematically, something like:
A useful distinction is:
Attention lets tokens exchange information.
FFN transforms each token's representation using learned nonlinear transformations.
So conceptually:
Attention asks, "What information from elsewhere matters?"
FFN asks, "Given that information, how should this representation change?"
15. One Transformer Block
Now we can finally put the pieces together.
A simplified Transformer block looks like:
Input
↓
Self-Attention
↓
Residual + Norm
↓
FFN
↓
Residual + Norm
↓
Output
The exact ordering varies between implementations, but the major components remain recognizable.
And one block isn't enough.
16. Stacking Transformer Blocks
A Transformer stacks many blocks:
Input
↓
Transformer Block 1
↓
Transformer Block 2
↓
Transformer Block 3
↓
...
↓
Transformer Block N
The representation is repeatedly transformed.
It's tempting to say:
"The first layer learns grammar, the next layer learns facts, and the next layer learns reasoning."
That's too simplistic.
The model's learned information is distributed across many layers and parameters.
A better way to think about it is:
Each layer repeatedly transforms the representations, allowing increasingly useful and contextual patterns to emerge.
17. The Encoder
Now we can look at the Encoder itself.
In the original Encoder–Decoder Transformer, the Encoder is a stack of Transformer blocks using self-attention without a causal restriction.
If the input is:
"The cat is sitting on the mat."
the Encoder can allow every input token to attend to the other input tokens.
Conceptually:
"The" ↔ "cat" ↔ "is" ↔ "sitting" ↔ ...
This is often described as bidirectional self-attention.
The Encoder can see the complete input because its job isn't to generate the output token by token.
Its job is to build rich contextual representations of the input.
So the Encoder is roughly:
Input tokens
↓
Embeddings + positional information
↓
Transformer blocks
↓
Contextual representations
18. The Decoder
The Decoder has a different problem.
It is responsible for generating the output sequence.
And generation is autoregressive.
That means the Decoder can't simply look at everything in the target sequence, because future tokens don't exist yet during generation.
So the Decoder uses masked self-attention.
Suppose we're generating:
The cat sat on the mat
When predicting "mat", the model must not be allowed to use "mat" itself or anything after it.
The attention pattern is therefore causal:
T1 T2 T3 T4
T1 ✓ ✗ ✗ ✗
T2 ✓ ✓ ✗ ✗
T3 ✓ ✓ ✓ ✗
T4 ✓ ✓ ✓ ✓
Each token can attend to itself and previous tokens, but not future tokens.
That's why this is called causal masking.
19. The Decoder Has Two Different Attention Mechanisms
This is the part that makes the Encoder–Decoder architecture much easier to understand.
The Decoder needs to use:
- Its own previously generated tokens.
- The information produced by the Encoder.
The first is handled by:
Masked Self-Attention
The second is handled by:
Cross-Attention
So a simplified Decoder block looks like:
Previous output tokens
↓
Masked Self-Attention
↓
Cross-Attention ← Encoder output
↓
FFN
↓
Output representation
These two attention mechanisms have different jobs.
20. Self-Attention vs Cross-Attention
For self-attention, Q, K, and V come from the same sequence.
In Encoder self-attention:
Q ← Encoder representations
K ← Encoder representations
V ← Encoder representations
In Decoder self-attention:
Q ← Decoder representations
K ← Decoder representations
V ← Decoder representations
The Decoder's self-attention is masked, so it can only use previous positions.
Cross-attention is different:
Q ← Decoder representation
K ← Encoder output
V ← Encoder output
That means the Decoder is effectively asking:
"Given what I have generated so far, which parts of the encoded input should influence my next prediction?"
This is the bridge between the Encoder and Decoder.
21. Translation Makes the Difference Obvious
Take:
"I love programming."
and translate it into French.
The Encoder reads the whole English sentence:
"I" "love" "programming"
and produces contextual representations.
The Decoder then starts with a special beginning-of-sequence token and begins generating:
<START>
↓
J'aime
↓
programmer
↓
<END>
At every step, the Decoder can use two sources of information.
Its previously generated tokens:
masked self-attention
and the original English input:
cross-attention
So while generating "programmer", the Decoder isn't working from the word "programmer" out of nowhere.
It can use the Encoder's representation of the original input.
That's the purpose of the Encoder–Decoder design.
22. Training With Teacher Forcing
Now we need to separate training from generation.
During training, we already know the correct output.
Suppose the target is:
"I love programming."
The Decoder can be trained with the correct previous tokens:
<BOS> → I
<BOS> I → love
<BOS> I love → programming
This is called teacher forcing.
Instead of feeding the model its own prediction at every step during training, we give it the correct previous token and ask it to predict the next one.
That makes training much more manageable.
Without teacher forcing, one early incorrect prediction could become part of the next input and cause the errors to compound.
23. Training: Prediction, Loss, and Backpropagation
During training, the model makes a prediction for the next token.
We already know what the correct token should have been.
So we compare the prediction with the target using a loss function such as cross-entropy.
Suppose the correct token is:
Paris
but the model predicts:
Paris → 0.20
London → 0.30
Rome → 0.10
...
The model should increase the probability of "Paris".
Cross-entropy gives us a way to measure how bad the prediction was.
For a single target token, a simplified form is:
So:
P(correct) = 0.90
→ small loss
P(correct) = 0.01
→ large loss
That loss is then used by backpropagation to calculate gradients.
The gradients tell the optimizer how the model's parameters should change.
Conceptually:
Prediction
↓
Loss
↓
Backpropagation
↓
Gradients
↓
Update parameters
This happens through the entire network.
The goal is to make future predictions better.
24. Training vs Inference
This distinction is worth keeping clear.
Training
Input + target
↓
Prediction
↓
Cross-entropy loss
↓
Backpropagation
↓
Update parameters
The weights change.
Inference
Input
↓
Transformer
↓
Logits
↓
Probabilities
↓
Choose token
↓
Append token
↓
Repeat
The learned weights are being used.
The model isn't learning new weights from your prompt during normal inference.
It's using what it already learned.
25. From Hidden Representation to Logits
After the Decoder processes the current context, we still don't have a word or token.
We have a hidden representation.
The model applies a final learned projection into the vocabulary space.
Conceptually:
Hidden representation
↓
Vocabulary projection
↓
Logits
If the vocabulary contains 100,000 tokens, the model can produce roughly 100,000 logits for the next token.
For example:
"the" → 8.2
"program" → 7.4
"Python" → 6.9
"banana" → 0.7
These are raw scores.
They're not probabilities yet.
26. Logits Become Probabilities
We apply Softmax:
Now the scores become a probability distribution.
Conceptually:
"the" → 0.42
"program" → 0.28
"Python" → 0.19
"banana" → 0.01
...
The model has now produced a probability distribution over possible next tokens.
Then the decoding process chooses one.
And this takes us directly to autoregressive generation.
27. Autoregressive Generation
Suppose GPT receives:
"Python is"
It predicts a probability distribution for the next token.
Maybe the selected token is:
" a"
Now the context becomes:
"Python is a"
The model runs again.
Maybe it predicts:
" programming"
Now:
"Python is a programming"
And so on.
Conceptually:
Current context
↓
Transformer
↓
Logits
↓
Probabilities
↓
Choose next token
↓
Append token
↓
New context
↓
Transformer again
↓
Repeat
This is autoregressive generation.
The model doesn't generate the entire paragraph in one step.
It generates one token, adds it to the context, and predicts the next token.
28. Temperature and Sampling
One final part of the generation process is sampling.
Temperature is not part of the Transformer architecture itself.
It is part of the decoding process.
Suppose the model produces logits:
A → 8.0
B → 7.0
C → 5.0
Changing the temperature changes how concentrated the resulting distribution becomes.
Lower temperature generally produces a sharper, more predictable distribution.
Higher temperature generally produces a flatter distribution and allows more variation.
Conceptually:
Low temperature
↓
Sharper distribution
↓
More predictable choices
High temperature
↓
Flatter distribution
↓
More variation
So temperature doesn't teach the model anything.
It changes how we sample from the model's predictions.
29. Where GPT Fits
Now the original question becomes much easier to answer.
The original Transformer was:
Encoder
↓
Decoder
GPT-style models are:
Decoder-only
That means GPT does not have a separate Encoder stack.
It also does not need the Encoder-to-Decoder cross-attention used in the classic Encoder–Decoder Transformer.
Instead, the prompt goes directly through a stack of decoder-style Transformer blocks using causal self-attention.
Conceptually:
Prompt
↓
Tokenization
↓
Embeddings + positional information
↓
Causal Self-Attention
↓
FFN
↓
Transformer block
↓
Transformer block
↓
...
↓
Logits
↓
Probabilities
↓
Next token
Then that token is appended to the sequence and the process repeats.
30. GPT vs the Original Encoder–Decoder Transformer
The difference can be summarized very simply.
Original Transformer
Input
↓
Encoder
↓
Encoder representations
↓
Decoder
↓
Output
The Decoder uses:
- masked self-attention
- cross-attention to the Encoder
- FFN
GPT-style Transformer
Prompt
↓
Decoder-style blocks
↓
Logits
↓
Next token
It uses:
- causal self-attention
- FFN
- residual connections
- normalization
- stacked Transformer blocks
but there is no separate Encoder and no Encoder-to-Decoder cross-attention.
That's why GPT is called decoder-only.
31. The Three Transformer Configurations
At this point, the three major configurations are easier to remember.
Encoder-only
Input
↓
Encoder
↓
Representation
Mental model:
Understand the input.
BERT is a well-known example.
Encoder–Decoder
Input
↓
Encoder
↓
Representation
↓
Decoder
↓
Output
Mental model:
Transform one sequence into another.
T5 is a well-known example.
Decoder-only
Prompt
↓
Decoder-style blocks
↓
Next token
↓
Next token
↓
Next token
↓
...
Mental model:
Continue the sequence.
GPT-style models belong here.
These are not the only possible ways to use Transformers, but they're three important architectural patterns to recognize.
32. The Complete Picture
At this point, the pieces fit together into one pipeline.
For a classic Encoder–Decoder Transformer:
Text
↓
Tokens
↓
Embeddings
↓
Positional Information
↓
Encoder
├── Self-Attention
├── Add + Norm
├── FFN
└── repeated × N
↓
Encoded representation
↓
Decoder
├── Masked Self-Attention
├── Cross-Attention
├── FFN
└── repeated × N
↓
Projection
↓
Logits
↓
Softmax
↓
Probabilities
↓
Next token
↓
Repeat
And for GPT-style decoder-only models:
Text
↓
Tokens
↓
Embeddings
↓
Positional Information
↓
Causal Self-Attention
↓
FFN
↓
Transformer Block × N
↓
Projection
↓
Logits
↓
Softmax / Decoding
↓
Next token
↓
Append token
↓
Repeat
The architecture becomes much easier to reason about once each component has a specific job.
33. The Part That Matters When Reading Transformer Diagrams
There are a lot of names to remember:
Q, K, V
Attention
Softmax
Masking
Multi-Head Attention
Projection
Residual Connections
Layer Normalization
FFN
Cross-Attention
Logits
Cross-Entropy
Backpropagation
Teacher Forcing
Autoregressive Generation
At first, they look like unrelated pieces of terminology.
They aren't.
They form a pipeline.
Tokenization gives us discrete pieces of text.
Embeddings turn those pieces into vectors.
Positional information tells the model where those tokens occur.
Q, K, and V create learned views of those representations.
Attention allows tokens to exchange information.
Masking controls which tokens are allowed to interact during generation.
Multi-Head Attention gives the model multiple learned ways to inspect relationships.
Residual Connections and Layer Normalization help those transformations work across deep networks.
FFNs further transform the resulting representations.
Encoder layers build contextual representations of an input.
Decoder layers use previous output tokens to generate the next ones.
Cross-Attention lets an Encoder–Decoder model connect those two sides.
Projection maps the hidden representation into the vocabulary space.
Logits give a score to every possible next token.
Softmax converts those scores into probabilities.
Cross-Entropy measures how well the model predicted the correct token during training.
Backpropagation computes how the parameters should change.
Teacher Forcing makes supervised sequence training practical.
And finally:
Autoregressive generation turns one predicted token into the context for the next prediction.
That's the Transformer.
Not one mysterious box.
Not a collection of AI buzzwords.
A sequence of numerical transformations where each component has a specific role.
And once that structure is clear, GPT stops looking like something completely different from the original Transformer.
It becomes what it actually is:
a decoder-only Transformer architecture built around causal self-attention and autoregressive next-token prediction.
Top comments (0)