Every large language model you use today, GPT, Claude, Gemini, Llama, is built on one core invention: the Transformer. It came from a 2017 Google paper called Attention Is All You Need. Before that paper, models read text one word at a time, in order, using recurrent networks (RNNs and LSTMs). That was slow and made it hard to connect words that were far apart in a sentence.
The Transformer removed recurrence entirely and replaced it with attention, a mechanism that lets every word look at every other word at the same time. That single change is why modern LLMs can be trained on massive datasets in parallel and why they handle long, complex text so well.
This article walks through the full pipeline, from raw text to predicted output, in the order data actually flows through a model.
The Big Picture
Before the details, here is the full journey a sentence takes through an LLM:
- Tokenization – break text into small chunks (tokens)
- Vector Embedding – turn each token into a list of numbers
- Positional Encoding – add information about word order
- Self-Attention – let each token look at other tokens for context
- Multi-Head Attention – run several attention patterns in parallel
- Feed-Forward Network – process each token's information further
- Add & Norm (residual connections) – stabilize and preserve information
- Stack layers (Encoder / Decoder) – repeat the above N times for depth
- Linear + Softmax – convert the final vectors into word probabilities
- Sampling / Generation – pick the next word and repeat
Everything below explains each of these steps in detail.
1. Tokenization: Turning Text Into Pieces
A model cannot read raw text. It needs numbers. The first step is to split a sentence into tokens, which are usually not full words. Modern LLMs use subword tokenization methods such as Byte-Pair Encoding (BPE) or WordPiece.
Example:
Input text: "unbelievable results"
Tokens: ["un", "believ", "able", " results"]
Token IDs: [4521, 8832, 219, 991]
Why subwords instead of whole words? Two reasons:
- Vocabulary size stays manageable. A model does not need a separate entry for every possible word, including rare ones, typos, or made-up words. It can build them from smaller pieces.
- Unknown words are never a dead end. Even a word the model has never seen can be broken into familiar subword pieces.
The original Transformer paper used Byte-Pair Encoding with a vocabulary of around 37,000 tokens for English-German translation, and word-piece encoding with a 32,000-token vocabulary for English-French. Today's LLMs use similar ideas but with larger vocabularies, often 50,000 to 100,000+ tokens.
Once tokenized, each token is mapped to a unique integer ID using a lookup vocabulary. That ID is what enters the model.
2. Vector Embeddings: Giving Tokens Meaning
A token ID like 4521 is just a label. It carries no meaning by itself. The next step converts each token ID into a vector embedding, a list of numbers (commonly 512, 768, 4096, or more dimensions depending on model size) that captures the token's meaning in a mathematical space.
This mapping is learned during training. Tokens with similar meanings end up with similar vectors. That is why, in embedding space, the vector for "king" minus "man" plus "woman" lands close to the vector for "queen." The model is not memorizing definitions, it is learning relationships from patterns in massive amounts of text.
In the original Transformer, the embedding dimension was called d_model and set to 512 for the base model. The same weight matrix was shared between the input embedding layer, the output embedding layer, and the final linear layer before the output probabilities. This means the model has one consistent way to translate between tokens and vectors, coming and going.
3. Positional Encoding: Restoring Word Order
Here is a subtle problem. Self-attention (explained next) looks at all tokens simultaneously, with no built-in sense of order. But word order clearly matters: "the dog bit the man" and "the man bit the dog" use identical words in a different order with a completely different meaning.
To fix this, the Transformer adds a positional encoding vector to each token's embedding before it enters the network. This encoding is built from sine and cosine waves of different frequencies:
PE(position, 2i) = sin(position / 10000^(2i / d_model))
PE(position, 2i+1) = cos(position / 10000^(2i / d_model))
Each dimension of the encoding corresponds to a wave of a different frequency, so every position in the sequence gets a unique signature. The authors chose sine and cosine waves specifically because they let the model learn to attend to relative positions easily, and because the pattern can, in theory, generalize to sequence lengths longer than anything seen in training.
Some newer models use learned positional embeddings instead of fixed sine/cosine ones. The original paper tested both and found they performed almost identically.
At this point, each token is represented by: embedding vector + positional encoding. This combined vector is what flows into the attention layers.
4. Self-Attention: The Core Idea
This is the heart of the Transformer. Self-attention lets every token in a sequence look at every other token and decide how much to "pay attention" to it when building its own updated representation.
Query, Key, Value (Q, K, V)
Every token produces three vectors, all learned through separate weight matrices:
- Query (Q): what this token is looking for
- Key (K): what this token offers, as a label others can search against
- Value (V): the actual content this token contributes if selected
A simple way to picture it: think of a search engine. Your search term is the Query. Every document in the index has a Key (title, tags, description). The engine compares your Query against all Keys to find matches, then returns the Values (the actual documents) weighted by how well they matched.
Scaled Dot-Product Attention
The formula from the paper is:
Attention(Q, K, V) = softmax( (Q · K^T) / sqrt(d_k) ) · V
Step by step:
- Take the dot product of the Query with every Key. This produces a raw compatibility score between the current token and every other token.
- Divide by the square root of the key dimension (
d_k). This scaling step keeps the numbers from growing too large, which would otherwise push the softmax function into regions with extremely small gradients and slow down learning. - Apply softmax to turn the scores into a probability distribution that sums to 1. These are the attention weights.
- Multiply those weights by the Value vectors and sum them up. The result is a new vector for this token that blends in information from every other token, weighted by relevance.
Every word ends up with a new representation that reflects its context, not just its isolated meaning. This is how a model figures out that "bank" means something different in "river bank" versus "savings bank," purely from what surrounds it.
Masked Self-Attention
When a model generates text, it should not be able to "cheat" by looking at future words it hasn't produced yet. To prevent this, the decoder uses masked self-attention: positions are only allowed to attend to earlier positions, not later ones. This is done by setting the attention score for any future position to negative infinity before the softmax step, which makes its weight effectively zero. Combined with shifting the output sequence by one position, this guarantees that a prediction for a given position depends only on the known outputs before it. This is what makes the model autoregressive, generating one token at a time based only on what came before.
5. Multi-Head Attention: Many Perspectives at Once
A single attention calculation only captures one type of relationship at a time. Multi-head attention runs several attention operations in parallel, each with its own learned Q, K, and V projection matrices, so the model can capture different kinds of relationships simultaneously: one head might track grammatical structure, another might track long-distance references, another might track sentiment.
MultiHead(Q, K, V) = Concat(head_1, ..., head_h) · W_O
where head_i = Attention(Q·W_i^Q, K·W_i^K, V·W_i^V)
The original Transformer used 8 attention heads (h = 8), each working on a 64-dimensional slice (d_k = d_v = 64), so the total dimensionality across all heads matched the model's overall size of 512. The outputs of all heads are concatenated and passed through one more linear projection to produce the final result.
The paper found this mattered a lot: using just a single attention head performed noticeably worse, since averaging inhibits the model's ability to represent different types of relationships. But too many heads also hurt quality, since each head then works with too little dimensional space to be useful.
The Transformer uses multi-head attention in three distinct places:
- Encoder self-attention: every input token attends to every other input token
- Masked decoder self-attention: every output token attends only to earlier output tokens
- Encoder-decoder (cross) attention: each decoder token attends to the full encoder output, letting the decoder pull relevant information from the input sequence while generating
6. Feed-Forward Network
After attention, each token's vector passes through a small, fully connected feed-forward network, applied identically and independently to each position:
FFN(x) = max(0, x·W1 + b1)·W2 + b2
This is two linear layers with a ReLU activation in between. In the base Transformer, the input and output are 512-dimensional, but the hidden layer in between expands to 2048 dimensions before compressing back down. This gives the model extra capacity to process and transform the information that attention just gathered, on a per-token basis. Attention handles mixing information between tokens; the feed-forward layer handles processing information within each token.
7. Add & Norm: Residual Connections and Layer Normalization
Around both the attention sub-layer and the feed-forward sub-layer, the Transformer applies a residual connection: the input to the sub-layer is added back to its output, followed by layer normalization.
Output = LayerNorm(x + Sublayer(x))
This matters for two practical reasons. First, residual connections let gradients flow more easily through deep networks during training, which prevents the vanishing gradient problems that plagued earlier deep architectures. Second, layer normalization keeps the scale of values consistent as data passes through many stacked layers, which stabilizes training.
8. Stacking It All: Encoder and Decoder
A single attention + feed-forward block is called a layer. The Transformer stacks several identical layers on top of each other, N = 6 in the original paper, to build depth. Each layer refines the representation a little further.
Encoder
The encoder's job is to build a rich understanding of the input sequence. Each encoder layer has two sub-layers:
- Multi-head self-attention (every input token looks at every other input token)
- Position-wise feed-forward network
Decoder
The decoder's job is to generate the output sequence, one token at a time, using both what it has generated so far and the encoder's understanding of the input. Each decoder layer has three sub-layers:
- Masked multi-head self-attention (only looks at earlier output tokens)
- Encoder-decoder cross-attention (looks at the encoder's output)
- Position-wise feed-forward network
This encoder-decoder structure is what the original paper used for machine translation. Most modern LLMs (GPT-style models) are decoder-only: they drop the separate encoder and just stack masked self-attention decoder blocks, since their job is pure text generation rather than translating between two sequences. Models like BERT are encoder-only, built for understanding text rather than generating it.
9. Linear + Softmax: From Vectors to Word Probabilities
After the final decoder layer, the model has a vector for the next token position. This vector is passed through:
- A linear layer that projects it into a score for every single word in the vocabulary (tens of thousands of numbers, one per possible token)
- A softmax function that converts those raw scores into probabilities that sum to 1
The result is a full probability distribution over the entire vocabulary: how likely each possible next token is, given everything the model has seen so far.
10. Generation: How the Model Actually Writes Text
At inference time (when you chat with an LLM), the process becomes a loop:
- The model looks at all tokens so far and predicts a probability distribution for the next token.
- A token is picked from that distribution. This can be the single most likely token (greedy decoding) or a token sampled according to the probabilities with some randomness (which is why the same prompt can give different answers, controlled by a setting often called "temperature").
- That chosen token is appended to the sequence.
- The whole sequence, now one token longer, is fed back into the model to predict the next token.
- This repeats until the model produces a stop signal or hits a length limit.
This is why LLMs generate text one piece at a time and why longer responses take proportionally longer to produce: each new token requires another full pass through the model.
Why This Architecture Won
The Transformer paper measured three things that mattered for training large models efficiently:
- Computational complexity per layer: self-attention connects any two positions in a constant number of operations, while recurrent networks need a number of sequential steps proportional to sequence length.
- Parallelization: because self-attention has no step-by-step dependency like an RNN does, all tokens can be processed simultaneously on modern hardware like GPUs, drastically speeding up training.
- Path length between distant words: in a recurrent network, information from word 1 has to pass through every word in between to reach word 100. In self-attention, any two words are directly connected in a single step, which makes it far easier to learn long-range relationships.
On real benchmarks, the Transformer did not just match previous state-of-the-art translation models, it beat them while training dramatically faster. The big Transformer model reached a BLEU score of 28.4 on English-to-German translation and 41.8 on English-to-French, both new records at the time, while training in a few days on 8 GPUs rather than weeks.
Full Cheat Sheet Table
| Stage | What It Does | Key Detail |
|---|---|---|
| Tokenization | Splits text into subword units | Uses BPE / WordPiece, ~32K-100K+ vocabulary |
| Embedding | Converts tokens into meaning-carrying vectors | Learned during training, similar meanings cluster together |
| Positional Encoding | Injects word order information | Sine/cosine waves of varying frequency added to embeddings |
| Self-Attention | Lets each token gather context from others | Uses Query, Key, Value vectors and scaled dot products |
| Masking | Prevents seeing future tokens during generation | Sets future attention scores to negative infinity |
| Multi-Head Attention | Captures multiple relationship types at once | 8 heads in the original paper, each 64-dimensional |
| Feed-Forward Network | Processes each token's info independently | Two linear layers with ReLU, expands then compresses |
| Add & Norm | Stabilizes deep stacks of layers | Residual connection + layer normalization |
| Encoder | Builds understanding of the input | Self-attention + feed-forward, repeated N=6 times |
| Decoder | Generates output tokens one at a time | Masked self-attention + cross-attention + feed-forward |
| Linear + Softmax | Turns final vector into word probabilities | Produces a distribution over the whole vocabulary |
| Sampling | Picks the actual next word | Greedy or probability-based sampling, then loops |
The Transformer Architecture Diagram
Below is the original architecture diagram from Attention Is All You Need, showing the encoder (left stack) and decoder (right stack) side by side, with inputs flowing up from the bottom and output probabilities produced at the top.
Reading it bottom to top: input tokens are embedded and combined with positional encoding, pass through N stacked encoder layers (masked multi-head attention, add & norm, feed-forward, add & norm), while the decoder stack does the same on the output side but adds a middle layer of cross-attention over the encoder's output. The final decoder output goes through a linear layer and softmax to produce output probabilities.
Source Material
This article summarizes concepts from:
- Paper: Attention Is All You Need – Vaswani et al., Google Brain / Google Research, 2017. The original paper that introduced the Transformer architecture, tested on English-to-German and English-to-French machine translation.
- Video: Transformer walkthrough – a visual explanation of the same architecture, useful for seeing the matrix operations and data flow described above in action.
Quick Recap in One Paragraph
Text gets broken into tokens, each token becomes a vector, and position information gets added so the model knows word order. Self-attention lets every token gather context from every other token using Query, Key, and Value vectors, and multi-head attention runs several of these attention patterns in parallel to capture different relationships. Feed-forward layers then process each token's information further, with residual connections and normalization keeping training stable across many stacked layers. Finally, a linear layer plus softmax turns the model's internal representation into a probability distribution over the vocabulary, and the model samples one token at a time, feeding its own output back in, to generate text. That loop, repeated one token at a time, is what "the LLM is typing" actually looks like under the hood.
Top comments (0)