Every part of an LLM was invented before 2018, each one to fix something that broke. Learn them in that order and they stop feeling arbitrary.
Four generations of AI, one sarcastic review. All four were sure it was positive.
Read this movie review:
"Wow, what a masterpiece. Two hours of pure brilliance, and I only checked my watch twenty times. The acting was so convincing that I truly believed the actors wanted to be somewhere else. A perfect film if you love naps."
You know in two seconds that it's a one-star review. In 2018, BERT, Google's best reading model at the time, fine-tuned on 1,000 movie reviews, was 98 % sure it was positive. A model that just counts words: 76 %. An averaging network: 99 %. An LSTM that reads word by word: 93 %. Four generations of reading machines, and not one of them gets the joke.
That review comes from David Louapre's lecture NLP before LLMs on Hugging Face, which builds every one of those models from scratch. I turned the lecture into a 37-minute animated video, and this article is the short written version.
The wrong order
Most of us learn LLMs top-down. You start with the Transformer diagram: attention, layers, residual connections, and then "it predicts the next token". Every piece arrives as a design decision with no reason attached. Why predict the next word? Why turn words into vectors? Why tokens instead of words? Why attention?
Each of those pieces answers a problem someone ran into, usually years before the Transformer. Learn the problems in the order they showed up, and every part of an LLM comes with its reason.
Two problems run through the whole story:
- Any length in, a fixed size out. A review can be five words or five thousand, but a model needs the same number of inputs every time.
- Learn once, reuse. Labelled data costs money: someone has to read every review. Raw text is free. How much can a model learn from free text, and reuse for any task?
Every model also sits the same two exams (the lecture's two datasets), which makes the progress visible:
- The next-word exam: read two million words of Wikipedia (WikiText-2) and guess every next word. The score is perplexity, which you can read as how many words the model hesitates between. Lower is better.
- The review exam: 50,000 IMDB movie reviews, thumbs up or thumbs down. Plus a bonus round, the "Hard 13": short reviews any person gets right, full of negations, contrasts, sarcasm and swapped word order.
Here's the order.
1. Count what comes next (1913–2000s)
1913: Andrey Markov sorted the first 20,000 letters of Eugene Onegin by hand.
In 1913, Andrey Markov took the first 20,000 letters of Pushkin's Eugene Onegin, sorted them by hand into vowels and consonants, and counted what came next. After a consonant, the next letter was a vowel 66 % of the time. After a vowel, only 13 %. The previous letter already tells you something about the next one.
Do the same with words in a big text and you have an n-gram model. In WikiText-2, "New" appears 1,266 times, and 501 of those times the next word is "York". So P(York | New) = 501 / 1,266 ≈ 40 %. Then "Zealand" at 17 % and "Jersey" at 6 %.
The whole idea: count, then turn the counts into odds (our run on WikiText-2).
A model that knows nothing hesitates between all 33,278 words of the vocabulary. Knowing which words are common brings that down to 942. Four words of context, falling back to shorter contexts when the long one was never seen, gets to 278. Kneser-Ney smoothing, the best counting recipe of the era, reached 219.
What broke: the counts run out. Two million words of Wikipedia contain 0.06 % of the 1.1 billion possible word pairs, and 95 % of the five-word sequences in new text never appear in training at all. Where the model does know a long context, it often just copies the training text.
What survived: the task itself. Next-word prediction, and cross-entropy, the score that grades it, are exactly what LLMs are trained on today.
2. Count the words (1960s–2012)
To classify a review, the simplest move is a bag of words: count each of the 10,000 most common words in the review, and throw the order away. It gets 87.6 % on the review exam. Not bad, but look at what it learned: among the most positive "words" are "7" and "8", because people write their rating in the text.
A bag of words keeps the counts and loses the order.
On the Hard 13 it gets 7 right. "Not bad at all, actually" comes out negative, because "bad" is bad. "I expected it to be terrible, but it was great" and the same sentence with "great" and "terrible" swapped make the same bag, so they get exactly the same score, and one of them has to be wrong. And "Excellent movie, awesome soundtrack" shares no word with "Superb film, wonderful score": to a bag of words, every word is an island.
What broke: meaning and order. The linguist Zellig Harris put it in one line in 1954: language is not merely a bag of words.
What survived: not the bag. But it showed the next decade exactly what was missing.
3. Give every word an address (1986–2015)
Instead of one slot per word, give each word a short list of numbers, so that similar words get similar lists. Think of it as an address on a map of meaning. That address is an embedding.
In 2003, Yoshua Bengio's team built a neural language model on this idea: look up the addresses of the last three words, pass them through a small network, and get a probability for every possible next word. The addresses are learned along with everything else. With three words of memory, it beat the best counting model, which used four: 208 instead of 219. The price was speed: it scored all 33,000 candidates for every word.
In 2013, Tomas Mikolov's word2vec stripped this down to little more than the table of addresses. It also swapped the expensive question ("which of 33,000 words comes next?") for a cheap one ("did these two words really appear together, or is one of them random?"). That made it fast enough to get through a hundred billion words in a day. The maps even hold directions: king − man + woman lands near queen.
Here's what the headlines skip: the word closest to king − man + woman is… king itself. "Queen" only wins because the standard test forbids the question's own words. Allow them, and the score on that test drops from 74 % to 22 %.
What broke: averaging the addresses of a review is still a bag of words (a smoothie doesn't care in which order the fruit went in), so the review exam stays stuck around 88 %. And every word gets one address: in GloVe, "river" is only the 544th closest word to "bank".
What survived: embeddings. Every LLM starts by turning each token into a list of learned numbers.
4. Read in order (1990–2016)
A recurrent neural network (RNN) reads one word at a time and rewrites a running memory after each word. The same rule runs at every step, so it can read any length, and the memory after the last word is the fixed-size box. Problem 1, solved in principle.
Training it is the hard part. The learning signal has to travel back through every word, and it shrinks at each step. We measured it on a plain RNN: ten words back, it's down to a thousandth; fifty words back, to 10⁻¹⁷. That's the vanishing gradient. The LSTM (1997) fixed it with a second memory, a conveyor belt that runs straight through every step, with small gates that decide what to put on it, what to read off it, and how much of it to keep.
The LSTM finally uses the order: it gets both of the swapped-word "twins" right. But on the review exam it scores 87.2 %, the same wall, because it had to learn everything, the words included, from 20,000 reviews. As a next-word guesser, though, it's a big jump: 156, and the best LSTMs of 2017 got down to 66.
This step also brings tokens. Feed an LSTM letters and no word is ever unknown, but the text gets five times longer. Today's models meet in the middle: common words stay whole, rare ones are cut into common pieces. "Flabbergasting" becomes four tokens.
What survived: tokens. Reading one word at a time didn't.
5. Pre-train the whole reader (2015–2018)
An LSTM that guesses the next word is the same reader as one that grades reviews. So in 2015, Andrew Dai and Quoc Le trained it on raw text first, then fine-tuned it on reviews: 86.5 % → 92.4 %. The 88 % wall finally broke.
Pre-train first, then adapt: the wall breaks.
In 2017, Alec Radford and colleagues trained a big LSTM on 82 million Amazon reviews, one character at a time. One single number in its memory tracked sentiment, and on its own it scored 92 % on the movie exam. Nobody asked it to learn sentiment: it learned it because knowing the mood helps predict the next letter. A year later, Radford built GPT.
ULMFiT (2018) turned this into a recipe: learn the language on Wikipedia, adapt to your own texts, then train the classifier. It reached 95.4 %, and with only 100 labelled reviews it matched models trained from scratch on 10 to 100 times more. ELMo (2018) gave each word a new address in every sentence, so the two kinds of "bank" finally differ.
What survived: pre-train, then adapt. Problem 2, solved.
6. Translation invents attention (1990–2016)
Translation ran on counting too, graded by BLEU, which counts shared word groups: "I loved this great film" against "I loved this great movie" scores 67 out of 100. In 2014, sequence to sequence did the whole job with two networks: one reads the English into a memory, the other writes the French from it.
But the whole sentence passes through one fixed-size memory, like summing up a paragraph on one sticky note before translating it. We measured that bottleneck by teaching a small model to reverse strings of digits: it gets 99 % right with 10 digits, more than half have a mistake with 20, and none come out right with 30.
That same summer, Dzmitry Bahdanau, an intern in Yoshua Bengio's lab, let the decoder look back at every word of the source each time it writes one. That's attention. Same digits, same network, same training, plus attention: 99 % of the 30-digit strings come out perfect. In real translation, its grid is the word alignment, learned for free.
In 2016, Google switched Translate to a neural system with attention and cut its translation errors by about 60 %.
What survived: attention, born in translation.
7. Attention takes over (2017–2022)
One problem was left: an RNN still reads one word after another, while GPUs want to do thousands of calculations at once. In 2017, the Transformer dropped the RNN and kept only attention: every word looks at every other word, all at the same time. Each word also gets its position added, because attention alone would read a text like a bag of words.
Then the two halves went separate ways. BERT (2018) pre-trained the reader half with a fill-in-the-blank game. It's word2vec's game again, except the blank now sees the whole text, through 12 layers of attention. After those layers, river banks and money banks finally sit apart.
At BERT's last layer, the two kinds of "bank" separate (our run, bert-base-uncased).
Fine-tuned on just 1,000 reviews, BERT gets 89.2 %, beating every earlier model trained on 20 times more, and 10 of the Hard 13. But it's still 98 % sure the sarcastic review is positive. A thousand movie reviews don't teach you that checking your watch twenty times isn't praise.
GPT took the writer half. GPT-2 (2019) was never trained on WikiText-2 and still beat every model that was: its largest version hesitates between 18 words. GPT-3 (2020) removed the last task-specific piece: you write the task in the prompt. Teaching the model to follow instructions (InstructGPT, 2022) made the chatbot.
Back to that review
I gave the review to ChatGPT (GPT-6, a fresh chat), exactly as it is. Its answer begins: "Negative. The review uses sarcasm and irony to express dissatisfaction with the movie." Then it explains every joke: checking your watch twenty times means the movie was boring, and a perfect film if you love naps means it could put you to sleep. It gets the joke.
ChatGPT's answer, word for word.
Not every model does. Qwen3-4B, a small open model running on my own graphics card, gets 12 of the 13 hard reviews, but on this one it says positive, 20 times out of 20, even after about a thousand tokens of thinking. To it, checking your watch twenty times means you were so absorbed you rarely needed to. Sarcasm still takes a big model.
What we kept, and what we dropped
| Kept, inside every LLM today | Dropped |
|---|---|
| next-word prediction (Markov, Shannon) | tables of counts |
| cross-entropy, the score that grades it | bags of words |
| words as addresses (embeddings) | reading one word at a time |
| pieces of words (tokens) | |
| attention, born in translation | |
| pre-train, then adapt |
The order I'd learn it in
- n-grams, cross-entropy and perplexity
- bag of words with a linear classifier, and exactly where it fails
- embeddings: Bengio 2003, word2vec, GloVe
- RNNs, the vanishing gradient, LSTMs and tokens
- pre-training: Dai & Le, ULMFiT, ELMo
- sequence to sequence, then attention
- the Transformer, BERT and GPT
After that, the Transformer diagram stops being a list of design choices. Each box is the fix for something you've already seen break.
You don't have to take my word for any of it: David Louapre's lecture has the exercises to build every one of these models yourself, on the same datasets and the same exams.
And the 37-minute video shows every step moving: watch it on YouTube. My channel, The Next Token Lab, is where I keep explaining how AI actually works: https://www.youtube.com/@TheNextTokenLab
Where did you start learning LLMs, and what was the last piece that clicked?
Based on the lecture "NLP before LLMs" by David Louapre (Hugging Face), licensed CC BY 4.0, adapted. Numbers marked "our run" are my own runs (WikiText-2, GloVe, GPT-2 small, BERT base, a small translation model, Qwen3-4B). The others come from the lecture's runs or the original papers. Not affiliated with or endorsed by OpenAI, Google, Alibaba or Hugging Face.











Top comments (0)