Welcome back to AI From Scratch Series.
If you've made it to Day 4, you're basically doing an informal mini‑course over chai.
Quick recap so far:
Day 1 We met the basic trick - AI stores knowledge as weights and predicts the next word.
Day 2 We watched it train like a kid practicing basketball: guess, get feedback, adjust, repeat.
Day 3 We walked through what happens inside when it "thinks" - layers, neurons, little light bulbs firing.
Today is about the plot twist that took all of that and made it actually work at the scale of ChatGPT, Gemini, Claude and:
an idea called attention, wrapped in an architecture called the Transformer
Before attention: AI that forgot the start of the sentence
Before 2017, language models mostly used RNNs and LSTMs , fancy ways of reading text one word at a time, left to right.
Imagine trying to understand a long WhatsApp message where you can only remember the last few words clearly, and everything before that is a blur. That was old‑school AI.
By the time it reached the end of a long sentence, the beginning was basically fuzzy.
These models struggled with:
Long sentences losing contexts ("At the party yesterday, the friend of my sister who moved to Canada…," they'd lose track).
Slow Parallel training (they had to read word by word, so no big speed‑ups).
So what this means for you: early models could do some language tasks, but they hit a ceiling on how coherent and knowledgeable they could feel in long conversations.
The 2017 "Attention Is All You Need" moment
In 2017, a group of Google researchers dropped a paper literally titled "Attention Is All You Need."
Their move was kind of savage: "Let's throw away the old word‑by‑word reading style and build something that just… looks at everything at once and decides what to care about."
This new design was called the Transformer.
Instead of marching through the sentence in order, it looks at the whole sentence at once and decides which words matter for each position and which word means what by surrounding word context using an attention mechanism.
So what this means for you: that one design shift is why modern chatbots you use daily can keep track of long prompts, instructions, and context in a way older models simply couldn't.
Attention, in plain language: who should I care about right now?
Let's say the sentence is:
"The book that the boy who wore a red hoodie was reading was fascinating."
For the word "was" at the end, you don't care about "hoodie", you care about "book."
Your brain instantly jumps back and hooks "was" to "book," not "boy" or "hoodie."
**Attention **is the model doing the same thing:
for each word, it asks, "Which other words in this sentence are actually relevant to me?" and then focuses more on those.
You can think of it like a highlighter pen that moves around the sentence for every word:
When processing "was," the highlighter glows strongly on "book."
When processing "red," it glows on "hoodie."
When processing "boy," it might glow on "who" and "hoodie."
So what this means for you: instead of treating all words equally, your AI constantly re‑weights the sentence, pulling the most relevant parts into focus for each piece of the answer.
Self‑attention: the group chat in the model's head
More specifically, self‑attention means every word in the sentence can "talk" to every other word and decide how much it should matter.
Picture a group discussion:
Each person (word) is allowed to look around the room and think, "Whose opinion matters most for what I'm about to say?"
For this moment, maybe you care most about what the data guy said. Next moment, you care more about the designer.
In the model:
Every word creates tiny internal signals that say "here's who I might care about" and "here's what I mean."
The attention mechanism turns that into weights , basically, "Look 60% at this word, 30% at that one, 10% at those others."
Then it blends information accordingly.
So what this means for you: when the model answers, it's not reading your message in a straight line. It's constantly cross‑referencing parts of your text with each other, like a very fast group chat where everyone can instantly consult everyone else, and not one at a time serially.
Multi‑head attention: many spotlights at once
One attention pattern is nice, but language is messy.
Sometimes you care about grammar (who did what), sometimes about tone, sometimes about time, sometimes about location.
Transformers handle this with multi‑head attention.
Instead of one big spotlight, they use many smaller ones:
One head might focus on subject–verb relationships.
Another might track pronouns ("he", "she", "they").
Another might watch for time phrases ("yesterday", "next year").
All these heads look at the sentence in parallel, each with its own "perspective."
Then the model mixes their insights together.
So what this means for you: that feeling of "wow, it kept track of who I was talking about and the timeline and the tone" comes from multiple attention heads focusing on different aspects of your message at the same time.
Why this unlocked giant, smart-feeling models
Two big reasons transformers changed the game:
They handle long context well
Because every word can talk to every other word directly, it's much easier for the model to connect "this thing you said 20 tokens ago" to "this word I'm choosing now."They run fast on modern hardware
Old RNNs had to read word by word. Transformers can process all tokens in a sentence in parallel, which fits perfectly with GPUs and large clusters.
That parallelism is what made it realistic to train models with billions of parameters on huge text datasets.
So what this means for you: the reason you have chatbots that can write essays, translate, summarize papers, and code is not just "more data" or "bigger models", it's that attention + transformers made training big models actually practical.
Bringing it back to your mental picture
Let's merge this with your understanding from Days 1–3:
Day 1: AI is a next‑word prediction machine with lots of weights.
Day 2: Those weights were learned through endless cycles of "guess → compare → adjust."
Day 3: Inside, your text flows through layers and neurons like an assembly line of tiny reactions.
Day 4 (today): In those layers, transformers use attention so each word can see the whole sentence and decide what to care about before making its contribution.
If I had to summarise this whole blog - the magic of modern AI isn't some mysterious soul hiding in the model. It's a very disciplined system that reads everything at once, focuses on the right bits using attention, and then runs its familiar next‑word prediction game on top of that.
What's coming on Day 5
Now that you've got:
How AI stores knowledge (weights),
How it learns (training loop),
How it thinks (layers and neurons), and
How it uses attention to predict next word more efficiently without losing context (transformer and attention),
…we're ready for the next natural question:
"AI Doesn't Read Words. Here's What It Actually Reads."
On Day 5, we'll see how your text is chopped into tokens and turned into numbers the model can understand , and why things like tokenization and "context window" secretly control how much your AI can remember from your prompt and how coherent its answer can be.
What blew your mind most? Drop a comment!
Top comments (0)