Welcome back to AI From Scratch.
If you're still here on Day 6, you're officially that friend who "just wanted a simple overview" and then accidentally learned how half the field works.
Quick rewind:
Day 1: AI as a next‑word prediction machine.
Day 2: How it learns by failing and nudging weights.
Day 3: What's happening inside when it "thinks."
Day 4: Transformers and attention - the wiring that made modern AI possible.
Day 5: AI doesn't read words, it reads tokens and numbers.
Today's question:
If everyone keeps bragging about "50B parameters" or "1T parameters"…
what does making a model bigger actually change?
So, what even is a "parameter" again?
From Day 1 and 2: a parameter is just one tiny knob inside the model , a weight that says "when I see this pattern, react this much."
A model with 1 million parameters is like a brain with 1 million tiny switches.
A model with 1 trillion parameters is like a brain with a whole galaxy of switches.
More parameters = more capacity to store patterns from training data:
language quirks, world facts, coding tricks, writing styles, reasoning shortcuts.
So what this means for you: when people say "bigger model," they literally mean "a brain with way more knobs that can in theory capture way more detail."
Why making models bigger helped so much
Around 2020, researchers noticed something wild:
if you scale up model size, data, and compute in the right way, performance improves in a pretty smooth, predictable way, these are the famous scaling laws.
In practice, that meant:
- 10× more compute → noticeably lower error on language tasks.
- Bigger models kept getting better, not just a tiny bit, but enough to be worth the extra GPUs.
That's why we went from models with millions of parameters to ones with billions and then hundreds of billions , the graph kept trending in the right direction.
So what this means for you: the "era of huge models" wasn't just hype, the data really did show that, for a while, simply scaling up size (plus data and compute) kept unlocking better performance.
The spooky part: new abilities just… appear
As people scaled models up, something surprising happened:
bigger models started doing things they were never explicitly trained to do.
Examples researchers noticed:
- Smaller models: decent at basic text completion.
- Larger ones: suddenly could translate, do few‑shot learning ("here are 3 examples, now do the 4th"), solve simple math, write code, all from the same next‑word training objective.
These are often called emergent abilities, skills that seem to "switch on" once you pass a certain size, even though the training recipe didn't change.
So what this means for you: when GPT‑3 felt qualitatively different from GPT‑2, it wasn't because someone manually added "write emails" mode - it was a side‑effect of pushing model size, data, and compute past a certain threshold.
But it's not "just make it huge" - data matters a lot
Then another twist: bigger isn't always better if you don't feed it enough data.
DeepMind's **Chinchilla **work showed that GPT‑3‑style models were actually under‑_trained _on data for their size.
They trained a _smaller _model (around 70B parameters) on more tokens than previous giants, and it beat much larger models that had less data.
Roughly speaking, they found:
for a fixed compute budget, you should grow model size and dataset size together, instead of only cranking up parameters.
So what this means for you: a 1T‑parameter model trained on too little or low‑quality data can be dumber than a well‑trained 70B model. Size gives capacity; data and training actually fill it with something useful.
Small vs large models in the real world
Outside research papers, teams now run into a very practical question:
"Do we really need the giant model for this job?"
Rough pattern people see:
- **Large models (10B–70B+ parameters):
- **Better at complex reasoning, multi‑step tasks, and understanding long context.
- Often lower hallucination rates on factual queries (though still not perfect).
- Heavier: more GPUs, more energy, more latency and cost.
- Small models (<1B–a few B parameters):
- Fast, cheap, can sometimes run on a laptop or phone.
- Great when you fine‑tune them for a very specific domain.
- Weaker at open‑ended reasoning and multi‑language, but easier to deploy privately.
So what this means for you: bigger models tend to feel "smarter" on broad, messy tasks, but for focused, everyday jobs (like one company's support emails), a smaller, tuned model can actually be the better call.
What does scaling really buy you?
If we strip away the marketing, going from 1M to 1B to 1T parameters mainly buys you:
_- More capacity
- The model can store and express richer patterns about language, code, and the world, especially when paired with enough training data.
- Better generalization
- It handles weirder prompts, rare edge cases, and "I've never seen this exact thing, but I can reason it out from patterns I have seen."
- Longer, more coherent chains of thought
- With larger models and bigger context windows, you can give longer instructions and documents and still get reasonable, on‑topic answers.
- New capabilities at certain sizes
- Translation, coding help, chain‑of‑thought reasoning, few‑shot learning, these start to show up more clearly the bigger you go._
But in exchange, you pay in compute, latency, energy, and money, which is why there's now a whole movement around "small but smart enough" models.
So what this means for you: "Is bigger smarter?" is the wrong question. The better question is: "For this job, is the extra capability from a larger model worth the extra cost and complexity?"
Zooming out: where we are by Day 6
Let's connect the dots from the whole series so far:
The model predicts the next token using weights (parameters).
Those weights were learned through the training loop.
Inside, transformers and attention structure the thinking process.
Input text becomes tokens and embeddings inside a fixed context window.
Scaling up size + data + compute follows surprisingly smooth laws… until data or money runs out.
So what this means for you: when you hear "this new model is 4× bigger," you now know that really means "its brain has more room for patterns, but whether that translates to real gains depends on data, training, and what you're using it for."
Teaser for Day 7, The Training Upgrade Nobody Talks About
Today we stayed at the "raw capability" level , how big the brain is, and how much that matters.
But there's another twist coming:
Why does the same model feel dumb in one setting and super helpful in another?
On Day 7 - "How Does AI Go From Dumb to Useful? The Training Upgrade that matters"
we'll get into:
**Base models vs instruction‑tuned models
**What "RL from human feedback" actually changes in behavior
Why some AIs feel like they're arguing with you, and others feel like a polite sugarcoating assistant
In other words: you've seen how we build the brain and how big it can get.
Next, we'll talk about how we teach that brain to talk to humans in a way that's actually useful to you.
What blew your mind most? Drop a comment!
Top comments (0)