For years, the rule was simple: bigger is better. More data, more parameters, more compute. Each generation of models was significantly smarter than the last. GPT-2 was impressive. GPT-3 was astonishing. GPT-4 was a leap. Then came GPT-5. It was better, but not vastly better. The leap was smaller. The scaling law was breaking.
This is the end of the scaling era. The returns on scale are diminishing. Bigger models are no longer vastly smarter. The industry is facing a crisis of diminishing returns.
The Scaling Law
The scaling law was the foundation of the AI boom.
The Rule:
More data → better performance.
More parameters → better performance.
More compute → better performance.
The Evidence:
GPT-2 (1.5B parameters): Good.
GPT-3 (175B parameters): Great.
GPT-4 (1.8T parameters): Excellent.
The Promise:
GPT-5 (10T parameters): Superhuman.
GPT-6 (100T parameters): Godlike.
A Contrarian Take: The Scaling Law Was Never a Law. It Was a Trend.
We called it a "law." But it was just a trend. Trends do not last forever.
The scaling law was a curve. Curves plateau. We are hitting the plateau.
Why the Scaling Law Broke
There are several reasons for the diminishing returns.
- Data Exhaustion:
The internet is finite.
We have already scraped most of it.
New data is lower quality.
- Compute Limits:
Training larger models is exponentially expensive.
The cost is not worth the gain.
- Architectural Limits:
The transformer architecture has limits.
Adding more parameters does not always help.
A Contrarian Take: The Scaling Law Did Not Break. It Evolved.
The scaling law is not dead. It is changing.
We are moving from scaling size to scaling efficiency. The goal is no longer bigger models. It is smarter models.
The Evidence: GPT-5 vs. GPT-4
The performance gap between GPT-5 and GPT-4 is smaller than the gap between GPT-4 and GPT-3.
The Gains:
GPT-4 was a massive leap over GPT-3.
GPT-5 is a modest improvement over GPT-4.
The Speculation:
GPT-5 is better at reasoning.
It is better at long contexts.
But it is not fundamentally smarter.
A Contrarian Take: The Gains Are in the Details.
GPT-5 may not be vastly smarter. But it is more reliable, more consistent, and more efficient.
The gains are not about raw intelligence. They are about refinement.
The Economic Reality
The cost of training larger models is staggering.
The Numbers:
GPT-3: ~$4.6 million.
GPT-4: ~$100 million.
GPT-5: ~$1 billion.
The Return:
The return on investment is diminishing.
A $1 billion model is not 10x better than a $100 million model.
A Contrarian Take: The Economics Will Force a Shift.
The AI companies cannot keep spending billions on marginal gains.
The future is not about bigger models. It is about cheaper models.
What Comes Next
If bigger is no longer better, what is the path forward?
- Efficiency:
Make models more efficient.
Use sparse attention, quantization, and distillation.
- Specialization:
Build smaller, domain-specific models.
A medical AI does not need to know poetry.
- Architecture:
Explore new architectures (Mamba, SSMs, hybrids).
The transformer may not be the end.
A Contrarian Take: The Future Is Not One Model. It Is Many.
We are moving from a world of one giant model to a world of many small models.
The future is not GPT-6. It is a swarm of specialized AIs.
What You Can Do
You cannot change the scaling laws. But you can adapt.
- Focus on Use Cases:
Do you need a giant model?
Maybe a smaller model is enough.
- Experiment with Open-Source Models:
Llama, Mistral, and Qwen are competitive with GPT-4.
They are smaller, cheaper, and open-source.
- Be Skeptical of Hype:
The next model will not be a god.
Manage your expectations.
The Last Scaling Law
The last scaling law is not about models. It is about you.
You ask: "What is the future of AI?"
The model says: "The future is in your hands."
You realize: The scaling law is not about the model. It is about the user.
If you could build a model optimized for one specific task, what would it be? And why?
Top comments (0)