Hello, I'm Shrijith Venkatramana. I'm building git-lrc, an AI code reviewer that runs on every commit. Star Us to help devs discover the project. Do give it a try and share your feedback for improving the product.
There is a strange fact at the heart of modern AI:
If you make a language model bigger, give it more data, and spend more compute training it, its performance improves according to remarkably predictable mathematical relationships.
This was not obvious in 2019.
At the time, it was reasonable to think that neural networks might eventually hit some messy combination of architectural limitations, optimization failures, or diminishing returns. Instead, Jared Kaplan and colleagues at OpenAI found something much more interesting: over enormous ranges of model sizes, datasets, and compute budgets, language-model loss followed approximate power laws.
That observation became one of the intellectual foundations of the modern LLM industry.
It also led to one of the great practical mistakes of early LLM scaling.
OpenAI's 2020 work suggested that, given a fixed compute budget, you should spend disproportionately on larger models and relatively little on data. Two years later, DeepMind's Chinchilla experiments showed that this allocation was badly wrong: many famous models were simply too large and insufficiently trained.
For developers, scaling laws are more than an interesting piece of ML history. They explain why model providers obsess over parameters, tokens, GPUs, and training runs—and why the economics of inference can completely change what "the best model" means.
Let's work from intuition to the math.
1. The basic idea: intelligence has a surprisingly smooth price curve
Imagine training models with:
- 100 million parameters
- 1 billion parameters
- 10 billion parameters
- 100 billion parameters
You might expect their performance to jump around unpredictably.
Instead, something closer to this happens:
Model size Loss
----------- ----
100M 3.8
1B 3.1
10B 2.6
100B 2.2
The improvements get smaller, but they do not suddenly disappear.
The important observation is that the relationship is approximately a power law:
Loss ~= constant + A / N^alpha
where:
N = number of model parameters
alpha = scaling exponent
The exponent is small.
That is crucial.
If:
alpha = 0.08
then multiplying the model size by 10 gives:
improvement factor = 10^0.08 ~= 1.20
That sounds unimpressive.
But now multiply the model by 100:
100^0.08 ~= 1.45
And by 1,000:
1000^0.08 ~= 1.74
The returns diminish, but they remain remarkably persistent.
This is why "just make the model bigger" worked for so long.
Kaplan, McCandlish, Henighan and their OpenAI colleagues measured these relationships across models and training regimes spanning more than seven orders of magnitude. Their 2020 paper found that loss scaled predictably with three quantities:
- model size,
- dataset size,
- compute used for training.
That was the important discovery.
The question then became:
If compute is limited, where should you spend it?
2. A little history: how GPT-3 turned scaling into an engineering strategy
The story becomes more interesting when you look at what happened around the research.
In the late 2010s, OpenAI had already demonstrated the usefulness of Transformer language models. GPT-2, released in 2019, had 1.5 billion parameters.
Then came GPT-3.
The 2020 GPT-3 paper described a 175-billion-parameter model—more than 100 times larger than GPT-2.
That sounds absurd if you think about models as software projects.
But scaling laws provided a rationale:
If loss continues following a predictable curve as model size increases, then increasing model size is not merely making the model "larger." It is buying measurable improvements.
This changed the engineering question.
Instead of:
"What clever architecture should we invent?"
you could increasingly ask:
"How much improvement do we get for another $10M of compute?"
That is a radically different research program.
And it is why the modern LLM race became so capital-intensive.
The underlying model is relatively simple:
more parameters
|
v
more computation
|
v
lower training loss
|
v
better downstream capability
The first three arrows are where scaling laws are strongest.
The last arrow is messier.
That distinction matters.
3. The Kaplan equations: three things you can scale
Kaplan et al. essentially studied three independent axes.
Let:
N = number of parameters
D = number of training tokens
C = training compute
L = validation loss
A simplified version of the relationships looks like:
L(N) ~= L_inf + A / N^alpha_N
L(D) ~= L_inf + B / D^alpha_D
L(C) ~= L_inf + E / C^alpha_C
The exact fitted constants aren't the interesting part.
The shape is.
Scaling model size
L(N) ~= L_inf + A N^(-alpha_N)
Increasing N reduces loss.
But each additional increase buys less than the previous one.
Scaling data
Similarly:
L(D) ~= L_inf + B D^(-alpha_D)
More training tokens improve the model.
Again, diminishing returns.
Scaling compute
Compute roughly captures the interaction between the two:
C ~= k N D
for dense Transformer training, because each token requires work proportional to the number of parameters.
This is the first really useful equation for a developer:
training compute ~= parameters x training tokens
More precisely, the FLOP count has architecture-dependent constants, but the approximation is excellent for intuition.
Suppose you have:
N = 10B parameters
D = 300B tokens
Then the basic scaling quantity is:
N x D = 3 x 10^21
You cannot escape this multiplication.
A 10x larger model trained on the same amount of data costs roughly 10x as much compute.
A model of the same size trained on 10x more tokens also costs roughly 10x as much.
So now we have an optimization problem:
Given fixed C:
How should I divide C between
model size N and data D?
This is where Kaplan's work became operationally important.
4. The surprising Kaplan result: make the model very large
Kaplan et al. estimated that compute-optimal training favored increasing model size aggressively.
In simplified terms, their results suggested relationships roughly like:
N_opt ~= C^0.73
D_opt ~= C^0.27
The exact fitted exponents depend on the formulation and regime, but the qualitative result was striking:
As your compute budget increases, model size should grow much faster than the number of training tokens.
Why?
Because a larger model was substantially more sample-efficient.
Imagine you have a fixed $100M compute budget.
You could choose:
Small model + enormous dataset
or:
Large model + smaller dataset
Kaplan's measurements suggested the second option would give better loss.
That thinking was highly influential.
It helps explain the era of enormous models:
GPT-3 175B
Gopher 280B
MT-NLG 530B
The industry was effectively following the logic:
compute is scarce
|
v
make the model very large
|
v
models become increasingly capable
And for a while, this looked like the correct recipe.
Then DeepMind did something rather inconvenient.
5. Chinchilla: the industry discovered it had been starving its models
In 2022, Jordan Hoffmann and colleagues at DeepMind published Training Compute-Optimal Large Language Models.
They trained more than 400 models ranging from roughly 70 million to over 16 billion parameters, using datasets ranging from 5 billion to 500+ billion tokens.
Their conclusion was almost the inverse of the prevailing practice:
Large language models were being trained on far too little data.
They found that, under their compute-optimal regime, model size and training tokens should scale approximately together:
N doubles
|
v
D should roughly double
The famous practical rule became approximately:
~20 training tokens per parameter
for the regime studied by Chinchilla.
The experiment that made this famous was Chinchilla itself.
DeepMind trained:
Gopher:
280B parameters
~300B tokens
Chinchilla:
70B parameters
~1.4T tokens
Chinchilla had one quarter as many parameters but roughly four times as much training data, while using approximately the same training compute.
And it performed substantially better across a broad collection of evaluations.
This was a beautiful example of why scaling laws matter.
The question wasn't:
"Which model is bigger?"
It was:
"Which allocation of compute produces the most capable model?"
Those are very different questions.
A back-of-the-envelope example
Suppose you have:
C = fixed compute budget
Consider two hypothetical models.
Model A:
N = 100B
D = 1T tokens
Model B:
N = 50B
D = 2T tokens
Ignoring architecture-specific constants:
C_A ~= 100B x 1T
C_B ~= 50B x 2T
Therefore:
C_A ~= C_B
Same rough training compute.
But Model B has twice as many training tokens per parameter.
If you are in the undertrained regime, that can produce a much better model.
This was Chinchilla's central lesson.
6. The economics: training-optimal isn't necessarily deployment-optimal
Here is where things get particularly interesting for developers.
Suppose two models have comparable quality:
Model A: 400B parameters
Model B: 100B parameters
If Model B required more training data to achieve that quality, you might still prefer it.
Why?
Inference.
Every generated token requires the model's weights to participate in computation.
For a dense Transformer, inference cost is approximately proportional to:
parameters x generated tokens
at a high level.
So if:
Model A = 400B
Model B = 100B
then, all else equal:
Inference compute A
------------------- ~= 4
Inference compute B
That difference becomes enormous at scale.
Suppose your product generates:
1 billion tokens/month
A 4x difference in inference compute is no longer an academic detail.
It affects:
- GPU count
- latency
- electricity
- cooling
- datacenter capacity
- API margins
- price per token
- maximum concurrent users
And this exposes an important limitation of Chinchilla-style scaling.
Training-optimal is not necessarily business-optimal.
If you are training a model once and serving it for billions of requests, paying somewhat more during training to obtain a smaller model can be economically rational.
Later research explicitly incorporated inference demand into scaling-law optimization and found that sufficiently high inference workloads can favor models smaller than the standard Chinchilla-optimal solution.
Think of it as an amortization problem:
Total cost
=
training cost
+
(number of inference tokens x cost/token)
For a research model used 10,000 times:
training cost dominates
For a model used by hundreds of millions of people:
inference cost dominates
The optimal model can therefore change dramatically.
This is why "the best model" is an incomplete engineering concept.
There is:
best model for training compute
best model for latency
best model for inference cost
best model for quality
best model for a fixed hardware budget
best model for a business with 1B requests
Scaling laws give you the machinery for thinking about those tradeoffs.
7. What developers should actually take away
You probably aren't going to train a 500-billion-parameter model in your garage.
So why should you care?
Because scaling laws teach a much more general engineering lesson:
LLM performance is often governed by smooth resource tradeoffs rather than isolated architectural breakthroughs.
When evaluating a model, ask three questions.
1. How much model capacity am I buying?
N = parameters
More parameters generally provide more capacity, with diminishing returns.
But parameter count alone is increasingly misleading because architectures such as MoE can have huge total parameter counts while activating only a subset per token.
2. How much information has the model actually seen?
D = training tokens
A 100B model trained on 1T tokens and a 100B model trained on 10T tokens are very different objects.
Parameter count without training-token count tells you surprisingly little about how well-trained a model is.
3. What is my compute budget?
Training:
C_train ~= N x D
Inference:
C_infer ~= N x generated_tokens
These two equations are enough to explain a surprising amount of the economics of LLMs.
And they give you a useful mental model when reading model announcements.
If someone announces:
"New 500B parameter model!"
your immediate questions should be:
How many tokens was it trained on?
How much compute was used?
How many parameters are active per token?
What is the inference cost?
What quality does it achieve per dollar?
The headline parameter count is only one coordinate in a much larger optimization problem.
Conclusion: scaling turned AI into an industrial science
Kaplan's 2020 paper was important because it suggested that LLM progress wasn't just a sequence of lucky architectural discoveries.
There was a measurable curve.
Make the model larger:
loss falls
Give it more data:
loss falls
Give it more compute:
loss falls
And, remarkably, these improvements often followed power laws over huge ranges.
Then Chinchilla demonstrated the other half of the story:
You don't merely want a large model.
You want the right model for your compute budget.
That distinction transformed the field.
The interesting question today isn't simply:
"How big can we make the model?"
It is:
"Given a fixed amount of silicon, data, money, energy, and inference demand, what allocation produces the most useful intelligence?"
That is a much more interesting engineering problem.
And it is one reason scaling laws are worth understanding even if you never train an LLM yourself.
If you had $10 million to build an LLM today, would you spend it on a larger model, more training data, better data quality, or cheaper inference—and why?
*AI agents write code fast. They also silently remove logic, change behavior, and introduce bugs -- without telling you. You often find out in production.
git-lrc fixes this. It hooks into git commit and reviews every diff before it lands. 60-second setup. Completely free.*
Any feedback or contributors are welcome! It's online, source-available, and ready for anyone to use.
HexmosTech
/
git-lrc
Free, Micro AI Code Reviews That Run on Git Commit
| 🇩🇰 Dansk | 🇪🇸 Español | 🇮🇷 Farsi | 🇫🇮 Suomi | 🇯🇵 日本語 | 🇳🇴 Norsk | 🇵🇹 Português | 🇷🇺 Русский | 🇦🇱 Shqip | 🇨🇳 中文 | 🇮🇳 हिन्दी |
git-lrc
Free, Micro AI Code Reviews That Run on Commit
GenAI today is a race car without brakes. It accelerates fast -- you describe something, and large blocks of code appear instantly. But AI agents silently break things: they remove logic, relax constraints, introduce expensive cloud calls, leak credentials, and change behavior -- without telling you. You often find out in production.
git-lrc is your braking system. It hooks into git commit and runs an AI review on every diff before it lands. 60-second setup. Completely free.
In short, git-lrc helps Prevent Outages, Breaches, and Technical Debt Before They Happen
At a glance: 10 risk categories · 100+ failure patterns tracked · every commit…
Top comments (0)