Bigger AI models aren't always better: how a 70B model beat a 280B one by reading more

For a few years, the way to build a better AI model was simple. Make it bigger.

Then in 2022 DeepMind trained a model called Chinchilla. It had 70 billion parameters. It beat their own older model, Gopher, which had 280 billion. A quarter of the size, better scores.

How? Chinchilla didn't have a smarter design. It just read a lot more.

I wanted to see that for myself, so I trained a pile of tiny models on my laptop. Here's what happened.

First, two words

• Parameters: the adjustable numbers inside a model. "70B" means 70 billion of them. Think of it as the size of the model's brain.
• Tokens: the chunks of text a model reads while training. Think of it as how much it studied.

Training costs compute, and compute is roughly parameters times tokens. (Times about 6, if you want the exact napkin version.) That means you can spend a budget two ways: a big brain that studies a little, or a smaller brain that studies a lot.

The old advice, and the correction

In 2020 a paper from OpenAI (Kaplan et al.) found that a model's error drops smoothly and predictably as you add parameters, data, or compute. Great, you can plan with that. Their advice for a fixed budget was to go for a very large model, train it on a relatively modest amount of data, and stop before it fully converges.

Two years later, the Chinchilla paper (Hoffmann et al.) re-ran the question more carefully and came to a different answer: model size and training tokens should grow together. Double the model, double the data.

To prove it, they trained Chinchilla with about the same compute budget as Gopher, but spent it differently:

	Parameters	Training tokens	MMLU score
Gopher	280B	300B	60.0%
Chinchilla	70B	1.4T	about 67.5%

(MMLU is a big multiple-choice knowledge test. The paper's abstract says 67.5%, a table inside says 67.6%. Same story either way.) Chinchilla also beat GPT-3 (175B), Jurassic-1 (178B) and Megatron-Turing NLG (530B) on a wide range of tasks.

The conclusion in the paper: a lot of the big models of the time were undertrained. Too much brain, not enough studying.

The rule of thumb people took from it: roughly 20 tokens of training data per parameter. Chinchilla itself is 1.4T / 70B, which is 20.

A calculator

That rule plus the napkin formula gives you a tiny calculator. If compute is 6 × N × D and D = 20 × N, then compute = 120 × N². Flip it around:

from math import sqrt

def split_budget(compute_flops, tokens_per_param=20):
    n = sqrt(compute_flops / (6 * tokens_per_param))
    return n, tokens_per_param * n

for c in [1e22, 5.76e23, 1e24]:
    n, d = split_budget(c)
    print(f"{c:.2e} FLOPs -> {n/1e9:.1f}B parameters, {d/1e12:.2f}T tokens")

1.00e+22 FLOPs -> 9.1B parameters, 0.18T tokens
5.76e+23 FLOPs -> 69.3B parameters, 1.39T tokens
1.00e+24 FLOPs -> 91.3B parameters, 1.83T tokens

The middle line is the compute number from the Chinchilla paper's table. Out comes 69B parameters and 1.4T tokens. That's Chinchilla, from one line of arithmetic.

Now let me try it on my laptop

Billions of parameters isn't something I can do. But the idea, "for a fixed budget there's a best model size", should show up at any scale. So I made a toy version.

The setup:

• Data: about 12 million characters of Python source code from the standard library, with 5% held out to test on.
• Models: small GPT-style transformers working on single characters, from about 3 thousand to 3.2 million parameters.
• Budgets: four fixed compute budgets, from 1e11 to 3e12 FLOPs.
• The trick: for each budget and each model size, I worked out how many tokens fit in the budget (tokens = compute / (6 × parameters)) and trained exactly that much. Big model, little data. Small model, lots of data. Same cost.

35 training runs. Then I measured the loss on the text the models never saw (lower is better).

Each coloured line is one budget. The star is the winner.

What I saw:

1. The oversized models lost, badly. At the biggest budget, the 3.2-million-parameter model got a loss of 2.61. A model about 150 times smaller, trained on far more text, got 1.75. The big one only got to read about 155 thousand tokens. That's under 0.05 tokens per parameter, a huge brain with one skim of the book.

2. The winner got bigger as the budget grew. Best sizes were about 7.4k, 7.4k, 10.7k and 20.6k parameters, for budgets of 1e11, 3e11, 1e12 and 3e12.

3. The bottom of each curve is flat. At 1e12, models of 7.4k, 10.7k and 20.6k parameters scored 2.068, 2.039 and 2.047. You don't need the exact perfect size. You need to avoid being way too big.

budget (FLOPs)	best size (params)	tokens it trained on	held-out loss
1e11	7,408	2.2M	2.477
3e11	7,408	6.7M	2.282
1e12	10,688	15.6M	2.039
3e12	20,640	24.2M	1.750

That's the same shape the Chinchilla paper describes: for a fixed budget, too-big loses, and the best model grows as the budget grows.

What I am not claiming

This is a toy, so I want to be straight about it:

• I did not reproduce "20 tokens per parameter". My winners trained on hundreds or thousands of tokens per parameter. That's very different from 20. I think several things are mixed in: single-character text is a very different problem from words, my runs are tiny (from a few dozen to about twenty thousand training steps), and the position embeddings count as parameters, which is a big share of a 7k-parameter model. I didn't tune settings per model size either.
• The biggest budget's winner is at the edge. At 3e12, the best model was the smallest one I could afford to test, because even smaller ones would have needed more than about 2.5 passes over my data. So its true best size is probably lower, and I can't tell from this data.
• Some small models read the data more than once. For the very smallest ones I let the training text repeat up to about 2.5 times. That's harmless at these sizes, but it isn't the clean "one pass over fresh data" setup in the paper.
• One run per setting. No repeats, so small differences could be noise. The big gaps are not.

So treat this as "the pattern shows up even in a toy", not "I verified Chinchilla's numbers". For the actual numbers, read the paper.

Why you should care even if you never train a model

• Parameter count is a weak signal for quality. A smaller model that trained on a lot of data can beat a larger one that didn't. When you pick a model, the question "how much was it trained on?" matters as much as "how big is it?"
• It's the reference point for splitting a training budget. "Just make it bigger" stopped being the default answer, and "how much data does this size need?" became a standard question.
• And then it got stranger. Some labs now train small models on far more data than the 20x rule says, on purpose. That's the next post.

If you want to play with it, the whole experiment is about 100 lines of PyTorch. Change the data, the budgets or the sizes. I'd love to hear if you see the same curves, or different ones.