Scaling laws: why model loss is a straight line on a log-log plot, and why GPT-3 was undertrained

You'd expect training a bigger language model to be a gamble. It isn't — and that is one of the most useful facts in modern AI. Across many orders of magnitude, a model's test loss falls as a smooth power law in the compute you spend: plot loss against compute on log-log axes and the points fall on a straight line. I built two demos that compute the real curves in the browser, and together they explain both why labs can plan nine-figure training runs and why GPT-3, of all things, was "undertrained."

Loss is a power law down to an irreducible floor

The full law has two parts: a reducible term that shrinks as a power of scale, plus an irreducible floor L₀ you can never beat. That floor is the intrinsic entropy of language — even a perfect model can't predict the next token with certainty.

def loss(C, L0, Cc, alpha):
    return L0 + (C / Cc) ** (-alpha)
# reducible part -> 0 as C grows, so loss -> L0 (the floor)

The reducible term (C/Cc)^(−α) is a straight line on a log-log plot; as compute grows the curve rides that line, then bends flat onto L₀. Steeper α means more gain per FLOP. The practical payoff is extrapolation: fit a few small, cheap runs, subtract the floor, fit a line in log-log space, and you can forecast a model 100× or 1000× bigger before paying for it — the exact trick used to de-risk frontier runs.

C    = np.array([1e17, 3e17, 1e18, 3e18, 1e19])
loss = np.array([3.10, 2.74, 2.45, 2.22, 2.04])
L0   = 1.69
alpha, logCc = np.polyfit(np.log(C), np.log(loss - L0), 1)
alpha = -alpha                        # the slope is -alpha

But respect the floor. As C grows the reducible part vanishes and predictions asymptote to L₀ — never predict below it, and treat far extrapolations with care, because the fit itself can bend or break outside the range you measured. My first demo lets you drag compute along the curve and watch the returns-per-FLOP collapse as you approach the floor.

The three knobs, and C ≈ 6ND

Three quantities drive everything: N parameters, D training tokens, C compute (FLOPs). They're linked by a rule of thumb — a forward+backward pass over the data costs about C ≈ 6·N·D. So a fixed compute budget is a curve of (N, D) trade-offs: a bigger model means fewer tokens you can afford, and vice-versa. Which raises the real question: for a fixed budget, how should you split it?

Kaplan said scale N. Chinchilla said grow both.

The first scaling-laws paper (Kaplan et al., 2020) concluded you should spend most new compute making the model bigger, growing data only modestly. GPT-3 (175B params, ~300B tokens) is the poster child of that era. Then DeepMind's Chinchilla (2022) re-fit the loss surface over both size and data and showed Kaplan had undertrained everyone.

# Chinchilla parametric fit:
#   L(N, D) = E + A / N^alpha + B / D^beta
E, A, B     = 1.69, 406.4, 410.7
alpha, beta = 0.34, 0.28

Neither term wins alone — starve either N or D and the loss climbs. Minimise that surface at fixed C = 6ND and the optimum grows N and D at nearly the same rate, so their ratio stays roughly constant at about 20 tokens per parameter. That single number is the headline result.

def optimal_split(C):
    best = None
    for r in np.logspace(-1, 3, 400):     # try many tokens/param ratios
        N = np.sqrt(C / (6*r)); D = np.sqrt(C * r / 6)
        L = loss(N, D)
        if best is None or L < best[0]: best = (L, N, D, r)
    return best                            # r* lands near ~20 : 1

Why "GPT-3 was undertrained"

Put GPT-3 on the curve: 175B params but only ~300B tokens ≈ 1.7 tokens per parameter — deep in the undertrained zone, far left of the ~20 optimum. The precise sense of "undertrained" is not that GPT-3 was bad; it's that the same compute could have bought a better and cheaper model. Chinchilla proved it directly by training a 70B model on 1.4T tokens (20:1) on the same compute and beating the 280B Gopher — a model 2.5× smaller and far cheaper to serve. My second demo is a U-shaped loss-vs-ratio curve with a marker you can jump to GPT-3, to Chinchilla-optimal, or over-fed, and read off the loss each way.

Know the limits

Scaling laws are a forecast, not a guarantee. Loss can't drop below L₀; the compute-optimal recipe eventually wants more tokens than exist (the data wall — synthetic data is one answer); the laws predict loss, not downstream capability, which can jump in surprising ways; and modern models (Llama-style) deliberately train past 20 tokens/param because they optimise for cheap inference, not training-optimal loss. The lesson is not "make it bigger" — it's "grow N and D together, and know when the curve flattens."

assert predicted_loss > L0             # can't beat the floor
assert 20 * N < tokens_available       # else you hit the DATA WALL

Drag the compute and the N/D split, and watch the curves:
https://dev48v.infy.uk/ai/days/day40-scaling-laws.html