Training LLMs: Pretraining, Loss Curves, and Scaling Laws
Every language model you have ever used is a pile of numbers shaped by one process: predict the next token, check how wrong the guess was, adjust the numbers, repeat a few trillion times. The process is simple to describe and expensive to run. GPT-3 cost about 3.14 x 1023 floating point operations. Llama 2 70B took 1,720,320 GPU hours on A100 hardware. Those numbers are so large they stop meaning anything, so we did what this series always does: we shrank the problem until we could measure it ourselves. We trained three tiny decoder transformers on the text of our own published articles, 539,064 characters from 53 posts, and recorded every loss curve, every learning rate, and every repeated epoch. The small models behaved like the big ones in the ways that matter, and they surprised us in two places where the textbooks are quieter than they should be.
This is Deep Part 3 of the AI Engineer Deep series. Deep Part 1 built attention by hand, and Deep Part 2 built a tokenizer from scratch and measured what vocab size actually buys. This part takes the tokens those two parts produced and trains on them. If you have not read them, the short version is this: attention is how a model looks back at earlier tokens, and the tokenizer decides what the model ever gets to see. Training is what happens next.
The 30-second answer
Pretraining is next-token prediction at scale. The model reads a chunk of text, guesses the next token, and the loss is the cross-entropy between its guess and the real token, measured in nats. A lower loss means the model is less surprised by real text. How fast that loss falls is governed by scaling laws: Kaplan and colleagues at OpenAI showed in 2020 that test loss falls as a power law in parameters, data, and compute, and DeepMind corrected the recipe in 2022 with Chinchilla, showing that parameters and training tokens should grow together at about 20 tokens per parameter. Most famous models before Chinchilla were too big and too hungry for their data. GPT-3 had 175 billion parameters and saw only 300 billion tokens, 1.7 tokens per parameter. Chinchilla had 70 billion parameters, saw 1.4 trillion tokens, and beat the 280 billion parameter Gopher on MMLU, 67.6 percent against 60.0 percent, on the same compute budget. Our own runs repeated the same mistake at toy scale, and the loss curves show exactly where the compute went.
What pretraining actually is
Strip away the infrastructure and pretraining is one loop. Take a sequence of tokens, hide the last one, ask the model to predict it, measure the error, and update the weights so the next guess is slightly better. The error measure is cross-entropy loss: the negative log probability the model assigned to the true token, averaged over the sequence. If the model gives the true token a probability of 0.5, that token costs 0.69 nats. A probability of 0.1 costs 2.30 nats. A model that guesses at random over our 112-character vocabulary scores ln(112), which is 4.72 nats. That number is the floor every training run starts from, and it is the first thing to check when a run begins: if the loss does not start near ln(vocab size), something is wrong before the first update.
Three ingredients decide the final loss: the parameter count N, the training tokens D, and the compute C. Kaplan et al. fitted power laws for each one, holding the other two out of the way: loss falls as N to the power minus 0.076, as D to the power minus 0.095, and as compute to the power minus 0.050. The exponents are small, and that is the first hard truth of this field. Doubling the parameters buys a few percent of loss, not half. The second hard truth is the compute rule that ties the three together: training a transformer costs about 6ND floating point operations, six times parameters times tokens. The 6 comes from the forward pass (two operations per parameter per token) plus the backward pass (four more). Check it against the published numbers and it holds: 6 times 175 billion times 300 billion is 3.15 x 1023, and the GPT-3 paper reports 3.14 x 1023. For Llama 2 70B, 6 times 70 billion times 2 trillion is 8.4 x 1023.
ModelParametersTraining tokensTokens per parameterTraining compute (FLOPs)
GPT-3 (2020)175B300B1.73.14 x 1023
Gopher (2021)280B300B1.1About 5.0 x 1023
Chinchilla (2022)70B1.4T20.0About 5.8 x 1023
Llama 2 70B (2023)70B2.0T28.68.4 x 1023
Read the tokens per parameter column and the whole history of scaling falls out of it. GPT-3 and Gopher were starved: one token per parameter is nowhere near enough to teach 175 billion weights anything durable. Chinchilla fixed the ratio to 20 and won. Llama 2 pushed past it to 28.6, deliberately overtraining a smaller model because a smaller model is cheaper to serve for years after training ends. The training bill is paid once. The inference bill is paid forever, and Meta chose to pay more upfront to pay less later. That is a product decision wearing a scaling law costume, and it is the right way to read these numbers: the law tells you the cheapest way to reach a loss, and your serving budget tells you whether to follow it.
The experiment: three tiny transformers on our own articles
We trained three decoder-only transformers in PyTorch on a single CPU, using the same recipe for each: character-level tokens from the text of our 53 published articles, 485,157 training tokens and 53,907 held-out validation tokens, block size 64, batch size 16, AdamW with weight decay 0.01, 20 warmup steps, cosine decay to zero, gradient clipping at 1.0, and 400 steps each. The only thing that changed was the model: 2 layers at width 96, 4 layers at width 192, and 4 layers at width 256. Character level keeps the vocabulary at 112 and the models small enough to train in minutes, and it keeps the comparison honest because all three models see exactly the same data in exactly the same order.
ModelParametersStepsTokens seenWall timeThroughputTrain lossVal lossCompute (FLOPs)
Small, 2 layers, width 96251,520400409,60016.3 s25,135 tok/s2.47692.51016.18 x 1011
Medium, 4 layers, width 1921,835,136400409,60096.5 s4,245 tok/s2.23222.30274.51 x 1012
Large, 4 layers, width 2563,233,280400409,600164.6 s2,489 tok/s2.16712.18217.95 x 1012
For scale, our largest run used 7.95 x 1012 FLOPs. GPT-3 used 3.14 x 1023, about 40 billion times more. The gap is the point: nothing here is a frontier result, and everything here is a real measurement of the same dynamics.
What the loss curves showed
All three curves have the same shape, and it is the shape every practitioner learns to read. The loss falls off a cliff in the first 50 steps, from 4.9 nats to about 2.8, then grinds down slowly for the remaining 350 steps. The first 50 steps teach the model the alphabet, the spaces, and the most common words. Everything after that is the long tail: rarer words, longer dependencies, the difference between plausible text and our text. Here is the small model curve, loss sampled every 50 steps: 4.94, 2.88, 2.67, 2.81, 2.60, 2.52, 2.48, 2.54, 2.43. Notice step 150, where the loss jumps back up to 2.81 from 2.67. That is not a bug. That is a batch of hard text, or a step where the optimizer overshot, and the curve recovers within 50 steps. A loss curve that never bumps is a loss curve that is not being measured honestly, or a model too small to notice its own data. Ours bumps, and then it keeps falling, which is what matters.
The headline result is the validation loss column: 2.51, 2.30, 2.18. Bigger helped, every time, exactly as Kaplan predicts. But look at the price. Going from the small model to the large one multiplied the parameters by 12.9 and the compute by 12.9, and bought 0.33 nats of validation loss. In perplexity, which is e raised to the loss and reads more like ordinary intuition, the models went from 12.3 to 10.0 to 8.9, against a random baseline of 112. The first 7.3 times the parameters bought 0.21 nats. The next 1.8 times bought 0.12. The returns are shrinking exactly the way a power law with a small exponent says they should, and anyone planning a training budget should stare at that column until it sinks in: most of the gain comes early, and the rest is bought at full price.
Then we did the Chinchilla arithmetic on our own runs, and it stung. For a fixed compute budget C, Chinchilla says the optimal split is about 20 tokens per parameter, which works out to N equal to the square root of C divided by 120. Our small run spent 6.18 x 1011 FLOPs, so its optimal model was about 72,000 parameters trained on 1.44 million tokens. We trained 251,520 parameters on 409,600 tokens: 3.5 times too big, on 3.5 times too little data. Our large run spent 7.95 x 1012 FLOPs, so its optimal model was about 257,000 parameters on 5.15 million tokens. We trained 3.23 million parameters on 409,600 tokens: 12.6 times too big, on 12.6 times too little data. We made the GPT-3 mistake at toy scale, in miniature, on purpose and by accident at the same time: we fixed the step count and let the parameter count do the growing, which is exactly what the field did from 2020 to 2022. The loss still fell when we scaled up, because more parameters always help when data is held fixed, but the compute would have bought more loss reduction spent on tokens. If we run this experiment again, the fourth model will be 250,000 parameters trained for 5 million tokens, and we expect it to beat the 3.2 million parameter model on the same FLOPs. That is a prediction, written down before the run, which is the only honest way to make one.
Learning rate: too slow, just right, and surprisingly fine
We swept the learning rate on the small model, 200 steps each, everything else fixed. At 3 x 10-4 the model finished at a validation loss of 3.14, barely better than halfway down the cliff: the steps were too small to use the budget. At 3 x 10-3 it finished at 2.65, the best of the three. At 3 x 10-2, ten times hotter, it finished at 2.67, essentially tied. That last number surprised us, and we checked it twice before believing it, because every training guide warns that a learning rate that high should diverge. It did not diverge here, and the reason is instructive: AdamW normalizes each update, gradient clipping capped the worst steps at a norm of 1.0, and 20 warmup steps let the optimizer find its footing before the full rate kicked in. At 250,000 parameters, on a forgiving character task, the safety nets held. Do not take this as permission to train a 70 billion parameter model at 3 x 10-2. The lesson is narrower and more useful: learning rate sensitivity grows with model size and with task difficulty, and the failure mode at small scale is wasted budget (too cold) rather than explosion (too hot). The standard practice exists for a reason: warm up, decay with a cosine schedule to near zero, and pick the rate by sweeping on a small model first, exactly as Kaplan did when the schedule in that paper used 3,000 warmup steps and cosine decay. The schedule is not decoration. In our runs the cosine tail is where the last 0.1 nats came from, as the shrinking steps let the model settle into the minimum instead of orbiting it.
Learning rateStepsTrain lossVal lossVerdict
3 x 10-42003.09393.1440Too cold, budget wasted
3 x 10-32002.67432.6486Best of the three
3 x 10-22002.64482.6732Stable, but no better
Repeating data: eight epochs and the loss kept falling
The second surprise came from repetition. We took a fixed 50,000 token slice of our articles and trained the small model on it for eight full epochs, measuring validation loss after each one. The folklore says repeating data is wasteful, that a model should see fresh text or nothing. The measured numbers say otherwise at this scale: validation loss fell every single epoch, from 2.84 after epoch 1, to 2.76, 2.68, 2.61, 2.59, 2.51, 2.49, and 2.43 after epoch 8. Eight passes over the same 50,000 tokens bought 0.41 nats, and the curve was still falling when we stopped.
Does this contradict Muennighoff and colleagues, who showed in 2023 that repeating data stops paying off after about four epochs? No, and the reason is the difference between underfitting and memorizing. Their result, fitted on models from 10 million to 9 billion parameters, is that up to four epochs on fixed data costs almost nothing against fresh data: an 8.7 billion parameter model trained for four epochs on 44 billion unique tokens finished only 0.5 percent higher in validation loss than a model trained once on 178 billion unique tokens. Past about 16 epochs the value of each repeated token decays toward zero. Our model is 250,000 parameters, and 50,000 tokens is far more text than it can memorize, so at epoch 8 it is still learning general patterns from the slice, not reciting it. The generated sample proves it: after eight epochs the model writes word-like text with real words mixed into the nonsense, "Teabe, mod allyspounchminge se on is reatusitche 4 and in a the a seoutersis bres you model it curses ase minl seand to a mevery with atiblerestery a pulale veise topt's aut ogleves in you rew a rictit". It has learned spelling and spacing and the rhythm of English, and nothing else, which is exactly what eight epochs over 50,000 tokens should teach a quarter-million parameters. The practical rule we take from both results is this: repeat data freely while the model is small relative to the data, watch the validation loss, and stop repeating when the validation curve flattens while the training loss keeps falling, because that gap is memorization, and memorization is the tax you pay for too many epochs.
How to read a loss curve like an engineer
A loss curve is a diagnostic instrument, and most of its value is in the failure modes. A curve that falls fast then flattens is normal: the easy structure goes first. A sudden vertical spike that recovers within a few dozen steps is usually one bad batch, and it is harmless if it recovers. A spike that does not recover, or a loss that climbs for hundreds of steps, means the learning rate is too high or the data has a poisoned batch, and the fix is to cut the rate and check the data, not to add more steps. A loss stuck at ln(vocab size) means the model is not learning at all: check that the targets are shifted by one, that the loss is actually connected to the parameters, and that the optimizer is stepping. A loss of NaN means the numbers have overflowed, usually from a learning rate far too high or a division by zero in the normalization, and the run is dead: restart from the last checkpoint with a lower rate. A training loss that falls while the validation loss rises is overfitting, and the fixes in order are more data, fewer epochs, stronger regularization, or a smaller model. And a curve that looks too smooth is suspicious: real training is noisy, and heavy smoothing hides the spikes that tell you something changed. Keep the raw curve, smooth a copy for presentation, and never make a decision from the smoothed version alone.
One more thing the curve cannot tell you: whether the model is any good. Loss is a single number about next-token surprise, and it says nothing about whether the model answers questions, writes working code, or tells the truth. That is why the field built evals, and why Part 5 of the original series, on evals, matters more the further you go. A loss of 2.18 on our articles is a real measurement, and it is also a number about a model that writes "allyspounchminge". Train for the loss, but judge by the evals.
What this means if you train anything
Four rules survived contact with our own measurements. First, budget in FLOPs, not in vibes: compute is 6ND, so write the number down before the run starts and check that the parameter count and the token count are both earning their share. Second, follow the Chinchilla ratio as a starting point, about 20 tokens per parameter, and only depart from it deliberately, the way Llama 2 did for inference cost. Third, sweep the learning rate on the smallest model that shows the dynamics, then scale the winning recipe up, because the rate that wastes a small budget will waste a large one faster. Fourth, treat the loss curve as a health monitor, not a scoreboard: watch for spikes, plateaus, and the gap between train and validation, and keep the evals separate from the training loss entirely. Pretraining is the most expensive step in the whole LLM pipeline, and the most common failure is not a bad architecture. It is a good architecture fed the wrong ratio of parameters to data, at the wrong learning rate, with nobody watching the curve until the bill arrives. Our three toy models cost a few minutes of CPU and taught us that lesson for free. The teams training the real ones paid millions to learn it.
What is next in this series
This is Deep Part 3 of the AI Engineer Deep series, a line by line revisit of the original 8 part series with real research and real measurements. Deep Part 4 takes on LLM inference: KV cache, sampling, and temperature measured, including what temperature actually does to output quality and what the cache costs in memory. The original series still stands as the map: Deep Part 2, Tokenization is where the tokens in this article came from, Part 3, How LLMs Actually Work is where tokens and temperature first appeared, and Part 8 on LLMOps shows what the training bill turns into once the model ships. If you want the cost side of this story now, our measured guide to cutting the LLM API bill puts numbers on five techniques.
Previous: Tokenization: BPE from Scratch, Vocab Size, and What Tokenizers Hide
Next: Deep Part 4: LLM Inference: KV Cache, Sampling, and Temperature Measured (coming next in this series)
Originally published on LogicStack. Run the code examples right in your browser with our free interactive Python and SQL playgrounds — no setup, no sign-up.
Top comments (0)