Your tuned model scored 0.82 on the internal eval and the training loss settled at 0.41. You retune on a new base, get 0.86 and 0.55, and somebody asks whether the model got better. The honest answer is that neither pair of numbers is a measurement of the same quantity, and the reason is worth understanding precisely rather than as a caution.
The baseline moved underneath the number
A fine-tune does not have a quality. It has a quality relative to the base model it started from. Everything you report about a tuned model — its eval score, its loss curve, the failure modes it no longer has — is a statement about the difference between a starting point and an end point, and when you change base models you change the starting point without changing the scale you are reporting on.
This is not a subtlety about measurement error. It is the same category of mistake as comparing two students’ improvement by looking only at their final exam marks. The final mark is real, but if one of them started three grades higher, the number does not tell you which teaching worked. With base models the gap is often large: a newer base can already do, untuned, most of what your previous tune was built to add. In that situation your retuned model can score higher on every eval and still be a worse decision than not tuning at all — a possibility your before-and-after numbers cannot express, because there is no cell in them for the new base untuned.
Why training loss is the worst offender
Training and validation loss are the numbers people reach for first, and they are the least portable thing in the whole report. Supervised fine-tuning loss is cross-entropy averaged per token. Two properties of that definition break the comparison.
The denominator is a tokenizer. Loss is a sum over tokens divided by a token count, and the token count for a fixed string is a property of the base model’s tokenizer, not of the string. Move to a base with a different vocabulary and identical training text becomes a different number of tokens, so the same modelling quality yields a different average. A vocabulary that encodes your domain more efficiently produces fewer, individually harder-to-predict tokens, which typically raises average loss on text the model handles better. The general mechanism is covered in how tokenizers differ; here the consequence is that a loss value without its tokenizer attached is not a quantity you can compare.
The support is a different vocabulary. Cross-entropy is computed over a distribution across the vocabulary. A model with a 200k-token vocabulary and one with a 50k-token vocabulary are being scored on differently sized prediction problems. Even at equal capability the numbers are not on one axis.
There is a third, more mundane reason: platforms do not all compute the reported loss the same way. Whether the prompt tokens are masked out of the loss, whether special tokens count, and how examples are packed into a batch all shift the reported number without shifting model quality. Training loss is a diagnostic for the run that produced it — is it descending, has it plateaued, has it started overfitting the validation split — and it is only that. Never put it in a comparison table across platforms or base models.
An eval score has two terms and reports one
A task eval is more portable than loss, because it is defined on your data and your grader rather than on the model’s internals. It is still confounded. Write the score on a tuned model as two terms:
score(tuned_on_B) = capability(B) + delta(tuning | B)
capability(B) what the base already does on your task
delta(tuning | B) what your tuning added, given that base
You measure the left-hand side. You care about the right-hand terms separately, and a single number does not separate them. Worse, the delta term is conditional on the base: the same dataset produces a large delta on a weak base and a small or negative one on a strong base, because there is less headroom and more prior behaviour to disturb. A tuning set that taught a smaller model your output format may, on a stronger base that already follows format instructions, do nothing except narrow its range — which shows up as a small eval improvement and a large regression on inputs your eval set does not contain.
The four cells you actually need
The fix is not statistical sophistication. It is measuring the two cells everyone skips. Freeze one held-out evaluation set and one grader, and run four models through it.
- A — old base, untuned. The original starting point. You may have to go and get this; most teams never measured it after the first week.
- B — old base, tuned. What you have been running in production.
- C — new base, untuned. The cell that decides whether you need a tune at all.
- D — new base, tuned. The candidate.
Now the arithmetic is interpretable. B - A is what your tuning was worth on the old base. D - C is what it is worth on the new one. C - B is the question that matters commercially: whether the new base, with no tuning, already beats the tuned model you run today. And D - B — the only comparison most migrations actually make — is a sum of a base change and a tuning change with no way to attribute it.
Two of these frequently come out in the direction nobody planned for. If C - B is positive, the correct outcome of the migration is to delete the fine-tune, along with its training data retention obligation and its access control problem. If D - C is near zero while B - A was large, your tuning was teaching capability the new base has, and continuing to tune buys you maintenance cost for nothing.
Run all four cells against the same frozen eval set, the same grader version and the same decoding settings, in the same window. If the grader is itself a model, pin its version — an updated grader silently rescales every cell and the differences stop meaning anything. See baseline drift in regression suites for why a moving grader is the failure that is hardest to notice.
Three things that quietly change the comparison
The prompt format is part of the model. Base models are trained with different chat templates, different system message handling and different tool-call encodings. Sending byte-identical prompts to both bases is not a fair test if one of them expects its instructions somewhere else; you are partly measuring how well your prompt happens to suit an unfamiliar template. Adapt the prompt to each base, and record that you did.
Decoding settings are not neutral. Temperature and nucleus sampling ranges are not defined identically across APIs, and a value copied across can land somewhere different on the new model’s distribution. Evaluate at the lowest-variance settings each API allows and hold them fixed across all four cells, and see what to do when a sampling control has no counterpart if one of them is simply absent on the new side.
Your eval set has aged into the training data. An eval set assembled two years ago, especially one built from public material, may be in a newer base model’s pretraining corpus. If cell C is startlingly good on public-looking items and ordinary on items drawn from your own recent traffic, treat that as a contamination signal and rebuild the set from recent production inputs.
What the four numbers let you decide
The output of this exercise is not a score. It is a decision with a reason attached, and there are only four of them: keep tuning on the new base because D - C is materially positive; stop tuning because C - B is positive and D - C is not; stay where you are because C and D are both worse than B; or rebuild the eval set because the cells disagree with what you see in production, which means the measurement, not the model, is what you have been managing.
Whichever it is, record the four numbers with the base model strings, the eval set version and the date beside them. The next base model change is not a hypothetical, and the single most useful thing you can leave behind is cell A and cell C — the untuned baselines — because those are the ones nobody thinks to measure until they need them and the model has been retired.
Top comments (0)