TL;DR — Teams treat SFT, preference optimization, and distillation as swappable, independently-testable modules in a post-training pipeline. They aren't. The order you apply them changes the final model in ways that per-step evaluation never catches, because LoRA weight deltas, preference gradients, and distillation targets don't compose linearly. Testing each stage in isolation gives you false confidence about the stacked result.
Most post-training pipelines are built like software pipelines: discrete stages, each with its own eval, each assumed to be independently swappable. Supervised fine-tune, then preference-optimize, then distill down to a smaller deployment model. Each stage passes its own benchmark. The team ships. Then the production model does something none of the individual stage evals predicted, and nobody can explain why.
The reason is simple and almost never discussed: these stages don't commute. The order you apply LoRA fine-tuning, DPO-style preference optimization, and distillation is not a pipeline convenience — it's a mathematical fact that changes the destination, not just the route.
The modular post-training myth
The mental model most teams use looks like function composition. SFT adapts the base model to a task. Preference optimization aligns it to human judgments. Distillation compresses it into something cheaper to serve. Each step has a clean before/after eval, so it's tempting to treat them as interchangeable Lego blocks — swap in a new preference dataset, re-run distillation, ship the delta.
That model works for data pipelines because transformations on structured data are mostly associative in practice, even when they're not in theory. It fails for post-training because every one of these stages is a gradient step through a non-convex loss landscape, and the landscape itself shifts depending on what you've already done to the weights.
Run SFT then DPO, and DPO's implicit reward model is shaped around the SFT model's output distribution — it's optimizing relative to a reference policy that already encodes the SFT biases. Run DPO on the base model first, then SFT on top, and you get a model where supervised gradients can actively undo the preference signal, because supervised loss doesn't know preference ordering exists. These are not the same model with different training schedules. They are different models that happen to share a benchmark score.
Why LoRA deltas aren't vectors you can add
LoRA's whole appeal is that it produces a low-rank weight delta you can store, swap, and in theory combine. The adapter-merging ecosystem has run with this: take a LoRA trained for style, another for domain knowledge, a third for safety behavior, and linearly combine them into one model. This works as a demo. It is not a guarantee.
A LoRA delta is a local linear approximation of a highly nonlinear update, valid near the point in weight space where it was trained. Two adapters trained independently were each optimized as if they were the only intervention on the base model. Add them together and you're assuming their gradients point into orthogonal, non-interacting subspaces of the weight matrix. Sometimes they do. Often they partially cancel, or worse, interact constructively in a direction neither training run intended — amplifying a behavior past where either adapter alone would have pushed it.
This is the same failure mode as naively averaging model weights across unrelated fine-tunes: it works until it doesn't, and you find out in production, not in eval.
Preference optimization moves the target, not just the weights
DPO and its relatives optimize relative to a reference policy — usually the SFT checkpoint. That reference isn't a fixed backdrop; it's baked into the loss function itself. Every preference-optimized model is implicitly a statement about a specific prior model's behavior.
Now distill that model. Standard distillation treats the teacher's output distribution as ground truth and trains a student to match it. But the teacher's distribution is the product of an optimization that was defined relative to an SFT checkpoint the student never saw. The student is learning to imitate the surface behavior of a preference-optimized policy without inheriting the reference frame that made that policy coherent. You get a student that mimics the teacher's token-level choices in-distribution and diverges sharply the moment a prompt pushes slightly outside the distillation set — because it never learned the preference ordering, only the teacher's realization of it on sampled prompts.
This is why distilled models so often regress on exactly the alignment properties the teacher was tuned for, even when offline distillation metrics look clean. The metric measures output match on a fixed set. It doesn't measure whether the underlying preference structure survived compression.
Distillation compounds the error, not cancels it
There's an intuition, borrowed loosely from ensemble theory, that distillation should smooth out idiosyncrasies and produce a cleaner, more generalizable student. That intuition assumes the teacher's errors are noise around a correct signal. In post-trained LLMs, the "errors" aren't noise — they're the accumulated, order-dependent artifacts of SFT-then-DPO-then-whatever-came-next. Distillation doesn't average those artifacts away. It transfers them, often with a sharper gradient signal than the original training produced, because cross-entropy against a confident teacher distribution is a stronger training signal than the original preference loss.
The practical result: a distilled model can be more confidently wrong about an edge case than its teacher was, simply because distillation sharpened a behavior the teacher held with low confidence into something the student holds with high confidence.
What this means for how you build the pipeline
Stop evaluating post-training stages as independent components with independent pass/fail gates. A LoRA adapter that scores well in isolation tells you almost nothing about how it behaves merged with two other adapters. A DPO checkpoint that improves preference win-rate against its own reference tells you nothing about what a distilled student built from it will preserve.
The fix isn't more evals per stage — it's evaluating sequences, not stages. Treat the full pipeline (SFT order, preference optimization order, distillation source) as the unit under test, and re-run end-to-end evaluation whenever any upstream stage changes, even if that stage's own benchmark didn't move. If you're merging LoRA adapters, test the merged behavior on prompts that probe interaction effects, not just the union of each adapter's individual eval set. If you're distilling a preference-optimized model, hold back a slice of the preference comparison data itself and check whether the student still respects those orderings, not just whether it matches teacher tokens.
None of this is exotic. It's the same discipline integration testing applies to software modules that technically pass their unit tests but break when wired together. Post-training pipelines just haven't adopted it yet, because the modularity illusion — SFT, DPO, distillation as clean, swappable stages — is seductive and mostly works, right up until the specific combination you shipped is the one where it doesn't.
Top comments (0)