TL;DR — Post-training pipelines chain LoRA fine-tuning, preference optimization (DPO/RLHF), and distillation as if they're interchangeable quality knobs. They aren't. Low-rank adapters often lack the capacity to represent sharp preference distinctions, and distilling on teacher outputs alone throws away the exact ranking signal DPO was built to encode. The fix is matching adapter rank to objective and distilling on preference pairs or full logit distributions, not just hard labels.
Most post-training pipelines look the same: take a base model, run supervised fine-tuning with LoRA adapters for domain adaptation, run DPO or another preference optimization pass to align tone and safety, then distill the whole thing down to a smaller model for serving. Each stage is treated as an independent quality improvement, stacked like middleware. It isn't. These three techniques optimize different objectives over different distributions, and running them in sequence without understanding that interaction quietly cancels out the gains of the earlier steps.
The specific failure mode worth naming: distillation erases preference tuning, and low-rank adapters often can't hold preference signal in the first place. Both problems come from the same root cause — treating "make the model better" as one objective when it's actually three.
Three objectives, not one knob
Supervised fine-tuning with LoRA teaches a model what to say. It's behavior cloning: minimize the loss between the adapter's output distribution and a set of target completions. The objective is dense and local — every token has a correct next token, more or less.
Preference optimization teaches a model which of two responses to prefer. DPO, and RLHF more broadly, don't have a single correct completion to clone. They shift probability mass across an entire distribution of plausible outputs based on relative ranking. The signal is sparse and comparative, not dense and absolute.
Distillation teaches a smaller model to imitate a larger one. Standard implementations do this by training the student on the teacher's generated outputs — which collapses distillation back into supervised fine-tuning on synthetic data. That collapse is the trap. If your teacher's most valuable property is a preference-shaped distribution, and your distillation method only samples point completions from it, you are throwing away the shape and keeping only the samples.
Why LoRA rank fights the DPO objective
Low-rank adapters constrain the update to a thin subspace of the weight matrix. That's fine for SFT, where the target behavior is usually a stylistic or domain shift — vocabulary, format, tone — that lives comfortably in a low-dimensional update. It's a much rougher fit for preference optimization.
DPO's gradient wants to widen the gap between chosen and rejected completions across the full output distribution, which frequently requires reshaping probability mass in ways that don't compress into a rank-8 or rank-16 adapter. What you get instead is something that looks like preference tuning on your eval set — the handful of comparison pairs you tested — but reverts under distribution shift, because the adapter didn't have the capacity to encode a general preference direction. It memorized the specific contrast pairs in your training set rather than learning the underlying ranking function. This is preference collapse: strong offline eval numbers, weak generalization, and a model that reverts to base behavior the moment a prompt looks slightly different from anything in the DPO dataset.
The practical implication is that adapter rank should scale with objective complexity, not with a fixed budget you reuse across every fine-tuning stage. Domain-adaptation LoRA can run lean. Preference-optimization LoRA generally needs more capacity, applied to more layers, especially in the later blocks where output distributions actually get shaped. Treating rank as a single global hyperparameter set once for "the fine-tuning job" is a category error.
How distillation throws the signal away
Assume you've done the DPO stage correctly and have a model that reliably prefers concise, well-hedged, non-sycophantic answers. Now you distill it down for cheaper serving. The common approach: generate a large set of completions from the aligned model, then supervise-fine-tune a smaller model on those completions.
This works for surface behavior — the student will sound like the teacher on the training distribution. But it does not transfer the preference structure. The student never sees the rejected completions, never sees the margin between chosen and rejected, and never sees the teacher's calibrated uncertainty across close calls. It only sees winners. A model trained purely on winners tends to overfit to surface patterns of the winning style — certain phrasing, certain hedges — without inheriting the underlying discrimination ability. Push it slightly off-distribution and the "aligned" behavior degrades faster than the teacher's did, because the safety margin that DPO built in was never encoded in the training signal the student received.
This shows up as a specific, measurable pattern: a distilled model that passes your standard eval suite because the eval prompts resemble the teacher's generation distribution, but fails red-team or adversarial prompts that require the discrimination the teacher learned and the student never saw.
What to actually do about it
Treat distillation as a preference-transfer problem, not an output-cloning problem, whenever the teacher's value came from preference optimization. Two approaches work meaningfully better than output-only cloning:
Distill on the teacher's full output distribution — matching logits or top-k probabilities, not just the argmax completion — so the student inherits the shape of the teacher's confidence, including its uncertainty near decision boundaries.
Regenerate preference pairs using the teacher as the labeler, and run the student through its own DPO pass against those pairs, rather than a single SFT pass on teacher completions. This re-derives the ranking signal instead of assuming it survives compression.
On the LoRA side, size adapter rank and target modules to the objective, not to a convenient default. If preference optimization is layered on top of an existing LoRA-tuned checkpoint, consider whether the preference stage needs its own higher-rank adapter, or whether it needs to touch the base weights directly rather than compressing through the same low-rank subspace used for domain adaptation.
The sequencing question nobody asks
The deeper issue is that teams design these pipelines stage by stage, optimizing each step's eval in isolation, and never ask what the next stage does to the previous one's output distribution. LoRA compresses. DPO reshapes. Distillation compresses again. Each compression step is lossy with respect to the objective of the step before it unless you explicitly design the loss to preserve that signal.
None of this means avoid LoRA, avoid DPO, or avoid distillation. It means stop treating post-training as a linear checklist of quality upgrades and start treating it as a chain of lossy transformations, each of which needs to be told explicitly what to protect from the one before it. The order you run these stages in, and the specific loss you use at each handoff, matters more than which specific algorithm you pick within any single stage.
Top comments (0)