Less than expected, and the honest story is why that answer took three extra checks to trust. Across a 3x3 grid of LoRA rank and training steps on Qwen2.5-3B-Instruct, general-task retention never once dropped below the unadapted baseline.
Key takeaways
- The registered kill-gate for H1 (forgetting grows with adapter rank and training steps) did not fire: retention rose 9.1 percentage points between the smallest and largest configuration, the opposite of the hypothesis.
- Every one of nine grid cells showed general-task retention at or above the unmodified baseline, a pattern suspicious enough to investigate before trusting it.
- Two checks ruled out ordinary bugs: the raw generated text showed no parsing artifact, and reloading the identical unmodified model twice produced zero differences in its answers.
- The real gap was a missing seed control in the original grid. Retraining the two extreme configurations under new seeds showed the smallest configuration's exact 100% retention was a coincidence, while the largest configuration's gain replicated in three out of three independent runs.
- A configuration with meaningful domain accuracy gain and high general-task retention existed in 8 of the 9 tested combinations.
What did the experiment measure, and what did it show?
A LoRA adapter fine-tuned on a narrow domain is supposed to specialize a model without erasing what it already knew. I registered two hypotheses to test that. H1: forgetting on an unrelated general task grows with adapter rank and training steps. H2: a configuration exists that gains real domain accuracy while keeping general-task retention above 95%.
The domain task was Overruling, a binary legal classification: does a sentence overrule a prior case holding, yes or no. The general task was 120 fixed multiple-choice questions sampled across 57 MMLU subjects, evaluated identically before and after every fine-tune. The grid covered LoRA rank 4, 16 and 64 crossed with 50, 200 and 500 training steps on Qwen2.5-3B-Instruct, nine cells, one training run per cell.
KG2 fired comfortably: 8 of 9 cells reached over 10 percentage points of domain accuracy gain while keeping general-task retention above 95%. KG1 did not fire at all. Retention moved from 100.0% at the smallest configuration to 109.1% at the largest, a rise, not the registered 5-point drop that would have confirmed forgetting scales with rank and steps.
What ruled the pattern out as a bug?
A 9-point rise sounds like good news, but the full table was stranger than one rise: all nine cells landed at or above the 100% baseline, never once below it. Under ordinary sampling noise on a 120-question eval, roughly half the cells should have landed under the baseline by chance alone. A number that only ever moves one direction is a reason to check the method before writing the conclusion.
The first suspect was parsing: if the unmodified model answered more verbosely and the scoring regex failed to extract a letter from it, that alone could manufacture a fake accuracy gap. Comparing the raw generated text from the baseline model against the highest-gain configuration on the same 120 questions ruled this out: both answered in a clean one-letter format almost every time, 1 unparseable reply out of 120 for the baseline and 0 out of 120 for the adapted model.
The second suspect was the 4-bit quantized loading itself. Every cell reloaded the base model fresh from the same checkpoint, and quantization kernels can in principle introduce small numerical differences between loads. Loading the unmodified model twice, independently, and comparing its raw answers on the same 120 questions showed zero differences. Loading and quantization were fully deterministic here, and could not explain the pattern either.
What did reseeding the two extreme configurations show?
The original grid trained each of the nine cells exactly once, with no fixed seed for the LoRA adapter's random initialization. That is the real gap, not a bug but a missing control. I retrained the smallest configuration (rank 4, 50 steps) and the largest (rank 64, 500 steps) under two new explicit seeds each.
The smallest configuration's retention landed on both sides of the baseline across the new seeds, 98.5% and 104.5%, consistent with a near-zero true effect and explaining why the original single run happened to land on exactly 100.0% by coincidence. The largest configuration stayed above baseline in both reseeded runs, 110.6% and 106.1%, for three out of three independent trainings above 100% in total. That pattern replicates. It is not the artifact of one lucky run, and it is the opposite of what H1 predicted. The two cells chosen were the extremes of the grid on purpose: if the real effect were forgetting that scales with rank and steps, the gap between them was the place most likely to show it clearly, and it still did not.
What does this mean for fine-tuning a narrow domain?
Within this tested range, LoRA fine-tuning on Overruling did not cost general capability, and the strongest configuration mildly, reproducibly improved it instead. I do not have a confirmed mechanism for why. A plausible but untested explanation is that training toward a short, committed answer format transfers a general "commit to an answer" behavior that happens to help multiple-choice scoring too, independent of any actual knowledge gained. A control run, fine-tuning on irrelevant or randomly labeled data to isolate whether the effect comes from this domain specifically or from LoRA training in general, was not part of this registered run. It is the obvious next step, not a claim this one makes.
The practical lesson is less about LoRA and more about measurement discipline: a result that looks too clean, with every cell moving the same direction, earned three separate checks before it earned a sentence in this article. Two of those checks found nothing. The third found a real design gap and, underneath it, a real effect.
FAQ
Does this mean LoRA fine-tuning never causes forgetting?
No. It means forgetting was not observed in this specific range: one 3B model, LoRA rank up to 64, up to 500 steps, one narrow binary-classification domain. A longer generative domain task, a different model family, or a larger rank could behave differently.
Why only 120 general-task questions?
It kept nine training-and-evaluation cycles plus the follow-up reseeding runs inside a reasonable time on a single consumer GPU. The tradeoff is coarseness: 120 questions across 57 MMLU subjects cannot detect forgetting that is small and spread thinly across many subjects.
What is the best explanation for the improvement?
No confirmed one. The leading untested hypothesis is a general answer-format effect from fine-tuning toward short, committed responses, rather than genuine new general knowledge.
Will the full grid be rerun with multiple seeds?
Not committed yet. The two-cell reseed check here was a targeted fix for the specific anomaly this run produced, not a full redo of all nine cells.
References
- Yifeng Xiong, Xiaohui Xie. OPLoRA: Orthogonal Projection LoRA Prevents Catastrophic Forgetting during Parameter-Efficient Fine-Tuning, 2026. arxiv.org/abs/2510.13003
- Seungju Back et al. Understanding LoRA as Knowledge Memory: An Empirical Analysis, 2026. arxiv.org/abs/2603.01097
Top comments (0)