Why orthogonal projections eliminate catastrophic forgetting in autoregressive Transformers—and the mathematical reason behind the 62% downstream plasticity penalty.
The Continual Learning Dilemma in LLMs
Fine-tuning Large Language Models sequentially without retraining from scratch is notoriously broken. When you adapt an LLM to Task B after Task A, it systematically overwrites the weight configurations that enabled performance on Task A. This is Catastrophic Forgetting.
While Parameter-Efficient Fine-Tuning (PEFT) techniques like LoRA (Low-Rank Adaptation) isolate task-specific updates into small low-rank matrices:
$$\Delta W = \frac{\alpha}{r} B A$$
a critical bottleneck arises during deployment: How do you consolidate multiple sequential adapters into a single base model without incurring catastrophic interference?
Directly merging delta weights:
$$W_{\text{base}} \leftarrow W_{\text{base}} + \Delta W$$
destroys historical task performance.
To evaluate this frontier, we implemented and benchmarked Wake-Sleep Null-Space Consolidation (WS-NSC) on SmolLM-135M. The code is open-sourced, and the complete benchmark data is permanently registered under DOI 10.5281/zenodo.23763908.
Here is the empirical reality of what happens when you mathematically force parameter updates into the orthogonal null-space of prior activations.
The Mechanics of WS-NSC
The framework operates in a biologically inspired two-phase regime:
[Wake Phase: Online LoRA Training]
│
▼
[Activation Manifold Collection: C = Σ (x^T x) / M]
│
▼
[Sleep Phase: Null-Space Orthogonal Projection]
│
▼
[Consolidation: W_base ← W_base + ΔW @ P_null]
1. Wake Phase (Online Learning)
The model freezes base parameters $W_0$ and trains a standard LoRA adapter $(A_t, B_t)$ on the active task stream $\mathcal{D}_t$.
2. Manifold Estimation
Forward hooks register linear input activations $x \in \mathbb{R}^{B \times L \times d_{\text{in}}}$ across reference tokens and historical tasks. We compute the uncentered empirical second-moment covariance matrix:
$$C = \frac{1}{M} \sum_{i=1}^M x_i^\top x_i \in \mathbb{R}^{d_{\text{in}} \times d_{\text{in}}}$$
3. Eigendecomposition & Null-Space Projection
We extract eigenvectors $U$ and eigenvalues $\Lambda$ such that $C = U \Lambda U^\top$. We retain the top-$k$ principal components accounting for a variance threshold $\rho \in [0.90, 0.95]$:
$$\frac{\sum_{i=1}^k \lambda_i}{\sum_{j=1}^d \lambda_j} \ge \rho$$
The parameter update $\Delta W$ is then projected onto the exact null-space of $U_k$:
$$P_{\text{null}} = I - U_k U_k^\top$$
$$\Delta W_{\text{safe}} = \Delta W \cdot P_{\text{null}}$$
Because $x U_k \approx x$ for historical activations, projecting into $P_{\text{null}}$ ensures:
$$\Delta W_{\text{safe}} \cdot x^\top \approx 0$$
Theoretical guarantee: Prior task forward passes experience zero functional shift.
The Empirical Benchmark (SmolLM-135M)
We configured a 4-stage sequential continual learning setup:
- Invariant Anchor: WikiText-2 (Unsupervised language modeling)
- Task 1: GSM8K (Symbolic/mathematical reasoning)
- Task 2: IMDb (Sentiment classification)
- Task 3: AG News (Topic classification)
All experiments were executed on an A100 GPU using PyTorch AMP across all 7 projection matrices (q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj).
Benchmark Metrics
| Consolidation Strategy | Wake Gain (%) | Post-Sleep Gain (%) | Plasticity Retention | Forgetting (nats) | Relative Forgetting (%) | Anchor Drift $\Delta$ | Protected Rank ($k/d$) | Update Norm Kept |
|---|---|---|---|---|---|---|---|---|
| Direct-Merge | 16.35% | 16.32% | 99.8% | $+0.103$ | $+5.833\%$ | $+0.057$ | $0.000$ | $100.0\%$ |
| WS-NSC ($\rho=0.90$) | 15.29% | 5.76% | 37.7% | $\mathbf{-0.007}$ | $\mathbf{-0.390\%}$ | $\mathbf{-0.005}$ | $36.2\%$ | $73.9\%$ |
| WS-NSC ($\rho=0.95$) | 15.33% | 3.74% | 24.4% | $\mathbf{-0.007}$ | $-0.387\%$ | $-0.002$ | $54.5\%$ | $61.6\%$ |
Three Core Technical Takeaways
1. Catastrophic Forgetting is Completely Neutralized
In direct merging, catastrophic forgetting is glaring: backward tasks degrade by $+5.83\%$, and the general-domain anchor drifts by $+0.057$ nats.
Under WS-NSC ($\rho=0.90$), backward forgetting drops to $-0.39\%$ and anchor drift drops to $-0.005$ nats. Historical knowledge is preserved without storing or replaying raw pretraining datasets.
2. The Severe Plasticity Tax
While stability is solved, it incurs an acute functional cost. Under $\rho=0.90$, $36.2\%$ of the linear subspace is locked down, causing post-sleep task performance to retain only $37.7\%$ of what was acquired during the wake phase.
Pushing protection to $\rho=0.95$ locks $54.5\%$ of the rank, collapsing retained gains down to $24.4\%$. In deep nonlinear architectures, hard orthogonal projection chokes forward model capacity.
3. Weight Norm $\neq$ Functional Capability
A common assumption in model merging is that preserving weight magnitude preserves task utility.
Our data disproves this: retaining $73.9\%$ of parameter update norm resulted in retaining only $37.7\%$ of task gain. Spectral truncation clips the highest-energy activation vectors needed for downstream task execution.
Reproducibility & Open Source
The full codebase, automated plotting pipelines, and raw execution logs are open for external verification:
- GitHub Repository: https://github.com/AkshatRaj00/ws-nsc
- Permanent Scientific Record (DOI): 10.5281/zenodo.23763908
To reproduce the benchmark locally:
git clone https://github.com/AkshatRaj00/ws-nsc.git
cd ws-nsc
pip install -r requirements.txt
# Run empirical sequential evaluation
python src/ws_nsc.py --model HuggingFaceTB/SmolLM-135M --amp --batch-size 4 --eval-batch-size 4 --var-retention 0.90 0.95
# Generate loss trajectories & Pareto frontier plots
python src/plot_results.py
Citation
@article{raj2026wsnsc,
title={Wake-Sleep Null-Space Consolidation: Quantifying the Stability-Plasticity Frontier in Sequential LLM Adaptation},
author={Raj, Akshat},
year={2026},
publisher={Zenodo},
doi={10.5281/zenodo.23763908},
url={https://doi.org/10.5281/zenodo.23763908}
}
Dev.to / Medium Tags to Use:
machinelearning datascience ai deeplearning python
Top comments (0)