A PyTorch-based optimizer wrapper for continual learning via selective fine-tuning, guided by the condition number kappa (Îș) of model tensors. KappaTune identifies and updates only the least anisotropic parameters to preserve pre-trained knowledge and mitigate catastrophic forgetting.
kappaTune is designed to address the challenge of catastrophic forgetting in continual learning scenarios through the analysis of the condition numbers of a neural network's weight matrices. This approach updates only tensors with the smallest condition numbers due to a synergy of factors: their inherent numerical stability makes them less susceptible to training noise, and their less specialized nature allows for robust adaptation without overwriting critical, highly specific pre-training knowledge, thereby effectively mitigating catastrophic forgetting of foundational capabilities, as shown in the paper.
Top comments (0)