Catastrophic Forgetting is Low-Rank: A Function-Space Theory for Continual Adaptation
📰 ArXiv cs.AI
Learn how catastrophic forgetting in continual adaptation can be understood through a function-space theory, helping to identify vulnerable output-space directions and predict forgetting vectors
Action Steps
- Apply the function-space theory to identify vulnerable output-space directions in your model
- Use the cross-task kernel to predict the forgetting vector before new-task training
- Analyze the NTK regime to understand the effects of new-task training on old-task prediction
- Implement a frozen-backbone linear classifier to test the theory
- Evaluate the performance of your model using the closed-form predictor for the forgetting vector
Who Needs to Know This
ML researchers and engineers working on continual learning and adaptation tasks can benefit from this theory to improve model performance and reduce forgetting
Key Insight
💡 Catastrophic forgetting can be understood and predicted through a function-space theory, allowing for more effective continual adaptation
Share This
💡 Catastrophic forgetting is low-rank! New function-space theory helps identify vulnerable directions & predict forgetting vectors #continuallearning #ML
Key Takeaways
Learn how catastrophic forgetting in continual adaptation can be understood through a function-space theory, helping to identify vulnerable output-space directions and predict forgetting vectors
Full Article
Title: Catastrophic Forgetting is Low-Rank: A Function-Space Theory for Continual Adaptation
Abstract:
arXiv:2606.18024v1 Announce Type: cross Abstract: Catastrophic forgetting in continual adaptation is usually studied through parameter drift, replay, or distillation, but these views do not identify which output-space directions are vulnerable. We give a function-space account in the NTK regime: new-task training induces old-task prediction drift through the cross-task kernel, yielding a closed-form predictor for the forgetting vector before any new-task gradient step. In frozen-backbone linear-
Abstract:
arXiv:2606.18024v1 Announce Type: cross Abstract: Catastrophic forgetting in continual adaptation is usually studied through parameter drift, replay, or distillation, but these views do not identify which output-space directions are vulnerable. We give a function-space account in the NTK regime: new-task training induces old-task prediction drift through the cross-task kernel, yielding a closed-form predictor for the forgetting vector before any new-task gradient step. In frozen-backbone linear-
DeepCamp AI