Teacher Supervision over Representation Equivalence Classes

1. 1Identify the representation equivalence class of a pretrained teacher model using orthogonal-and-isotropic-scaling transformations
2. 2Apply teacher supervision over the equivalence class to guide the training of a student model
3. 3Compare the performance of the student model trained with equivalence class supervision to one trained with traditional logits or feature matching
4. 4Configure the student model to learn the teacher's equivalence class rather than its absolute features
5. 5Test the robustness of the student model to different orthogonal-and-isotropic-scaling transformations of the teacher's representation

ML researchers and engineers working on knowledge distillation can benefit from this approach to improve the performance of their student models

💡 A pretrained representation is identifiable only up to an orthogonal-and-isotropic-scaling equivalence class, so supervising over the equivalence class can improve knowledge distillation

🤖 Improve knowledge distillation by supervising over representation equivalence classes rather than absolute coordinates! #AI #ML