EMA on LoRA ? [R]
📰 Reddit r/MachineLearning
Explore EMA on LoRA adapters for self-teaching and soft label generation, and discover papers with successful empirical results
Action Steps
- Search for papers on arXiv and other academic databases using keywords like 'EMA on LoRA' and 'self-teaching adapters'
- Implement on-policy self-distillation using EMA as a teacher for generating soft labels
- Experiment with fine-tuning and adapter-based training to compare results
- Evaluate the performance of EMA on LoRA adapters using metrics like accuracy and loss
- Compare the results with other self-teaching methods and adapter architectures
Who Needs to Know This
ML researchers and engineers can benefit from this knowledge to improve their model training and adaptation techniques, especially when working with large language models and adapters
Key Insight
💡 Using EMA on LoRA adapters can potentially improve model training and adaptation by generating soft labels for self-teaching
Share This
Explore EMA on LoRA adapters for self-teaching and soft label generation #MachineLearning #LoRA
Key Takeaways
Explore EMA on LoRA adapters for self-teaching and soft label generation, and discover papers with successful empirical results
Full Article
Hi guys Does anyone know of papers where EMA on LoRA adapters has been used successfully? Im interested in cases where the EMA adapter acts as a self-teacher generating soft labels for the trainable adapter. On-policy self-distillation [1] uses ema for the teacher. However, they seem to fully fine-tune. Any empirical results showing the idea is working on lora/ left models? [1] https://arxiv.
DeepCamp AI