Anytime Training with Schedule-Free Spectral Optimization

📰 ArXiv cs.AI

Learn to train neural networks without fixed learning-rate schedules using SF-NorMuon, a schedule-free spectral optimizer, to improve performance and reduce re-tuning costs

advanced Published 25 May 2026

Action Steps

Implement SF-NorMuon in your neural network training code
Compare the performance of SF-NorMuon with well-tuned AdamW baselines
Apply SF-NorMuon to datasets with varying availability to demonstrate its schedule-free benefits
Test the robustness of SF-NorMuon to different hyperparameter settings
Configure SF-NorMuon to work with your existing neural network architecture

Who Needs to Know This

Machine learning engineers and researchers can benefit from this technique to improve the efficiency and scalability of their neural network training pipelines

Key Insight

💡 SF-NorMuon closes the performance gap between schedule-free and well-tuned AdamW optimizers

Key Takeaways

Learn to train neural networks without fixed learning-rate schedules using SF-NorMuon, a schedule-free spectral optimizer, to improve performance and reduce re-tuning costs

Full Article

Title: Anytime Training with Schedule-Free Spectral Optimization

Abstract:
arXiv:2605.23061v1 Announce Type: cross Abstract: Standard neural network training relies on learning-rate schedules tied to a fixed horizon, leading to strong path dependence and costly re-tuning as data availability changes. Schedule-Free (SF) methods address this by removing explicit schedules, yet SF-AdamW, the current state-of-the-art anytime optimizer, consistently underperforms well-tuned AdamW baselines. We propose SF-NorMuon, a schedule-free spectral optimizer that closes this gap: with

Read full paper → ← Back to Reads