Low-Dimensional and Transversely Curved Optimization Dynamics in Grokking

1. 1Apply PCA to attention weight trajectories to identify low-dimensional execution subspaces
2. 2Analyze the variance captured by principal components to understand optimization dynamics
3. 3Probe the geometry of the execution subspace to gain insights into grokking
4. 4Investigate the implications of low-dimensional optimization dynamics for transformer training and generalization

ML researchers and AI engineers benefit from this study as it provides insights into the training dynamics of transformers, which can inform the development of more efficient and effective AI models

💡 Optimization dynamics in transformers evolve predominantly within a low-dimensional execution subspace

🤖 Grokking in transformers: low-dimensional execution subspaces revealed through geometric analysis