Training Transformers in Cosine Coefficient Space

1. 1Parameterize weight matrices of a transformer in the 2D discrete cosine transform (DCT) domain
2. 2Retain only the lowest-frequency coefficients to reduce dimensionality
3. 3Reconstruct the full weight matrix via the inverse DCT at each forward pass
4. 4Update the spectral coefficients directly through backpropagation

ML researchers and engineers working on transformer models can benefit from this approach to improve model performance and efficiency, and it can be applied to various NLP tasks

💡 Parameterizing weight matrices in the DCT domain can improve transformer model performance

💡 Train transformers in cosine coefficient space for improved performance