Orthogonal Quadratic Complements for Vision Transformer Feed-Forward Networks

arXiv:2604.09709v1 Announce Type: cross Abstract: Recent bilinear feed-forward replacements for vision transformers can substantially improve accuracy, but they often conflate two effects: stronger second-order interactions and increased redundancy relative to the main branch. We study a complementary design principle in which auxiliary quadratic features contribute only information not already captured by the dominant hidden representation. To this end, we propose Orthogonal Quadratic Complemen