Interpreting Video Representations with Spatio-Temporal Sparse Autoencoders
📰 ArXiv cs.AI
Spatio-Temporal Sparse Autoencoders improve video representation interpretation by recovering temporal coherence
Action Steps
- Apply standard Sparse Autoencoders (SAEs) to video representations to decompose them into interpretable features
- Use spatio-temporal contrastive objectives to improve temporal coherence
- Implement Matryoshka hierarchical grouping to further enhance feature assignments and autocorrelation
Who Needs to Know This
Machine learning researchers and engineers working on video analysis tasks can benefit from this study to improve their models' performance and interpretability
Key Insight
💡 Spatio-temporal contrastive objectives and hierarchical grouping can recover and exceed raw temporal coherence in video representations
Share This
📹 Improve video representation interpretation with Spatio-Temporal Sparse Autoencoders!
Key Takeaways
Spatio-Temporal Sparse Autoencoders improve video representation interpretation by recovering temporal coherence
Full Article
Title: Interpreting Video Representations with Spatio-Temporal Sparse Autoencoders
Abstract:
arXiv:2604.03919v1 Announce Type: cross Abstract: We present the first systematic study of Sparse Autoencoders (SAEs) on video representations. Standard SAEs decompose video into interpretable, monosemantic features but destroy temporal coherence: hard TopK selection produces unstable feature assignments across frames, reducing autocorrelation by 36%. We propose spatio-temporal contrastive objectives and Matryoshka hierarchical grouping that recover and even exceed raw temporal coherence. The co
Abstract:
arXiv:2604.03919v1 Announce Type: cross Abstract: We present the first systematic study of Sparse Autoencoders (SAEs) on video representations. Standard SAEs decompose video into interpretable, monosemantic features but destroy temporal coherence: hard TopK selection produces unstable feature assignments across frames, reducing autocorrelation by 36%. We propose spatio-temporal contrastive objectives and Matryoshka hierarchical grouping that recover and even exceed raw temporal coherence. The co
DeepCamp AI