Steering Vision-Language Models with Joint Sparse Autoencoders

📰 ArXiv cs.AI

Learn to control vision-language models using Joint Sparse Autoencoders for improved cross-modal representations and steering directions

advanced Published 25 Jun 2026

Action Steps

Apply Joint Sparse Autoencoder (JSAE) to vision-language models to factorize sequence-pooled vision and language activations
Use an explicit alignment constraint to ensure shared and interpretable image/caption-level features
Configure JSAE to jointly optimize vision and language representations
Test the effectiveness of JSAE in controlling vision-language models
Evaluate the interpretability of the learned representations using metrics such as sparsity and alignment

Who Needs to Know This

AI engineers and researchers working on vision-language models can benefit from this technique to improve model interpretability and controllability. This can be particularly useful in applications such as image captioning and visual question answering.

Key Insight

💡 Joint Sparse Autoencoders can be used to improve the interpretability and controllability of vision-language models by learning shared and interpretable image/caption-level features