Control Reinforcement Learning: Interpretable Token-Level Steering of LLMs via Sparse Autoencoder Features

📰 ArXiv cs.AI

Learn to steer LLMs using interpretable token-level features via Control Reinforcement Learning (CRL) and sparse autoencoders

advanced Published 5 May 2026

Action Steps

Train a sparse autoencoder to decompose language model activations into interpretable features
Implement Control Reinforcement Learning to select features for steering at each token
Evaluate the learned policy using intervention logs to identify features that change model outputs
Apply CRL to various NLP tasks to improve model transparency and control
Compare the performance of CRL with other interpretability methods

Who Needs to Know This

NLP researchers and engineers can use CRL to develop more transparent and controllable language models, while data scientists can apply this technique to improve model interpretability

Key Insight

💡 CRL enables transparent and controllable language models by identifying features that change model outputs when amplified

Key Takeaways

Learn to steer LLMs using interpretable token-level features via Control Reinforcement Learning (CRL) and sparse autoencoders

Full Article

Title: Control Reinforcement Learning: Interpretable Token-Level Steering of LLMs via Sparse Autoencoder Features

Abstract:
arXiv:2602.10437v3 Announce Type: replace-cross Abstract: Sparse autoencoders (SAEs) decompose language model activations into interpretable features, but existing methods reveal only which features activate, not which change model outputs when amplified. We introduce Control Reinforcement Learning (CRL), which trains a policy to select SAE features for steering at each token, producing interpretable intervention logs: the learned policy identifies features that change model outputs when amplifi

Read full paper → ← Back to Reads