Sparse Feature Coactivation Reveals Causal Semantic Modules in Large Language Models

arXiv:2506.18141v3 Announce Type: replace-cross Abstract: We identify semantically coherent, context-consistent network components in large language models (LLMs) using coactivation of sparse autoencoder (SAE) features collected from just a handful of prompts. Focusing on concept-relation prediction tasks, we show that ablating these components for concepts (e.g., countries and words) and relations (e.g., capital city and translation language) changes model outputs in predictable ways, while amp