Hierarchical Sparse Circuit Extraction from Billion-Parameter Language Models through Scalable Attribution Graph Decomposition

📰 ArXiv cs.AI

Extract sparse circuits from large language models efficiently using Hierarchical Attribution Graph Decomposition, reducing search cost and handling feature reuse

advanced Published 23 Jun 2026

Action Steps

Apply Hierarchical Attribution Graph Decomposition to billion-parameter language models to extract sparse circuits
Train cross-layer transcoders to initialize the decomposition process
Use spectral coarsening to reduce the dimensionality of attribution graphs
Employ graph-neural-network-guided hierarchical traversal to identify sparse circuits
Verify the extracted circuits using causal intervention verification

Who Needs to Know This

ML engineers and researchers working with large language models can benefit from this technique to improve model interpretability and efficiency

Key Insight

💡 Hierarchical Attribution Graph Decomposition reduces the search cost of extracting sparse circuits from billion-parameter language models

Key Takeaways

Extract sparse circuits from large language models efficiently using Hierarchical Attribution Graph Decomposition, reducing search cost and handling feature reuse

Full Article

Title: Hierarchical Sparse Circuit Extraction from Billion-Parameter Language Models through Scalable Attribution Graph Decomposition

Abstract:
arXiv:2601.12879v2 Announce Type: replace-cross Abstract: Extracting sparse circuits from billion-parameter transformers is constrained by $O(2^n)$ search cost and pervasive feature reuse across co-active pathways. Hierarchical Attribution Graph Decomposition (HAGD) addresses this through four stages: cross-layer transcoder training, spectral coarsening of attribution graphs, graph-neural-network (GNN)-guided hierarchical traversal, and causal intervention verification, reducing worst-case compl

Read full paper → ← Back to Reads