Stochastic Sparse Attention for Memory-Bound Inference
📰 ArXiv cs.AI
Learn how Stochastic Sparse Attention (SANTA) reduces memory usage in autoregressive decoding by sparsifying value-cache access, and apply this technique to improve inference efficiency
Action Steps
- Implement SANTA by sampling a subset of indices from the post-softmax distribution
- Aggregate only the sampled value rows to reduce memory access
- Use the unbiased estimator of the post-softmax value aggregation to improve decoding accuracy
- Compare the performance of SANTA with traditional attention mechanisms
- Apply SANTA to memory-bound inference tasks, such as long-context autoregressive decoding
Who Needs to Know This
ML engineers and researchers working on large-scale language models can benefit from this technique to optimize memory usage and improve decoding efficiency
Key Insight
💡 SANTA sparsifies value-cache access by sampling a subset of indices, reducing memory usage while maintaining decoding accuracy
Share This
🚀 Reduce memory usage in autoregressive decoding with Stochastic Sparse Attention (SANTA) 🚀
Key Takeaways
Learn how Stochastic Sparse Attention (SANTA) reduces memory usage in autoregressive decoding by sparsifying value-cache access, and apply this technique to improve inference efficiency
Full Article
Title: Stochastic Sparse Attention for Memory-Bound Inference
Abstract:
arXiv:2605.01910v1 Announce Type: cross Abstract: Autoregressive decoding becomes bandwidth-limited at long contexts, as generating each token requires reading all $n_k$ key and value vectors from KV cache. We present Stochastic Additive No-mulT Attention (SANTA), a method that sparsifies value-cache access by sampling $S \ll n_k$ indices from the post-softmax distribution and aggregates only those value rows. This yields an unbiased estimator of the post-softmax value aggregation while replacin
Abstract:
arXiv:2605.01910v1 Announce Type: cross Abstract: Autoregressive decoding becomes bandwidth-limited at long contexts, as generating each token requires reading all $n_k$ key and value vectors from KV cache. We present Stochastic Additive No-mulT Attention (SANTA), a method that sparsifies value-cache access by sampling $S \ll n_k$ indices from the post-softmax distribution and aggregates only those value rows. This yields an unbiased estimator of the post-softmax value aggregation while replacin
DeepCamp AI