Cactus: Accelerating Auto-Regressive Decoding with Constrained Acceptance Speculative Sampling

1. 1Implement speculative sampling with smaller draft models
2. 2Apply constrained acceptance criteria to allow for slight variations in generated distributions
3. 3Use techniques like top-$k$ or temperature sampling to improve acceptance rates
4. 4Evaluate and refine the Cactus approach for specific use cases and models

ML researchers and engineers on a team can benefit from Cactus to improve decoding efficiency, while working with large language models

💡 Constrained acceptance speculative sampling can improve decoding efficiency without sacrificing accuracy

🚀 Cactus accelerates auto-regressive decoding for LLMs with speculative sampling!