VIA-SD: Verification via Intra-Model Routing for Speculative Decoding

1. 1Build a slim submodel from the full verifier using intra-model routing
2. 2Run speculative decoding with the slim submodel to verify rejected tokens
3. 3Configure the draft-verify method to use the slim submodel for verification
4. 4Test the VIA-SD approach on a dataset to evaluate its performance
5. 5Apply the VIA-SD technique to existing LLMs to reduce inference costs

AI engineers and researchers working on LLMs can benefit from this technique to optimize their models, while data scientists can apply this knowledge to improve their NLP pipelines

💡 Intra-model routing can be used to derive a slim submodel from the full verifier, reducing the computational cost of verification in speculative decoding

💡 Reduce LLM inference costs with VIA-SD, using intra-model routing for verification!