Speculative Decoding for 2x Faster Whisper Inference

📰 Hugging Face Blog

Speculative decoding can speed up Whisper inference by 2x

advanced Published 20 Dec 2023

Action Steps

Implement baseline Whisper model
Apply speculative decoding technique
Optimize speculative decoding for efficient inference
Test and evaluate performance gains

Who Needs to Know This

Machine learning engineers and researchers working on speech transcription models can benefit from this technique to improve inference speed

Key Insight

💡 Speculative decoding can significantly improve the inference speed of speech transcription models like Whisper

Key Takeaways

Speculative decoding can speed up Whisper inference by 2x

Full Article

Published Time: 2023-12-20T00:00:00.315Z

# Speculative Decoding for 2x Faster Whisper Inference

[![Image 1: Hugging Face's logo](https://huggingface.co/front/assets/huggingface_logo-noborder.svg)Hugging Face](https://huggingface.co/)

* [Models](https://huggingface.co/models)
* [Datasets](https://huggingface.co/datasets)
* [Spaces](https://huggingface.co/spaces)
* [Buckets new](https://huggingface.co/storage)
* [Docs](https://huggingface.co/docs)
* [Enterprise](https://huggingface.co/enterprise)
* [Pricing](https://huggingface.co/pricing)
*
*
* * *

* [Log In](https://huggingface.co/login)
* [Sign Up](https://huggingface.co/join)

[Back to Articles](https://huggingface.co/blog)

# [](https://huggingface.co/blog/whisper-speculative-decoding#speculative-decoding-for-2x-faster-whisper-inference) Speculative Decoding for 2x Faster Whisper Inference

Published December 20, 2023

[Update on GitHub](https://github.com/huggingface/blog/blob/main/whisper-speculative-decoding.md)

[- [x] Upvote 31](https://huggingface.co/login?next=%2Fblog%2Fwhisper-speculative-decoding)
* [![Image 2](https://huggingface.co/avatars/42b34eda17abafaba2c96906cecc9c73.svg)](https://huggingface.co/Shivangsharma "Shivangsharma")
* [![Image 3](https://cdn-avatars.huggingface.co/v1/production/uploads/1611199236853-noauth.jpeg)](https://huggingface.co/iprathore "iprathore")
* [![Image 4](https://cdn-avatars.huggingface.co/v1/production/uploads/6032802e1f993496bc14d9e3/w6hr-DEQot4VVkoyRIBiy.png)](https://huggingface.co/osanseviero "osanseviero")
* [![Image 5](https://huggingface.co/avatars/1fc151e025839e92af0beacd5a68ebff.svg)](https://huggingface.co/EricPeter "EricPeter")
* [![Image 6](https://cdn-avatars.huggingface.co/v1/production/uploads/618c42e4f35289d4bcab9cf1/wHmPhjRWUDdhyJMm8-jiJ.jpeg)](https://huggingface.co/ymoslem "ymoslem")
* [![Image 7](https://cdn-avatars.huggingface.co/v1/production/uploads/619f7ba90df8731e0d8b6c54/L0O4z0klhnyjw6eCjrNln.png)](https://huggingface.co/kadirnar "kadirnar")
* +25

[![Image 8: Sanchit Gandhi's avatar](https://cdn-avatars.huggingface.co/v1/production/uploads/1653243468328-61f91cf54a8e5a275b2b3e7c.jpeg)](https://huggingface.co/sanchit-gandhi)

[Sanchit Gandhi sanchit-gandhi Follow](https://huggingface.co/sanchit-gandhi)

[![Image 9: Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/sanchit-gandhi/notebooks/blob/main/speculative_decoding.ipynb)

* [Speculative Decoding](https://huggingface.co/blog/whisper-speculative-decoding#speculative-decoding "Speculative Decoding")

* [English Speech Transcription](https://huggingface.co/blog/whisper-speculative-decoding#english-speech-transcription "English Speech Transcription")
* [Baseline Implementation](https://huggingface.co/blog/whisper-speculative-decoding#baseline-implementation "Baseline Implementation")

* [Speculative Decoding](https://huggingface.co/blog/whisper-speculative-decoding#speculative-decoding-1 "Speculative Decoding")

* [Multilingual Speech Transcription](https://huggingface.co/blog/whisper-speculative-decoding#multilingual-speech-transcription "Multilingual Speech Transcription")

* [Strategies for Efficient Speculative Decoding](https://huggingface.co/blog/whisper-speculative-decoding#strategies-for-efficient-speculative-decoding "Strategies for Efficient Speculative Decoding")

* [Conclusion](https://huggingface.co/blog/whisper-speculative-decoding#conclusion "Conclusion")

* [Acknowledgements](https://huggingface.co/blog/whisper-speculative-decoding#acknowledgements "Acknowledgements")

Open AI's [Whisper](https://openai.com/research/whisper) is a general purpose speech transcription model that achieves state-of-the-art results across a range of different benchmarks and audio conditions. The latest [large-v3](https://huggingface.co/openai/whisper-large-v3) model tops the [OpenASR Leaderboard](https://huggingface.co/spaces/hf-aud

Read full article → ← Back to Reads

Speculative Decoding for 2x Faster Whisper Inference

Key Takeaways

Full Article

Related Videos