Faster assisted generation support for Intel Gaudi
📰 Hugging Face Blog
Hugging Face optimizes assisted decoding for Intel Gaudi, reducing latency and costs in text generation tasks
Action Steps
- Understand the importance of inference optimizations for text generation
- Explore assisted decoding as a method for speeding up text generation
- Optimize assisted decoding for Intel Gaudi using Hugging Face's adaptations
Who Needs to Know This
AI engineers and data scientists can benefit from this optimization to improve the efficiency of their text generation models, while product managers can consider the cost savings and improved user experience
Key Insight
💡 Assisted decoding can significantly reduce latency and costs in text generation tasks, making it an essential optimization technique for AI implementations
Share This
🚀 Faster text generation with Intel Gaudi! 🚀
Key Takeaways
Hugging Face optimizes assisted decoding for Intel Gaudi, reducing latency and costs in text generation tasks
Full Article
Published Time: 2024-06-04T00:00:00.401Z
# Faster assisted generation support for Intel Gaudi
[Hugging Face](https://huggingface.co/)
* [Models](https://huggingface.co/models)
* [Datasets](https://huggingface.co/datasets)
* [Spaces](https://huggingface.co/spaces)
* [Buckets new](https://huggingface.co/storage)
* [Docs](https://huggingface.co/docs)
* [Enterprise](https://huggingface.co/enterprise)
* [Pricing](https://huggingface.co/pricing)
*
*
* * *
* [Log In](https://huggingface.co/login)
* [Sign Up](https://huggingface.co/join)
[Back to Articles](https://huggingface.co/blog)
# [](https://huggingface.co/blog/assisted-generation-support-gaudi#faster-assisted-generation-support-for-intel-gaudi) Faster assisted generation support for Intel Gaudi
Published June 4, 2024
[Update on GitHub](https://github.com/huggingface/blog/blob/main/assisted-generation-support-gaudi.md)
[- [x] Upvote 3](https://huggingface.co/login?next=%2Fblog%2Fassisted-generation-support-gaudi)
* [](https://huggingface.co/adamm-hf "adamm-hf")
* [](https://huggingface.co/benzakenelad "benzakenelad")
* [](https://huggingface.co/Alan009 "Alan009")
[](https://huggingface.co/haimbarad)
[Haim Barad haimbarad Follow](https://huggingface.co/haimbarad)
[](https://huggingface.co/Intel "Intel")[Intel](https://huggingface.co/Intel)
[](https://huggingface.co/neharaste)
[Neha Raste neharaste Follow](https://huggingface.co/neharaste)
[](https://huggingface.co/Intel "Intel")[Intel](https://huggingface.co/Intel)
[](https://huggingface.co/joeychou)
[Tien Pei Chou joeychou Follow](https://huggingface.co/joeychou)
[](https://huggingface.co/Intel "Intel")[Intel](https://huggingface.co/Intel)
* [Speculative Sampling - Assisted Decoding](https://huggingface.co/blog/assisted-generation-support-gaudi#speculative-sampling---assisted-decoding "Speculative Sampling - Assisted Decoding")
* [Usage & Experiments](https://huggingface.co/blog/assisted-generation-support-gaudi#usage--experiments "Usage & Experiments")
* [Conclusion](https://huggingface.co/blog/assisted-generation-support-gaudi#conclusion "Conclusion")
As model sizes grow, Generative AI implementations require significant inference resources. This not only increases the cost per generation, but also increases the power consumption used to serve such requests.
Inference optimizations for text generation are essential for reducing latency, infrastructure costs, and power consumption. This can lead to an improved user experience and increased efficiency in text generation tasks.
Assisted decoding is a popular method for speeding up text generation. We adapted and optimized it for Intel Gaudi, which delivers similar performance as Nvidia H100 GPUs as shown in [a previous post](https://huggingface.co/blog/bridgetower), while its price
# Faster assisted generation support for Intel Gaudi
[Hugging Face](https://huggingface.co/)
* [Models](https://huggingface.co/models)
* [Datasets](https://huggingface.co/datasets)
* [Spaces](https://huggingface.co/spaces)
* [Buckets new](https://huggingface.co/storage)
* [Docs](https://huggingface.co/docs)
* [Enterprise](https://huggingface.co/enterprise)
* [Pricing](https://huggingface.co/pricing)
*
*
* * *
* [Log In](https://huggingface.co/login)
* [Sign Up](https://huggingface.co/join)
[Back to Articles](https://huggingface.co/blog)
# [](https://huggingface.co/blog/assisted-generation-support-gaudi#faster-assisted-generation-support-for-intel-gaudi) Faster assisted generation support for Intel Gaudi
Published June 4, 2024
[Update on GitHub](https://github.com/huggingface/blog/blob/main/assisted-generation-support-gaudi.md)
[- [x] Upvote 3](https://huggingface.co/login?next=%2Fblog%2Fassisted-generation-support-gaudi)
* [](https://huggingface.co/adamm-hf "adamm-hf")
* [](https://huggingface.co/benzakenelad "benzakenelad")
* [](https://huggingface.co/Alan009 "Alan009")
[](https://huggingface.co/haimbarad)
[Haim Barad haimbarad Follow](https://huggingface.co/haimbarad)
[](https://huggingface.co/Intel "Intel")[Intel](https://huggingface.co/Intel)
[](https://huggingface.co/neharaste)
[Neha Raste neharaste Follow](https://huggingface.co/neharaste)
[](https://huggingface.co/Intel "Intel")[Intel](https://huggingface.co/Intel)
[](https://huggingface.co/joeychou)
[Tien Pei Chou joeychou Follow](https://huggingface.co/joeychou)
[](https://huggingface.co/Intel "Intel")[Intel](https://huggingface.co/Intel)
* [Speculative Sampling - Assisted Decoding](https://huggingface.co/blog/assisted-generation-support-gaudi#speculative-sampling---assisted-decoding "Speculative Sampling - Assisted Decoding")
* [Usage & Experiments](https://huggingface.co/blog/assisted-generation-support-gaudi#usage--experiments "Usage & Experiments")
* [Conclusion](https://huggingface.co/blog/assisted-generation-support-gaudi#conclusion "Conclusion")
As model sizes grow, Generative AI implementations require significant inference resources. This not only increases the cost per generation, but also increases the power consumption used to serve such requests.
Inference optimizations for text generation are essential for reducing latency, infrastructure costs, and power consumption. This can lead to an improved user experience and increased efficiency in text generation tasks.
Assisted decoding is a popular method for speeding up text generation. We adapted and optimized it for Intel Gaudi, which delivers similar performance as Nvidia H100 GPUs as shown in [a previous post](https://huggingface.co/blog/bridgetower), while its price
DeepCamp AI