Faster Text Generation with TensorFlow and XLA
📰 Hugging Face Blog
Faster text generation with TensorFlow and XLA can achieve up to 100x speedup
Action Steps
- Use the Hugging Face transformers library with TensorFlow
- Enable XLA compilation for text generation
- Compare benchmarks with other frameworks like PyTorch
- Optimize model performance using techniques like greedy decoding or sampling
Who Needs to Know This
AI engineers and data scientists can benefit from this technique to improve the performance of their text generation models, while product managers can leverage this to enhance the overall user experience
Key Insight
💡 XLA compilation can significantly improve the performance of text generation models
Share This
🚀 Faster text generation with TensorFlow and XLA: up to 100x speedup! 🤖
Key Takeaways
Faster text generation with TensorFlow and XLA can achieve up to 100x speedup
Full Article
Published Time: 2022-07-27T00:00:00.107Z
# Faster Text Generation with TensorFlow and XLA
[Hugging Face](https://huggingface.co/)
* [Models](https://huggingface.co/models)
* [Datasets](https://huggingface.co/datasets)
* [Spaces](https://huggingface.co/spaces)
* [Buckets new](https://huggingface.co/storage)
* [Docs](https://huggingface.co/docs)
* [Enterprise](https://huggingface.co/enterprise)
* [Pricing](https://huggingface.co/pricing)
*
*
* * *
* [Log In](https://huggingface.co/login)
* [Sign Up](https://huggingface.co/join)
[Back to Articles](https://huggingface.co/blog)
# [](https://huggingface.co/blog/tf-xla-generate#faster-text-generation-with-tensorflow-and-xla) Faster Text Generation with TensorFlow and XLA
Published July 27, 2022
[Update on GitHub](https://github.com/huggingface/blog/blob/main/tf-xla-generate.md)
[- [x] Upvote -](https://huggingface.co/login?next=%2Fblog%2Ftf-xla-generate)
[](https://huggingface.co/joaogante)
[Joao Gante joaogante Follow](https://huggingface.co/joaogante)
* [Text Generation](https://huggingface.co/blog/tf-xla-generate#text-generation "Text Generation")
* [TensorFlow and XLA](https://huggingface.co/blog/tf-xla-generate#tensorflow-and-xla "TensorFlow and XLA")
* [Text Generation using TensorFlow with XLA](https://huggingface.co/blog/tf-xla-generate#text-generation-using-tensorflow-with-xla "Text Generation using TensorFlow with XLA")
* [Benchmarks and Conclusions](https://huggingface.co/blog/tf-xla-generate#benchmarks-and-conclusions "Benchmarks and Conclusions")
_TL;DR_: Text Generation on 🤗 `transformers` using TensorFlow can now be compiled with XLA. It is up to 100x faster than before, and [even faster than PyTorch](https://huggingface.co/spaces/joaogante/tf_xla_generate_benchmarks) -- check the colab below! [](https://colab.research.google.com/github/huggingface/blog/blob/main/notebooks/91_tf_xla_generate.ipynb)
## [](https://huggingface.co/blog/tf-xla-generate#text-generation) Text Generation
As the quality of large language models increased, so did our expectations of what those models could do. Especially since the release of OpenAI's [GPT-2](https://openai.com/blog/better-language-models/), models with text generation capabilities have been in the spotlight. And for legitimate reasons -- these models can be used to summarize, translate, and they even have demonstrated zero-shot learning capabilities on some language tasks. This blog post will show how to take the most of this technology with TensorFlow.
The 🤗 `transformers` library started with NLP models, so it is natural that text generation is of utmost importance to us. It is part of Hugging Face democratization efforts to ensure it is accessible, easily controllable, and efficient. There is a previous [blog post](https://huggingface.co/blog/how-to-generate) about the different types of text generation. Nevertheless, below there's a quick recap of the core functionality -- feel free to [skip it](https://huggingface.co/blog/tf-xla-generate#tensorflow-and-xla) if you're familiar with our `generate` function and want to jump straight into TensorFlow's specificities.
Let's start with the basics. Text generation can be deterministic or stochastic, depending on the `do_sample` flag. By default it's set to `False`, causing the output to be deterministic, which is also known as Greedy Decoding. When it's set to `True`, also known as Sampling, the output will be stochastic, but you can still obtain reproducible results through the `seed` argument (with the same format as in [stateless TensorFlow random number generation](https://www.tensorflow.org/api_docs/python/tf/random/stateless_categori
# Faster Text Generation with TensorFlow and XLA
[Hugging Face](https://huggingface.co/)
* [Models](https://huggingface.co/models)
* [Datasets](https://huggingface.co/datasets)
* [Spaces](https://huggingface.co/spaces)
* [Buckets new](https://huggingface.co/storage)
* [Docs](https://huggingface.co/docs)
* [Enterprise](https://huggingface.co/enterprise)
* [Pricing](https://huggingface.co/pricing)
*
*
* * *
* [Log In](https://huggingface.co/login)
* [Sign Up](https://huggingface.co/join)
[Back to Articles](https://huggingface.co/blog)
# [](https://huggingface.co/blog/tf-xla-generate#faster-text-generation-with-tensorflow-and-xla) Faster Text Generation with TensorFlow and XLA
Published July 27, 2022
[Update on GitHub](https://github.com/huggingface/blog/blob/main/tf-xla-generate.md)
[- [x] Upvote -](https://huggingface.co/login?next=%2Fblog%2Ftf-xla-generate)
[](https://huggingface.co/joaogante)
[Joao Gante joaogante Follow](https://huggingface.co/joaogante)
* [Text Generation](https://huggingface.co/blog/tf-xla-generate#text-generation "Text Generation")
* [TensorFlow and XLA](https://huggingface.co/blog/tf-xla-generate#tensorflow-and-xla "TensorFlow and XLA")
* [Text Generation using TensorFlow with XLA](https://huggingface.co/blog/tf-xla-generate#text-generation-using-tensorflow-with-xla "Text Generation using TensorFlow with XLA")
* [Benchmarks and Conclusions](https://huggingface.co/blog/tf-xla-generate#benchmarks-and-conclusions "Benchmarks and Conclusions")
_TL;DR_: Text Generation on 🤗 `transformers` using TensorFlow can now be compiled with XLA. It is up to 100x faster than before, and [even faster than PyTorch](https://huggingface.co/spaces/joaogante/tf_xla_generate_benchmarks) -- check the colab below! [](https://colab.research.google.com/github/huggingface/blog/blob/main/notebooks/91_tf_xla_generate.ipynb)
## [](https://huggingface.co/blog/tf-xla-generate#text-generation) Text Generation
As the quality of large language models increased, so did our expectations of what those models could do. Especially since the release of OpenAI's [GPT-2](https://openai.com/blog/better-language-models/), models with text generation capabilities have been in the spotlight. And for legitimate reasons -- these models can be used to summarize, translate, and they even have demonstrated zero-shot learning capabilities on some language tasks. This blog post will show how to take the most of this technology with TensorFlow.
The 🤗 `transformers` library started with NLP models, so it is natural that text generation is of utmost importance to us. It is part of Hugging Face democratization efforts to ensure it is accessible, easily controllable, and efficient. There is a previous [blog post](https://huggingface.co/blog/how-to-generate) about the different types of text generation. Nevertheless, below there's a quick recap of the core functionality -- feel free to [skip it](https://huggingface.co/blog/tf-xla-generate#tensorflow-and-xla) if you're familiar with our `generate` function and want to jump straight into TensorFlow's specificities.
Let's start with the basics. Text generation can be deterministic or stochastic, depending on the `do_sample` flag. By default it's set to `False`, causing the output to be deterministic, which is also known as Greedy Decoding. When it's set to `True`, also known as Sampling, the output will be stochastic, but you can still obtain reproducible results through the `seed` argument (with the same format as in [stateless TensorFlow random number generation](https://www.tensorflow.org/api_docs/python/tf/random/stateless_categori
DeepCamp AI