Hyperparameter Search with Transformers and Ray Tune
📰 Hugging Face Blog
Use Ray Tune for hyperparameter search with Transformers to improve model performance
Action Steps
- Install Ray Tune and Hugging Face Transformers library
- Define the search space for hyperparameters
- Use Ray Tune's built-in schedulers, such as PBT or Bayesian Optimization, to perform hyperparameter search
- Evaluate the performance of the model with the optimized hyperparameters
Who Needs to Know This
Data scientists and machine learning engineers can benefit from using Ray Tune to optimize hyperparameters for their Transformer models, leading to improved performance and efficiency
Key Insight
💡 Advanced hyperparameter tuning techniques like PBT and Bayesian Optimization can significantly improve model performance compared to simple grid search
Share This
🚀 Boost your Transformer model's performance with Ray Tune's hyperparameter search!
Key Takeaways
Use Ray Tune for hyperparameter search with Transformers to improve model performance
Full Article
Published Time: 2020-11-02T00:00:00.005Z
# Hyperparameter Search with Transformers and Ray Tune
[Hugging Face](https://huggingface.co/)
* [Models](https://huggingface.co/models)
* [Datasets](https://huggingface.co/datasets)
* [Spaces](https://huggingface.co/spaces)
* [Buckets new](https://huggingface.co/storage)
* [Docs](https://huggingface.co/docs)
* [Enterprise](https://huggingface.co/enterprise)
* [Pricing](https://huggingface.co/pricing)
*
*
* * *
* [Log In](https://huggingface.co/login)
* [Sign Up](https://huggingface.co/join)
[Back to Articles](https://huggingface.co/blog)
# [](https://huggingface.co/blog/ray-tune#hyperparameter-search-with-transformers-and-ray-tune) Hyperparameter Search with Transformers and Ray Tune
Published November 2, 2020
[Update on GitHub](https://github.com/huggingface/blog/blob/main/ray-tune.md)
[- [x] Upvote 4](https://huggingface.co/login?next=%2Fblog%2Fray-tune)
* [](https://huggingface.co/etomoscow "etomoscow")
* [](https://huggingface.co/Moreza009 "Moreza009")
* [](https://huggingface.co/AmrataYadav "AmrataYadav")
* [](https://huggingface.co/Cavali "Cavali")
[](https://huggingface.co/system)
[system system Follow](https://huggingface.co/system)
##### [](https://huggingface.co/blog/ray-tune#a-guest-blog-post-by-richard-liaw-from-the-anyscale-team) A guest blog post by Richard Liaw from the Anyscale team
* [Try it out today:](https://huggingface.co/blog/ray-tune#try-it-out-today "Try it out today:")
With cutting edge research implementations, thousands of trained models easily accessible, the Hugging Face [transformers](https://github.com/huggingface/transformers) library has become critical to the success and growth of natural language processing today.
For any machine learning model to achieve good performance, users often need to implement some form of parameter tuning. Yet, nearly everyone ([1](https://medium.com/@prakashakshay90/fine-tuning-bert-model-using-pytorch-f34148d58a37), [2](https://mccormickml.com/2019/07/22/BERT-fine-tuning/#advantages-of-fine-tuning)) either ends up disregarding hyperparameter tuning or opting to do a simplistic grid search with a small search space.
However, simple experiments are able to show the benefit of using an advanced tuning technique. Below is [a recent experiment run on a BERT](https://medium.com/distributed-computing-with-ray/hyperparameter-optimization-for-transformers-a-guide-c4e32c6c989b) model from [Hugging Face transformers](https://github.com/huggingface/transformers) on the [RTE dataset](https://aclweb.org/aclwiki/Textual_Entailment_Resource_Pool). Genetic optimization techniques like [PBT](https://docs.ray.io/en/latest/tune/api_docs/schedulers.html#population-based-training-tune-schedulers-populationbasedtraining) can provide large performance improvements compared to standard hyperparameter optimization techniques.
**Algorithm****Best Val Acc.****Best Test Acc.****Total GPU min****Total $ cost**
Grid Search 74%65.4%45 min$2.30
Bayesian Optimization +Early Stop 77%66.9%104 min$5.30
Population-based Training 78%70.5%48 min$2.45
If you’re leveraging [Transformers](https://github.com/huggingface/transformers), you’ll want to have a way to easily access powerful hyperparameter tuning solutions without giving up the customizability of the Transformers framework.
[Hugging Face](https://huggingface.co/)
* [Models](https://huggingface.co/models)
* [Datasets](https://huggingface.co/datasets)
* [Spaces](https://huggingface.co/spaces)
* [Buckets new](https://huggingface.co/storage)
* [Docs](https://huggingface.co/docs)
* [Enterprise](https://huggingface.co/enterprise)
* [Pricing](https://huggingface.co/pricing)
*
*
* * *
* [Log In](https://huggingface.co/login)
* [Sign Up](https://huggingface.co/join)
[Back to Articles](https://huggingface.co/blog)
# [](https://huggingface.co/blog/ray-tune#hyperparameter-search-with-transformers-and-ray-tune) Hyperparameter Search with Transformers and Ray Tune
Published November 2, 2020
[Update on GitHub](https://github.com/huggingface/blog/blob/main/ray-tune.md)
[- [x] Upvote 4](https://huggingface.co/login?next=%2Fblog%2Fray-tune)
* [](https://huggingface.co/etomoscow "etomoscow")
* [](https://huggingface.co/Moreza009 "Moreza009")
* [](https://huggingface.co/AmrataYadav "AmrataYadav")
* [](https://huggingface.co/Cavali "Cavali")
[](https://huggingface.co/system)
[system system Follow](https://huggingface.co/system)
##### [](https://huggingface.co/blog/ray-tune#a-guest-blog-post-by-richard-liaw-from-the-anyscale-team) A guest blog post by Richard Liaw from the Anyscale team
* [Try it out today:](https://huggingface.co/blog/ray-tune#try-it-out-today "Try it out today:")
With cutting edge research implementations, thousands of trained models easily accessible, the Hugging Face [transformers](https://github.com/huggingface/transformers) library has become critical to the success and growth of natural language processing today.
For any machine learning model to achieve good performance, users often need to implement some form of parameter tuning. Yet, nearly everyone ([1](https://medium.com/@prakashakshay90/fine-tuning-bert-model-using-pytorch-f34148d58a37), [2](https://mccormickml.com/2019/07/22/BERT-fine-tuning/#advantages-of-fine-tuning)) either ends up disregarding hyperparameter tuning or opting to do a simplistic grid search with a small search space.
However, simple experiments are able to show the benefit of using an advanced tuning technique. Below is [a recent experiment run on a BERT](https://medium.com/distributed-computing-with-ray/hyperparameter-optimization-for-transformers-a-guide-c4e32c6c989b) model from [Hugging Face transformers](https://github.com/huggingface/transformers) on the [RTE dataset](https://aclweb.org/aclwiki/Textual_Entailment_Resource_Pool). Genetic optimization techniques like [PBT](https://docs.ray.io/en/latest/tune/api_docs/schedulers.html#population-based-training-tune-schedulers-populationbasedtraining) can provide large performance improvements compared to standard hyperparameter optimization techniques.
**Algorithm****Best Val Acc.****Best Test Acc.****Total GPU min****Total $ cost**
Grid Search 74%65.4%45 min$2.30
Bayesian Optimization +Early Stop 77%66.9%104 min$5.30
Population-based Training 78%70.5%48 min$2.45
If you’re leveraging [Transformers](https://github.com/huggingface/transformers), you’ll want to have a way to easily access powerful hyperparameter tuning solutions without giving up the customizability of the Transformers framework.
[
DeepCamp AI