Accelerating PyTorch distributed fine-tuning with Intel technologies
📰 Hugging Face Blog
Accelerate PyTorch distributed fine-tuning with Intel technologies to reduce training time and cost
Action Steps
- Use Intel servers for distributed training
- Leverage Intel performance libraries for optimization
- Set up a cluster with oneCCL for distributed jobs
- Install necessary dependencies and launch single-node and distributed jobs
Who Needs to Know This
AI engineers and data scientists can benefit from this post to optimize their deep learning model training, while DevOps teams can utilize the insights to improve cluster setup and performance
Key Insight
💡 Using CPU-based clusters with Intel technologies can be a cost-effective and efficient way to fine-tune deep learning models
Share This
⚡️ Accelerate PyTorch distributed fine-tuning with Intel technologies! 🚀
Key Takeaways
Accelerate PyTorch distributed fine-tuning with Intel technologies to reduce training time and cost
Full Article
Published Time: 2021-11-19T00:00:00.040Z
# Accelerating PyTorch distributed fine-tuning with Intel technologies
[Hugging Face](https://huggingface.co/)
* [Models](https://huggingface.co/models)
* [Datasets](https://huggingface.co/datasets)
* [Spaces](https://huggingface.co/spaces)
* [Buckets new](https://huggingface.co/storage)
* [Docs](https://huggingface.co/docs)
* [Enterprise](https://huggingface.co/enterprise)
* [Pricing](https://huggingface.co/pricing)
*
*
* * *
* [Log In](https://huggingface.co/login)
* [Sign Up](https://huggingface.co/join)
[Back to Articles](https://huggingface.co/blog)
# [](https://huggingface.co/blog/accelerating-pytorch#accelerating-pytorch-distributed-fine-tuning-with-intel-technologies) Accelerating PyTorch distributed fine-tuning with Intel technologies
Published November 19, 2021
[Update on GitHub](https://github.com/huggingface/blog/blob/main/accelerating-pytorch.md)
[- [x] Upvote 2](https://huggingface.co/login?next=%2Fblog%2Faccelerating-pytorch)
* [](https://huggingface.co/iamsingularity "iamsingularity")
* [](https://huggingface.co/Aanuoluwapo65 "Aanuoluwapo65")
[](https://huggingface.co/juliensimon)
[Julien Simon juliensimon Follow](https://huggingface.co/juliensimon)
* [What this post is about](https://huggingface.co/blog/accelerating-pytorch#what-this-post-is-about "What this post is about")
* [Using Intel servers](https://huggingface.co/blog/accelerating-pytorch#using-intel-servers "Using Intel servers")
* [Using Intel performance libraries](https://huggingface.co/blog/accelerating-pytorch#using-intel-performance-libraries "Using Intel performance libraries")
* [Setting up our cluster](https://huggingface.co/blog/accelerating-pytorch#setting-up-our-cluster "Setting up our cluster")
* [Installing dependencies](https://huggingface.co/blog/accelerating-pytorch#installing-dependencies "Installing dependencies")
* [Launching a single-node job](https://huggingface.co/blog/accelerating-pytorch#launching-a-single-node-job "Launching a single-node job")
* [Setting up a distributed job with oneCCL](https://huggingface.co/blog/accelerating-pytorch#setting-up-a-distributed-job-with-oneccl "Setting up a distributed job with oneCCL")
* [Running a distributed job with oneCCL](https://huggingface.co/blog/accelerating-pytorch#running-a-distributed-job-with-oneccl "Running a distributed job with oneCCL")
* [Conclusion](https://huggingface.co/blog/accelerating-pytorch#conclusion "Conclusion")
For all their amazing performance, state of the art deep learning models often take a long time to train. In order to speed up training jobs, engineering teams rely on distributed training, a divide-and-conquer technique where clustered servers each keep a copy of the model, train it on a subset of the training set, and exchange results to converge to a final model.
Graphical Processing Units (GPUs) have long been the _de facto_ choice to train deep learning models. However, the rise of transfer learning is changing the game. Models are now rarely trained from scratch on humungous datasets. Instead, they are frequently fine-tuned on specific (and smaller) datasets, in order to build specialized models that are more accurate than the base model for particular tasks. As these training jobs are much shorter, using a CPU-based cluster can prove to be an interesting option that keeps both training time and cost under control.
### [](https://huggingface.co/blog/accelerating-pytorch#what-this-post-is-about) What this post is about
In this po
# Accelerating PyTorch distributed fine-tuning with Intel technologies
[Hugging Face](https://huggingface.co/)
* [Models](https://huggingface.co/models)
* [Datasets](https://huggingface.co/datasets)
* [Spaces](https://huggingface.co/spaces)
* [Buckets new](https://huggingface.co/storage)
* [Docs](https://huggingface.co/docs)
* [Enterprise](https://huggingface.co/enterprise)
* [Pricing](https://huggingface.co/pricing)
*
*
* * *
* [Log In](https://huggingface.co/login)
* [Sign Up](https://huggingface.co/join)
[Back to Articles](https://huggingface.co/blog)
# [](https://huggingface.co/blog/accelerating-pytorch#accelerating-pytorch-distributed-fine-tuning-with-intel-technologies) Accelerating PyTorch distributed fine-tuning with Intel technologies
Published November 19, 2021
[Update on GitHub](https://github.com/huggingface/blog/blob/main/accelerating-pytorch.md)
[- [x] Upvote 2](https://huggingface.co/login?next=%2Fblog%2Faccelerating-pytorch)
* [](https://huggingface.co/iamsingularity "iamsingularity")
* [](https://huggingface.co/Aanuoluwapo65 "Aanuoluwapo65")
[](https://huggingface.co/juliensimon)
[Julien Simon juliensimon Follow](https://huggingface.co/juliensimon)
* [What this post is about](https://huggingface.co/blog/accelerating-pytorch#what-this-post-is-about "What this post is about")
* [Using Intel servers](https://huggingface.co/blog/accelerating-pytorch#using-intel-servers "Using Intel servers")
* [Using Intel performance libraries](https://huggingface.co/blog/accelerating-pytorch#using-intel-performance-libraries "Using Intel performance libraries")
* [Setting up our cluster](https://huggingface.co/blog/accelerating-pytorch#setting-up-our-cluster "Setting up our cluster")
* [Installing dependencies](https://huggingface.co/blog/accelerating-pytorch#installing-dependencies "Installing dependencies")
* [Launching a single-node job](https://huggingface.co/blog/accelerating-pytorch#launching-a-single-node-job "Launching a single-node job")
* [Setting up a distributed job with oneCCL](https://huggingface.co/blog/accelerating-pytorch#setting-up-a-distributed-job-with-oneccl "Setting up a distributed job with oneCCL")
* [Running a distributed job with oneCCL](https://huggingface.co/blog/accelerating-pytorch#running-a-distributed-job-with-oneccl "Running a distributed job with oneCCL")
* [Conclusion](https://huggingface.co/blog/accelerating-pytorch#conclusion "Conclusion")
For all their amazing performance, state of the art deep learning models often take a long time to train. In order to speed up training jobs, engineering teams rely on distributed training, a divide-and-conquer technique where clustered servers each keep a copy of the model, train it on a subset of the training set, and exchange results to converge to a final model.
Graphical Processing Units (GPUs) have long been the _de facto_ choice to train deep learning models. However, the rise of transfer learning is changing the game. Models are now rarely trained from scratch on humungous datasets. Instead, they are frequently fine-tuned on specific (and smaller) datasets, in order to build specialized models that are more accurate than the base model for particular tasks. As these training jobs are much shorter, using a CPU-based cluster can prove to be an interesting option that keeps both training time and cost under control.
### [](https://huggingface.co/blog/accelerating-pytorch#what-this-post-is-about) What this post is about
In this po
DeepCamp AI