DeepSpeed | PyTorch Developer Day 2020

Key Takeaways

[Music] hello everyone i'm ishin i'm a research manager at microsoft today i'd like to share with you deep speed our dl training optimization library towards skill speed and democratizing large-scale training for everyone with great usability we may all experience the challenge of their training it's too slow to train high quality models on massive data we constantly encounter problems like off memory low throughput slow conversions furthermore the optimization ideas and techniques often scattered how to use them collectively and conveniently is non-trivial this motivates our work on deep speed directly target on the challenges we observe striving for three years on training efficiency for high throughput and scalability effectiveness for fast conversions and easy to use for improved development productivity what is deep speed deep speed is a dltrini optimization library on the software stack deep speed is optimization layer standing between dl models and training framework among the frameworks we primarily support pytorch users can leverage performance benefit of deep speed with a few lines of code changes on the pi touch model here is the example of bird model it shows the changes required to use deep speed the primary change is to call deep speed initialize to wrap the model and optimizer then you can think of this wrapped model as a deep speed engine and then with similar syntax as original use it for forward backward pass and parameter update and that's it minimum code changes then deep speed will handle the underlying performance challenges and applies efficiency and effectiveness optimizations that leads to speed and skill more specifically deep speed excels in five areas model skill speed democratization comprised training and usability which we will talk into one by one with respect to model skill state-of-art large models such as nvidia megatron lm and google t5 have about 10 billion parameters deep speed introduces a system breakthrough zero standing for zero redundancy optimizer to conquer the memory and the scaling challenges of very large models xero significantly reduces memory footprint while retaining compute and communication efficiency in the first stage we released last february 0 allows deep speed to run 100 billion parameter models efficiently it also part training of tiering lg 17 billion parameter model the largest model by then we continue to push for the frontier of training systems in early may we release zero tool which adds novel memory optimization on top of zero one and unblocks 200 billion parameter model training while planning to upstream zero related optimization to pytorch this september we release the 3d parallelism a flexible combination of zero power data parallelism with model and pipeline parallelism this allows us to run models with trillion parameters obtaining close to perfect memory and throughput scaling besides skill deep speed also excels in speed for very large models such as those over 25 billion parameters we observe up to 10x faster training comparing with using state-of-the-art model parallelism approach all performance boosts come from improved memory compute communication efficiency of zero this 10x faster training directly translates to up to 10x cost savings besides obtaining more efficient distributed executions through xero we also boost a single device performance we develop the fastest transformer kernels that leads to the world's fastest birth training another mission of deep speed is to democratize dr training for everyone in need from deep speed webinar survey we realized that more than half of our users only have one to four gpus but they do hope to explore advanced dl models and benefit from it in our september release we pushed out zero offload for these users zero offload leverages both cpu and gpu memory for training large models using a machine with a single gpu our users can run models of up to 13 billion parameters without running out of memory 10x bigger than existing approaches while obtaining compatib competitive throughput this feature democratized multi-billion parameter model training using a single gpu another innovation is one bit atom it reduces communication volume of original atom by up to 5x while achieving similar convergence efficiency in communication constraint scenarios like using ethernet we observe up to 3.5 x faster training one bit add-in democratizes distributed training and powers our users to skill on different types of gpu clusters and networks next compressed training compressed training allows us to use condensed form to represent store compute and communicate information deep speed leverage compressed training to save resources boosts training capability and efficiency one example is deep speed space attention compared with classic dense transformers our efficient spas kernel supports 10x longer sequence of model inputs and up to 6x faster training furthermore our spas kernels support efficient execution of flexible splash format and empower users to innovate under customs path structure our second example of compressed training is progressive layer dropping comparing with space attention it shows compressed training at very different granularity much called screen at transformer layer level at each training iteration we carefully drop a subset of layers reducing training time cost per iteration what's surprising is that this does not hurt accuracy instead it obtains comparable accuracy for faster conversions and more robust fine-tuning results last but definitely not the least deep speed is easy to use with only a few lines of code changes we can enable patos model to use deep speed as we showed as a previous bird example furthermore deep speed enables our users to train large models with up to 13 billion parameters using their loved data parallelism without worrying about model parallelism in contrast without deep speed models beyond 1.4 billion parameters will run out of memory with data parallelism alone yet another point on usability deep speed is infrastructure agnostic users can leave widget on their favorite environment like rml and pib systems local nodes and so on in summary in deep speed we drive innovations and share system breakthroughs to offer exceptional skill speed efficiency and good usability now in a bigger picture deep speed is an important part of microsoft's ai at skill initiative the initiative aims to enable next generation ai capabilities at scale across full stack from large-scale infrastructure to system software to models and to fuse ai into product now zooming into the system software layer this speed offers cutting edge system innovations with exceptional speed and skill we deliver this innovation largely and timely to pathology users through deep speed library onyx runtime inshot ort provides a high-performance cross-platform engine accelerated for both inference and training it's optimized for pytorch with deep integration underway we're also actually enabling deep speed innovations in it [Music] error ml offers enterprise grid service supporting the food development lifecycle for building training and deploying machine learning models faster at skill together with other pillars in this initiative our mission is to power everyone in need with the capabilities of ai at scale thank you all for your time to accelerate your model training and trial deep speed more features news tech talks deep dives and codes are available at our website deepspeed.ai and at our github thank you