Apex - Michael Carilli, NVIDIA

Key Takeaways

[Music] all right hello everyone my name is Michael Corelli I am a developer technology engineer at Nvidia on the PI torch frameworks team and today I'm going to be talking to you about training with mixed precision using a mixture of torch float or FB 32 and torch half or FP 16 to take full advantage of the hardware capabilities that nvidia x' latest GPUs provide so what are the benefits of this using mixed precision and our latest tensor core enabled architectures your networks can achieve substantial improved speed ups they can be more memory efficient while remaining just as accurate without needing to retune your hyper parameters alternatively the speed ups and memory savings can enable you to experiment with larger networks or larger batch sizes so first question you may be wondering is why why not just stick with the default torch float or FB 32 the answer is first of all FB 16 or torch dot 1/2 enables it takes up half the memory storage it can see a 2x speed-up for bandwidth bound operations but that's not the only benefit on NVIDIA stents or Core GPUs there's dedicated hardware support for matrix multiplies and convolutions with FP 16 input and these hardware cores these these tensor cores give 8 X improved computational throughput for such operations for matrix multiplies and convolutions so if your network happens to use a lot of matrix multiplies in convolutions you can see significantly greater than a 2 X + 2 n speed-up and I'll provide some concrete examples of that shortly so now you may be asking the opposite question why not just use torch dot have for everything the answer is that some operations like accumulations and optimizer updates also benefit from the wider dynamic range and increased precision of FP 32 the idea behind mixed precision is that by assigning each operation it's optimal precision you obtain the speed of FB 16 the precision of FB 32 and take full advantage of the hardware the full hardware capabilities of nvidia gpus and achieve high speed as well as stability so how well does this work in practice here's an example that shows we've achieved substantial speed ups on a diversity of highly relevant real world networks and the links below show that you can you can check out these examples yourself your speed ups may vary depending on whether your network is more compute bound more bandwidth bound or constrained by something else like data loading for example bert you can see achieved a pretty hefty speed-up because it uses a lot of very expensive matrix matrix multiplies for which the tensor cores are highly beneficial you may also be wondering does training with mixed precision affect my accuracy and in practice we found that all the networks that we've trained with FP 16 or rather with mixed precision have converged to comparable accuracy as pure FP 32 training with no hyper parameter changes so that's encouraging so how can you realize these benefits for your own network we've developed this tool called automatic mixed precision or amp whereby you can insert a few lines of Python into your script and it will do this entire recipe for you automatically here's how that looks in a simple example these three lines ensure that every operation runs in its appropriate precision this tool is available to try today through the Nvidia repository of apex utilities I've updated the landing pages to contain a link to this talk so you can also use those to track down the deep learning examples of Burton mask our CNN etc that showcase mixed precision best practices as well as the resulting speed ups so as pi torch developers naturally the best way to reach you guys is through pi torch itself so today I'm happy to announce that we are currently working with the PI towards core team to enable native support for mixed precision in pi torch this quarter I've already got an API request for comment up as well as the first PR so feel free to comment let us know what you need we have certainly tried to take into account all the complex use cases we've accounted for along the way to make sure that the implementation will be powerful and flexible but we're also interested to hear what you guys have to say [Music] [Applause] [Music] [Applause]