Tips N Tricks # 8: Using automatic mixed precision training with PyTorch 1.6

Key Takeaways

hello everyone and welcome to this new short video in this one I'm going to show you how you can use automatic mixed precision from PI Taj natively so PI dot 1.6 is going to have native support for automatic expression in training and mixer engine helps you in many different ways one of the things is your model will occupy less memory you so you can use larger batch sizes you can have faster training and till now we have been using it using Nvidia's apex but now since it's in built its natively supported by part or we can try using that one and see if it brings any kind of improvement so I should make a much longer video on a MP but I will probably if the time permits so you can you can read more about mixed provision training from this paper which is called an expression training and get to know more about it and in this one I'm going to show you how to use it so for just to start with we have already seen so this is the bird sentiment model that I'm using that I have trained a long time ago so if you have not taken a look at it you can take a look at this model it's also in the description box so I'm going to fight this line data parallel for now and then try to train the model so python train dot pi so I'm not changing anything in the model right now so let's see you let's see what happens so you can see that it's showing that the model is training and it's going to take around 32 minutes if we look at the memory consumption it's around 10 gigabytes of GPU memory 9009 9191 and so let me just stop it first and now we can try the mixed precision training and see what happens so q2 use mixed precision there are a few steps it's it's not very difficult so you can you have to import from torch to kuda import EMP which is automatic mixed precision so when you used Nvidia's epic see you used to import from apex import have a MP and then you have to define the scaler before anything begins so that's your gradient scaler grad scaler and then pass it on to the training function so let me just write a cheer scaler okay now we go to our training function and here when we are doing the forward pass so everything remains the same but when we are doing the forward pass we say like we have to use the context of auto casting so with a MP dot auto cast and here also you need to import so from taured CUDA import a MP so MP dot auto cast and then you do the forward pass of the model and also calculate the gloss and when you're done with that you have to do the backward function so in this one you have to just scaler dot scale loss and then backward and then the optimizer step so scaler dot step and then optimizer and then you have to update the scalar so scaler dot update so as you can see it's more straightforward and I think NVIDIA apex is also similar so there's not much difference and now we can start to train this model one more thing that I forgot was to include a scaler here okay and now we can train the model of and see what happens so as you can see now the model is training and showing 18 minutes so previously it was 32 minutes now it's 18 minutes so things are quite good it seems and if I look at memory consumptions now it's 8 gigabytes so we reduce 2 gigabytes of memory and that's that's how FB 16 or mix president training helps you automatic mix version so it's not just a p16 and one more thing to remember that in in the training we used data parallel in the original version so if you're using data parallel then you have to auto cast the forward function so what you can do is you can you can import from char store CUDA import EMP and then you can use you can use it in different ways so you can have the MP dot auto cast a decorator here or you can do with a MP dot auto cost so you can use this context and put everything inside this context so but we are going for the decorator and once you're once you've done that you can use the model in the same way so it's it's yeah it's not very difficult it's very simple and this is like one of the optimizations you should always go for it's going to make your training much faster provided your GPU supports mixed precision training which is Pascal or more yeah and that's it for today's video and I hope you liked it and subscribe my channel if you liked it and you liked click on the like button and share it with your friends so this is all about automatic mix precision in PI touch 1.6 it won't work with pythons 1.5 so you have to go to the nightly version just remember that and if you have any comments write me in the comment section and I would be happy to take a look and reply if you have any queries so thank you very much and see you next time goodbye