Five Ways To Increase Your Model Performance Using PyTorch Profiler

Key Takeaways

pytorch profiler version 1.9 has finally been released the goal of this release is to help data scientists target and visualize their execution steps that are the most costly in time and memory now let's take a look at a few new features memory view is an added feature that allows you to understand the time and memory consumption that may have caused performance bottlenecks like out of memory issues or your model taking a really long time to execute this memory view allows you to see which exact operator by the name is contributing to these high consumptions of time and memory so now perhaps speeding your model training is your goal so you conduct distributed training however debugging can be very complex and hard to diagnose without a distributed view like this you can actually observe issues within each individual node so each of these views as you can see over here can give you different information that can help you diagnose the reason for your bottleneck for example in this view over here if one of the computation and overlapping time of one worker is larger than the other this to a data scientist can suggest an issue in the workload not being balanced or a worker being a straggler which is an issue that needs to be optimized in the code now sometimes performance issues are beyond memory and node level issues in your model and perhaps you need to observe issues on the gpu level at every step and this is where the gpu utilization view comes in imagine you have a resin fifty model with a batch size of four and all of these important gpu metrics are low there's clearly a bottleneck since your goal is to get 100 gpu utilization down here it suggests us that perhaps increasing our batch size will reduce our bottleneck the next run will show 32 batch size and the utilization actually increased which is great but it's still not 100 so to further diagnose the trace view will allow you to see the overall utilization view in 10 millisecond buckets and you can see that there's an unusual dip in this area so let's investigate by zooming in the utilization in this kernel is 11 and beside it is around 50 but in order for us to get more finer detail we need to look at the sm efficiency which gives us finer details on each kernel and you can see that the reason why the utilization is not 100 is because of how sparse it is and the idle time between the eye the kernels as you can see performance issues are normally a black box and having the new release of pytorch profiler will allow you to diagnose and optimize your code better don't forget to check us out on github at kineto for more information and samples