Data Parallelism Using PyTorch DDP | NVAITC Webinar

Key Takeaways

hello everyone and welcome to this last session of our deep learning Series today we'll be covering the data parallelism using python http this is an update of our video on some multi GPU training using horrobot so we'll be covering very similar topics but using Python's DDP approach instead of follow-up my name is architect leading the Nvidia Technology Center in Iberia and today I will be your instructor for data parallelism using pythonp so let's look first at our agenda uh we'll start by understanding why do we need multiple gpus um and the benefits of this brings to our deep learning Pipelines then we'll move into understanding a bit better what data parallelism means and how pythorch approaches and finally we'll see the integration of Python's DDP into our toolkit in with two examples one using torch vision and another one using Dali so why do we need to use multiple views and what are the benefits of this and this is something that you already saw in our lecture uh using heart of it but it's nevertheless it's always good to refresh it so one thing we've been observing throughout the years is um models or deep learning models have been becoming larger and larger thanks to this we are able to solve the more complex tasks however this of course comes with a more competing power that we need actually to give you a reference on the computing power if you would try to train one of these models on a typical dual CPU server it would take us around 100 years to be able to reach convergence whereas if we trade it in our latest djx h100 which is power we by eight eight gp8 100 gpus I would take us around the month to reach convergence but not only we need more compute we also need more data right these trainings need a large and large amount of data as the as the model gets larger um of course in the beginning of feeding data into our model we will be in a region where it's what we call beyond the best guest ever since it's not going to be enough for the model to build proper boundaries for deciding classes or whatever tasks we're looking into but then we will actually reach this region where we do see a well negative correlation if we look into the error or positive correlation that we look into the um performance or let's say accuracy between the amount of data we feed into the model and the um basically performance of the model right but yes everything live this has its limits and of course at some point we will not see much of a benefit um an intuition behind this is if we look into let's say mnist while in the beginning we'll see that the more pictures of numbers that we feed to the model the better it gets at some point it has seen almost every way of representing the model oh sorry representing the number so that's why it would not get better luckily for us while we did see this correlation between the amount of data and the model complexity um this relationship is sublinear so these actually are very good news for us because that means that while our models are becoming larger and larger our this is will too but not at the same Pace as our models so we don't have to because it's otherwise would be invisible to have such large volumes of data but one of the implications of having these more complex models and also larger amounts of data is the trading time that it takes so for example if you would look into how long it would take to train a resonate 50 back in 2015 this will take around 30 hours whereas now or like in this years thanks to the the usage of multiple tributes at the same time we're able to create around train the model in around eight minutes of it with more than a minute this not only has implications in terms of saving us time for other tasks but also it does has a direct implication in our performance since the lower the trading time is the more we can iterate and modifier model and therefore reach uh reach better accuracy and actually we can see an example here in the beginning of uh imagine a challenge um all the teams had around 25 of the classes wrong whereas in latest years at least in 2017 more than half of the teams had only five percent of the classes wrong and probably with the statistics nowadays is even lower since people are using more and more gpus and they cannot and actually adapt better to the challenge so to summarize this part um one thing we're observing is right is model complexities become is going up and therefore so is that the same size and therefore is also not a trading time right so there's three things that are going up and we it's not desirable since this take as long as the trade and so on the only thing we can control are the three of these isn't trading type how we can do that basically by leveraging multiple gpus right so if we um paralyze our training into the different gpus this reduces significantly the Turning time now the question is how can we do this one possible way of parallelizing our training into several gpus is using data parallelism there are other ways of parallelism such as model parallelism but in this Workshop we will always only covering data parallelism so so the same data parallelism let's refresh how the training on a single GPU uh worked and then from there on we will continue so in a single GPU we will have first will be readings of data right we'll move this data into either CPU or GPU depending where you're planning to pre-process it and you will pre-process data queue it to create some matches that will be fed into the model that will transport the data into the model and from here we will be first calculating the activations for layer one then layer two and then finally the output this output will be used for computing some loss and then we will provide the loss through the third layer first and the second one and the first we will use these to execute the opposition step and update the weights and then we'll just iterate so in a multiple GPU setting with data parallelism uh the first change comes here so instead of free and later all you want what we do is we'll split the data into chunks or two subsets of data so if let's say in a system where we have two gpus ATP will have half of the data and then we transport these data into its corresponding against CP or GPU depending where we do the training uh pre-process it queue it to create batches moving to the model and then here we'll have a model which is basically a copy of so a copy of the same model right so it would be initialized quickly with the same weights it would be a copy of each other across all the gpus so then here we compute activations for the first layer second layer instead layer the loss and then here um back focus to the third layer and here's the first um the Second Story second difference in this in a multi-gpu training so here what we do is we will average these gradients across all the gpus do the same for the second layer and finally for the first layer and then we'll be using this average of all the gradients to compute our opposition step and to update our weights and then finally we'll just iterate over and over as we just saw before okay so here then we have different things that need to be taken care right first we need to be able to split the data and this is something that might not be so easy since we won't have all the time same data into the same GPU um since we've seen in other lectures that these Randomness behavior is actually desired another thing we need to do is we need to copy and initialize the model as I was mentioning and these are something that it wouldn't be so easy to do manually but we have to have some tools that allows us to uh copy the model initially with the weights and create this communication and then finally we need to make also make sure that this uh correlation of the gradient averaging it's properly done and that we wait until the gpus are have uh compute the forward pass and so on and also that the IE properties are updated properly and with the right gradients so for those operations um as I would say they have different Frameworks uh in another lecture we explore how about but in today's lecture we're going to explore the Python's approach to doing data parallelism or distributed data parallelism the reason why we are focusing on by touch for this lecture is because if we've seen that the community has been using more and more pythridge throughout the years so I think right now is around 80 or 90 of the papers out there are reading pytorch so um we thought that it would be interesting to explore the their library for DDP okay so before I start explaining how DDP works on pytorch let me give some small vocabulary or some terms that we need to know before so first of all the concept of worker so for those that are familiar with other parallelism structures or parallel Computing uh I work in the same as calling a process so basically in our case so in this deep learning models approach it will be so if it's process will be taking care of training the model in a single GPU right so basically doing all these forward pass and backward pass and so on also the processing of the data the queuing and so on this will be a process that will be taken care of um so basically we'll have as many processes as tpus we're using and hopefully as many gpus we have available in the in our cluster and the total Story number of workers will be our world size other Concepts that we need to see these are more related to the cluster and the relationship between the cluster and the processes first is let's say we have this cluster the cluster has two notes in it so with basically two separate nodes with different gpus uh and they're connected between them so in here each node has two two gpus so in total the total number of gpus we have available is 4 gpus then what we'll do is for each process will initialize so in this case let's say we have four gpus we can initialize up to four processes so have a world size of four uh each of these processes will have a unique identifier called rank um or Global rack these are going to be useful because sometimes we want to know which process is coming particularly if there are certain operations that we only want to do once for all the training let's say and we will see this later more in detail but if you want to Output some progress the training you don't want to be doing this for each process but you only need to do it once so in this case you can just take one of the process with a rank let's say rank 0 and say okay only rank 0 is allowed to Output these operations but the the rest of the process can ignore this this way you don't have four times the same printing in your training another concept is also the concept of local Rank and as you might see here local rank is a unique in fire but only per host or per node so as you see we have a local run 0 and 1 for the first host uh and then we have a local Rock zero and one also for a second host why do we need identifiers this will have different um benefits one as you see we we will pinning the process of the local rank 0 to the GP or zero so that would be a one one way of seeing it as in because you know basically it's a way of choosing which GPR we're going to be using but also let's say if you want to do some also operations as in downloading some data set uh you want to do once per node because you don't want to be reading data set across nodes uh this is something that you may want to do only once per process in each in each node so here only local rank for example zero will be downloading the data and we'll see that more detail a little later in a small snippet of the code okay so now that we understand how data parallelism or distributed data parallelism uh Works let's see how we can integrate the pytharch language into our own code for that we will be using a our own nvidc toolkit which I encourage you all to check it out here you have a lot of examples on how to integrate the different parts of the tools that we've been seeing throughout these lecture this lecture Series so for example mixed precision dally and so on into a single code Repository in particular we'll be looking into two different parts we'll be looking to uh how to train this on a more regular pytharch workflow using torch vision and also in a workflow using Dali which is the our own solution for data augmentation so first let's see how we Define each of these workers that we were talking so each of these processes and some of the variables that we're going to be needing for this so let's say you have your main function right that you will launch once you run your script these will be basically changed to uh at least different ways of doing this but in for this example we'll be using multi-processing spawn so what we do is we say okay in our main function please launch um as many workers as uh we gpus will have available so you say here spawn takes worker which is basically a function that we'll see now that will be defining the different variables and we'll be calling the training functions then says how many process we wanna launch in this case uh as I was mentioning we'll be doing as many gpus as we have available per node and then finally also the arguments that we will need for our training that basically come from the parser so let's look into the worker so the worker has the same is basically a function that first um will be using for taking the or defining different variables so we can Define the global rung as the node ID will look into what the node is exactly but it's basically the the identifier of the node you are uh right now running the code from then the number of gpus per node plus the local rank which is already coming from basically spawn and then we initialize the initialize the process sorry sorry this is the computation then here we will initialize the process um in this case we just need to tell okay for each process we use equalized function then you tell which is the backend that you're going to use for the plus to communicate to the other processes in this case we chose nickel which is a Nvidia Library where all the options available also then okay how many percent in total in this training and then what's your unit identifier or Global rank then we'll also Define the device and as I was mentioning earlier we can use the local rank for this so we'll say okay whatever um parts of like or this trading is going to be basically linked to this particular device so we're gonna if we have to move things to GPU we're going to move it to this particular device so in this let's say if it was processed with local rank 0 um it will be moved to the GPU zero again this there will be two GPU zeros once per one per note of course and then also um sometimes you want to say okay if we have um several so if you are using we already have several gpus available distributed this might change some parts of the training so as we have this uh distributed variable that we're going to use later for some conditional statements okay so now that we've seen how to install the process we also need to prepare the model as I was saying and it's quite simple um so first we need to move our Network into the device that that process is linked to as we saw in the previous slide and then once it's moved we can just wrap it with this function from the 13 and parallel Library it's called distributed data parallel and then if you write the network basically it will really take care of initializing the process sorry initializing the network with all this with all the same weights and can be copying it across all the gpus so as you see here all it takes is basically the actual Network and then the local rank of the network and then it will already take care of initializing it through all the different gpus another thing that we need to do as we was mentioning it uh when we talked about the data parallelism in general is we need to do the partitioning of the data and as I was mentioning also uh it's not it's important that we not we cannot see the same data the same GPU same way that you change the order of the batches you also want to change the data your fit to each GPU to increase the randomness of the trading so first we need to shuffle the data and then we need to split this data across all the workers and this is something that is also easy to do with python GDP so up here we'll see the first difference between our torch Vision code and our daily code so for the third tuition uh the it's quite simple we just use the also from from Python's libraries we can use the distributed sampler and then this will already take care of Speedy data in chunks and also taking care of the shuffling and then with things we can once we Define the sampler you can just fit this into our typical tortuition data loader and uh and in there it already takes the variable sampler so you just fill it into it now when it comes to Delhi this is a bit different we won't be using the distributed sampler but already dally um has kind of integrated this sort of behavior in its pipelines so basically if you use the proper short ID and the number of shards it will really take care of splitting data across the um basically all the insurance or so in this case all the gpus not only we need to say sample data we also need to make sure that we update December every Epoch so basically every Epoch will be only suffering data as I was saying right so never ebook each uh GPU will be receiving a different subset of data and also it's important that once we have the substitute data we feed it into the uh proper device and again this was defined earlier uh using using the local rank in Dali again we need to take care of moving the data into our GPU this will be integrated in the actual pipe so just make sure that you define properly the short ID the number of charts and one thing that I don't have here in the on the slip of the code but it's uh it's also relevant it's something I was mentioned earlier when I was talking about local Rank and Global rug and I promise you to show you a bit of uh how this looks into the code so um here it's a for example of how local ranking Global rank might be relevant so let's say we first want to download some data set and we want to be doing this once per node right as I was mentioned earlier so in this case what we'll be doing is okay only the process that has to look at rank zero this is what's going to happen once per node downloads the data and also very important until this hasn't happened the rest of the process cannot continue on the script because otherwise we may reach a point where uh let's say process would look around 1 tries to do something with data but data is not there yet because local rank 0 hasn't downloaded yet that's why we're going to use a barrier to make sure that all the processes stop there and only once the local rank 0 has reached out at that point they can continue and another example of here of rank and indicates Global rank as I was mentioning this when you're trying to print some uh evolution of your training this is probably something you wanted only the ones uh regardless how many verses you have since yeah at the end we are averaging ingredients and so on so that's why we're gonna take the process with global right zero which only happens once for all the gpus across all the nodes and let it take care of the printing these evolution of training and then finally for our script be able to run we need to decide some variables so we first and again Define the number of nodes so we're going to be using the node ID from where you are like launching the experiment and also the number of nodes the ogpu sorry that you want to be using for per note we can calculate the total World size so the total number of workers slash processors will have these can be calculated by use by multiplying the number of gpus per the number and the number of nodes and that's it we also defined some environment variables and then finally here we want to just basically run it and again this works because we're using as was mentioned earlier the multi-processing is uh spawn you can basically just now use a typical python um running a script uh command so basically you will just put python the script board going to be running and then the number of nodes in the number of gpus of course because these are two new variables that we need and that's it and then you're already variables you had already Finance batch size or the data so that was it from from this just let me recap a bit of what we've seen so one thing we need to do once we launch the process is to make sure that we initialize each process properly using init process group then we need to also make sure that each worker or process has just its own GPU assigned and we can use the local rank for this then we need to wrap our model using the DDP function to make sure that all the the copies of the model are initialized the same way and they also update the same way we can use a suited sampler to shuffle and split the data across all the gpus and finally barriers are going to be very useful for us whenever we want all the process to be synced and stop at some point until and make sure that they don't continue until all the process reach a certain point in the code is for example relevant when downloading data so before I wrap up some tips that might be relevant for you now that you are be using larger larger batch sizes so one is you need to multiply your number your lettering rate by the number of gpus that you're using this is something that you see here in the first paper I linked here where they have the expiration for these um so yeah I encourage you to check it out but as a quick note it's basically just multiply your number of of learning rate by the number of gpus then um one thing that you might be able to use is platform sync which is basically is synchronize the bathroom statistics across all the gpus I would only encourage you to use this uh if you have a small but not bad sizes per GPU otherwise um this will slow down the trading process basically because you have another layer of communication so if you really can fit a decent amount of uh samples or batch size in each GPU this might not be needed and finally because you're also now um using larger batch sizes you might run into some issues so maybe you also want to take out some other optimizers such as Lark Lars novocrat that they work much better on larger batch sizes and that was it for my side um before leaving I'll please make sure also you check our developer program in there you'll find a lot of sdks a lot of server has been optimized for gpus also a lot of training resources like the ones you just saw so um also there's a big community in there so you have a lot of people that uh participate in forums they also organize meetups and hackathons and it's actually something that's for free so yeah I definitely you should definitely check it out and that was it for my site uh thank you all for listening and as I was saying before check out our toolkit where you can find all the Snippets of the code