Arthur Douillard - Distributed Training in Machine Learning

Key Takeaways

[Music] Uh so thanks Amad for inviting me. Um so I'm going to talk about distributed learning um or distributed training from local parallelism to planetary computation. So I work at the mine. Uh my topics is to work on distributed learning but I aim to work on distributed learning methods that are a bit different from what usually people work I would say. Uh and in order to go beyond just distributed learning between like a small cluster of GPU to actually be able to use all GPU across the world. Um so the goal of this session I'm going to make a quick overview of existing methods that are methods that everyone currently use even small labs. uh and I'm also going to talk more about what I believe should be the future of those distributed learning method. I'm not going to go too much in the details. It's more like to give another view of what's existing and what I believe is the direction. That's my personal view. That's not the view of my employer and that's probably not the view of the consensus of the community both in AI and in distributed learning. I will also focus mostly on LLM. That's kind of the trendy thing. I know everyone's talk about LM and I will focus for the pre-training side of the LM of course the post training so I think we have a talk few days before about how do post training or one like and everything thinking uh the distributed aspect of the post training is very different from the pre training so here I'm only focusing on the pre training uh I want to start first with the beta lesson uh so I think know like in the previous lessons you had like probably talked about like scaling laws how to scale uh so the bit lesson by Satan which was written like 20 years ago uh is basically that you can make humanmade innovation to make a model better. So for example you can take like the convolutional neural networks then the reset then Adam then the self attention all those algorithm improvement managed to make us better AI but in the end like the Peter le is in the end you can be as smart as possible as a human but you want to develop a method that will be able to use as much as possible compute and the amount of compute so how many flops how many operation per seconds we have been doing this climb drastically across the years. So you can find here like a plot from 2010. So it was the beginning of the reset with Alex net and extra to 20 uh so I cannot see my screen to 24 2024 with like the gymn ultra and other models where know we use much more flops and bically like my the le is that we need to use as much compute as possible and we need to develop method that can use as much comput as possible. a success of transformer is that you were able to use much more compute than for example the previous ANM for language modeling because we could simply paralyze everything and just maximize the amount of comput so the question is that if we want to use as much comput as possible what do we need to focus on so here on screen a very simplified view that there's three axis you can work on uh one axis is the algorithm innovation so that's the most sexy axis is basically you invent like a new optimizer a new modification to the architecture. So for example the residual in reset the self attention transformer more lately for example the reinforcement learning from verifiable reward. So all these kind of stuff are algorith innovation that can unlock new thing but all these innovation are meaningless without the two other access. So it's more data uh so we are knowable to have like lm and chatbots that are extremely competent mainly because we are using the huge amount of internet data. uh so we want to have as much data as possible and that uh data human generated is starting to stagnate. So maybe in future we will have to do synthetic data. More data also means that not simply more bigger amount of quantity of data but also like better high quality of data. So that's one axis and another axis also to have simply bigger model. Uh so for example the old reset in the imaget time were maybe like 5 million parameters which is not a bit ridiculous given that no like yesterday Kim released an MOE transformer that was I think one trillion parameters in total. So you can see that from 5 million parameters 15 years ago to know one trillion parameters we just use much bigger model. Um so my guest is like to have a bit of lesson we do need algorithm innovation but we need to have much more data also. Now we are talking about trillion trillions of data of tokens of text token and we are talking about trillion of parameters all that require to use just much more compute. So now we can first focus on the model access. If I want to have like a model that is so big how can I train properly? uh if I take for example the lama 3.1 which was 45 billion parameters uh that was equivalent to more than close to two terabytes of gigab 2 terabytes of data just for the parameters of the model so you even in single chips which is like H100 which is like a server grade and computer grade like very high bigger than you could probably have in your bedroom you cannot fit the wall model on a single GPU and then if you want to fit also the optimizer so for example Adam optimizer that's going to use twice more um parameters and then you need to also subract more memory to start the activation and the gradients. So basically very quickly at some scale you won't be able to fit your whole model on a single uh chips. And here I'm talking about the parameter that was less than four 400 billion parameters. If I take the kimi that was released yesterday of one trillion parameters you will never be able to store it on a single GPU. There is some tricks of course to be able to fit those model on single GPU. You can count the parameters. So meaning using a lower precision for those parameters. You can actually not store the optimizer on your GPU memory but you can store it on the CPU memory. So optimizer offloading even you can offload part of the actual model. You can do grad pointing. So when you do the backward pass you don't need to actually store all the activation. you can recomp compute them as you do the backward pass which come with a cost of slower but reduce the memory usage. So there's many tricks for store model on a single GPU but they all come with head off of course usually which mean that they are either more unstable or slower and at some point that we want to actually use multiple GPUs in order to train a bigger model. So here I'm going to show very quickly two existing method and we have the pipeline parallelism PP. So this probably like the most intuitive one. You're just some simply going to split the model across the depth axis. So you're going to put one or several layers per GPU. So instead of showing the wall parameters on a single GPU, I will give the input tokens to the first GPU which going to process here the first layer. Then we output the tokens, we transfer it from the GPU0 to GPU one. So this transfer require communication and we do it until going to all the layers to get the output tokens. here the communication and that's very critical uh there's a lot of communication so that's basically the all activation on the forward pass and the gradient on the backward pass are going to happen in between each GPU if the GPU are well connected that may not be a problem but we're going to come back to that then you have another solution is the tensor parallel where tensor parallel basically is instead of splitting the layer the model per layer you actually going to put all the layers on the same GPU but you're going to split the matrix multiplication. So the exact matrix either colon wise or row wise across multiple GPU. So that's another way here you're splitting across the width axis of the model. Um here there's also like communication during a forward pass and backward pass because you do a subset of the matrix multiplication on one GPU another subset on another GPU but at some point you need to gather them back and here there's a communication between the two GPU this is for the model access now about the data access the question is like also we want to use more data as I say previously so the previous models imaget for example was like a few hundred thousand images. Now we're talking about using several dozen of trillions of tokens. Um so if you remember the scaling laws one paper from deep mine in 20 2011 2021 sorry called chila scaling laws stated that basically to be compute optimal you need for each neuron in your model to train with 20 tokens. So 20 to1 ratio. That mean that basically if I was to train a 10D dance parameter model I will need 200 billion tokens. Uh so this transcraining their model much more. So for example the llama which was 45 billion tokens the parameters was trained on more than 15 trillion tokens. That's a 38 to1 ratio. And actually if you want to train a very good small model such as for example the recent egging face small LM3 which was a 3B model you train it on 11 trillion tokens. So that's like a raso that is much much bigger than the transfer scanning lo makes sense because you want to when you have a small model you don't simply want to be complet but you want to be also as powerful as possible in order to deploy it. In theory, the trans scaling law say that you have to actually not train a 3D model for that kind of complete budget and token budget, but you should train a bigger model. But then if you train a bigger model, it's actually have to serve on consumer grade for GPU. So you want to package and to crumb as much knowledge as possible on a very small model. So you're going to do much more over training to ratio that is superior to 2021. The small model, small LM3 for example, it's parameters. So you can easily fit it on a single GPU that's very easy this even on consumer grade GPU but if you want to train 11 trillion tokens on a single GPU and thought you have to use a small batch size in order to fit all the activation and the gradients it's going to take decades because you cannot have a big batch size you're going to have to do several millions of steps and literally you can just make the estimation based on the numbers it's going to take you decades to train it of course you cannot train a small model for decades because in 10 years we will have a completely different kind of AI maybe just RGI. So you need like a much larger batch size in order just to use as much as fast possible uh to train as fast as possible and to maximally use your GPU but you cannot fit all everything on a single GPU. So then again you need to go multip multiGPU when you want to train with more data. So here it's a bit more simpler than the model parallelism. Basically everyone use a variant of data parallel BP. So here it's quite intuitive. You simply going to take your batch. So here we are talking about transformer for example. So your batch size is going to be several seconds. So seconds or multiple token on each second. And you're going to split each subset of the seconds of the batch size across a GPU. Then another subset across another GPU and etc. So we are splitting the batch across multiple GPU. Each um and each GPU all the entire set of the parameters of the model. It's a copy on each GPU. EGPU then going to do a forward. So we do simply all forward activation to get the loss on each GPU the loss that will respective on the subset of the data that you are seen. Then we do the backward pass. So each GPU is going to compute their own respective gradients with their own respective subset of the data and then we do a all reduce where all the layers are going to communicate all the gradants together. So now you have all the same gradients on all GPU and you can do an optimizer step and do the next step. So here um there's less communication in the end like less frequent communication as we are simply doing communication during the backward pass doing the forward pass it's uh no communication between GPU however we can only do one step so those are those methods are like completely orthogonal and they can all be combined together so in you can do pipeline in parallel so split the model across multiple layers transfer parallel split the model across the matrices is rows and columns and data parallel where you split the data across the batch access everything can be combined together that's what people do in practice uh there is different so especially for the latency so in ter pipeline parallel you have a much more sensitivity to latency than data parallel where you can kind of overlap the the gradient pass backward pass with the reducer there's also different bond with requirement and different amount of frequency. All those thing have trade-offs. And as well also like usually when you have a big cluster of GPU, you have different level speed. I can have like for example a bunch of GPU that are in a room, bunch of GPU in another room. Inside a room, we have a high bandwidth. In between the two rooms, we have a lower bandwidth. Therefore, you could do inside a room only tons of parallel between those GPUs. And then in between rooms while you have a slightly lower bandwidth, you could use data parallel. So there's also a wararchy of bond width which mean also of which distributed method to use at which moment. This is the only what I'm going to talk about existing deeply learning methods. Uh there is many other varants. So for example there is a second parallelism uh that's quite similar to data parism but here instead of giving an entire seconds to a different GPU here we're actually splitting across GPU the actual seconds so across multiple tokens we're splitting it for example for the fit forward activation we can split and each we do fit forward on each different subset of the second slant and then we do all gather again when we have to do the self attention we also have expert sharding So when we do mixture of experts where instead of having a single feed forward network we have multiple feed forward network in parallel okay we can actually shard uh the expert fit forward parameters on different devices so each device will have a single expert or multiple and then we just gather them back at the end of the mixture of expert layer and finally we have FSDP which is probably what most people use uh you can look up the literature uh there's multiple variant of FSDP it's kind of similar to data parallel but also shard the parameters and we do regularly n gather to recare to get back the parameters when we do a forward so that's a mix of basically doing model parallelism and data parallel and that's kind of the way to go when people train their model there's also other varants so I'm not talking about everything the me paper explain quite well uh how do we do properly the trans parl detail to be able to do it correctly uh pipeline parallelism is very useful because you can it's less sensitive to the bond width than tensor parallel and you can actually split your model across multiple location unfortunately pipelism has a problem of what is called bubble of time is that when we do the forward pass we are using all the GPU one after the other and then we do the backward pass but it's very hard to actually use all the GPU at the same time so basically the pipeline parallel will have for several seconds some GPU that will be idle and doing no computation So that's kind of wasted because if I give you a GPU, you want it to be fully used to be always working. So that you really want to avoid computation and the bubble of time is quite big. You can lose up to 15% of your time. There is some trade-off how to actually make it better. You can do microbatching smarter order of operation or asking backward pass. There's many tricks but just to remember that pattern parallelism is not the silver bullet because mainly on this problem so here I talk about like distributed learning methods that were more like classical so that's what people use which is fine when you collocate everything together uh so when you have everything in a single data centers for example or if it's between two data centers that are very well connected with like a very high bandwidth that can work the problem is like if you want to actually uh do across the world, you're going to be bottom link because all the distributed learning method I show you before uh they require communication. You cannot simply uh split your wall AI training either the model or the the data or both across multiple GPU and not have the GPU communicate. They need to communicate. That communication is actually uh can be several gigabyte or several terabytes of data that need to be communicated as fast as possible. If the communication is slow, we are actually waiting for the communication to finish. And while we wait, those GPUs are idle. They're not doing any work. And it's actually a very big waste of comput. As I said a bit lesson, you just want to use as much compute as possible. Meaning that your your GPU should be always working. That's really the only thing you need to remember. Um, so that's why people basically try to just cram as much possible GPU on a single location. For example, I think Xi with Musk, they managed to put like 200k GPU in a single location, but they really struggle and they had to have like some portable um gas plant in order to have enough electricity in that single location. But it come with a price. It's very hard to actually collocate all the GPU in a single location because sometime you don't have enough electricity on that location as I say for example for the XI problem and you could build like more nuclear plant for example for the data centers but being a nuclear plant is going to take you five or 10 years so it's actually faster to use multiple data centers at multiple location where there's already enough electricity but then you need to connect connect all the data centers together you also want to do computer arbitration is sometime uh compute can cheaper in one location than another. Maybe because those people use less compute at that location for that moment. So you want to do an arbitrage and use that compute that is cheaper. Or maybe electricity is suddenly cheaper on that location. So the compute is also cheaper and you want to use that. So you will be able to split your model across multiple location and use only the location that are the cheapest for you to train your AI. Uh and also you want to scavenge all the comput across the world. If you want to do a planetary computation where basically it's something where everyone could contribute in their bedroom their own GPU to a big AI training across the world here you will need to connect all the GPU which have a very poor bandwidth in between and be sure that they are always working and are not bottlenecked by that communication. So you cannot do it with the current methods data parallel transfer parallel and pipeline parallel they will require a high amount of bandwidth and you cannot do planetary computation with those method. Um the good thing about those methods is that fundamentally do not change anything about the math behind the training. So for example if I do a data parallel and I'm splitting the data across multiple location I do a forward pass and a backward pass independently on each GPU and then I do all gather gradients to the communication I do a communication and then I continue my training whether I do that whether I just train my AI on a single GPU I train my AI data pile across multiple GPU the results will be exactly the same the whole loss dynamics the parameters everything except some numerical issues you're going to get the exact same results so it's very reassuring because you can do test on a few GPU, you saw it works and then you scale it to maybe 200 GPU for distributed and you see that you you're confident that as you do more distributed methods is not going to break because it's exactly the same mathematical results. So that's very critical those methods are very attractive for that reason. However, I believe that we need to break away from those methods. We need to take more risky method that won't be mathematically equivalent. So that bring a risk because it mean that your learning dynamics your loss your grad will be different and they may get worse but they will allow you to communicate much less and actually to do the communication across the world. So I'm going to show first two methods, two access of methods for both the model and data how you can actually break it uh and do plan computation while going away from being mathematically equivalent and then I will go to the third axis. So in pipeline um if you want to do pipeline parallelism across the internet where for example each of us we have a GPU in our bedroom and we own only a layer of the par of the model and then we just forward through that layer and then we send the communication to the other layers. So pipeline parallelism across multiple location in the US in Europe in India in Asia everything you can read with swarm. So swarm was a paper from a few years ago going to do exactly this thing paralism across the internet importantly with a bunch of hardware. So that's what you want basically. For example, you have a T4 and A100 kind of GPU where they all communicate a different speed which mean you need to do some kind of balancing. Sometime there's also like a failure failure both on the GPU failure both on the communication. So you need to be able to root. If for example I did my computation on the first stage so my first layer for example then I want to send to the second layer my second stage and one GPU for GGPU has failed you want to actually route the communication back to another GPU in order to forward. So it's actually quite complex doing this kind of methods and also it's still going to require quite a lot of communication in between the stage. So you probably want to use some methods such as for example more recently the purist paper where they do a compression they compress activation in pipeline parallel in order to use pipeline stage across the world with a very low bandwidth but this compression is very efficient in the paper show there is barely any loss of performance but still it's not mathematically equivalent which mean it bring with a cost a risk. So that was for the model axis. Now for the data axis you can simply use again data parallel as I said before in data parallel each GPU is going to see a different subset of the data. They're going to do a forward with their subset of data backward pass the subset of data and then they do all reduce where they communicate each of them gradantss. This reduce is where the communication happen the gradients are fairly large size is the same size of the wall model parameters and then basically you want to compress the gradients. So usually part of the problem your parameters may be in float 32 but you're usually going to communicate your gradient in bloat 16. So it's twice less bits. So it's already compression of time too but that's not enough. So if you want to go across the world and let's say you have a communication bandwidth that maybe in the dozen or hundred of megaby seconds and you want to communicate a model that is like several gigabytes or several terabytes, you need to compress as much as possible in order to not wait for the communication. So there's for example three paper that are quite famous cocktail SGD proposed to combine multiple compressor method. So top K you only take the top K values then the contise. So I think you can contise up to float four I think they do. So with only four bits per parameters um you can also do random specification so randomly you drop some value similar to drop out but here apply on the gradient. So that's quite aggressive but they show that for example that throughut so basically how much you can do a forward of token without being bottlenecked by the communication as if you're bottom by communication your output is worse you can find that basically the all reduce the classical methods if you have a high bandwidth is going to be very efficient but as soon as the bandwidth is actually very low so maybe one gigabytes or less the throughput is close to null because with all you actually have to communicate a lot of data and you just waiting for the communication you're wasting time. While with CO HD because they compress aggressively, they can still have a pretty good throughput with only 500 megaby seconds. Of course, it come with a risk because you actually compressing very aggressively and they won't be the same quality. Um, and you may have a loss of ML performance. You also have power. So, it's kind of similar. You do another kind of compression. Here's a low compression. So, your gradients are usually low rank. So you can export that and to do to project them in a lower space. So instead of projecting a gradient that maybe of size 1,00 time 1,00 you can project it to another space that's what I 1,00 time five. So going be a huge compression factor and you communicate much less data. Uh it also come with a risk but it was fairly used at large scale. If you read the Delhi paper so Delhi was like the first image generation from open eye. They actually have a big section in appendix where they describe open eye how they use the power. Maybe that's still what they use currently for GPT. And uh we have also demo more recent paper that's kind of similar. Uh it's not a law compression but they use a for 4 year transform based compression and they show that you can train also across the world in their networks. So that also compressed quite aggressively. All those methods are compressed very aggressively uh which come with a risk because you will never be able to get the exact same quality of gradients. The question is whether you can still get a good enough performance. So we saw um one way to do distributed across the world. So with a low bandwidth by doing for example a very hard compression of the activation in pipeline parallelism for data parallel we can do a compression of the gradients when we do the radius. So both of the method are quite similar is in a way you want to compress aggressively what data you actually communicate but it's also mean that in pipeline parallelism so you it's always like everything is in sync uh in data parallel everything also is in sync so all the GPU needs to go at the same speed together and they all have to be in lock step you just communicate less data but they are still all going forward at the same moment in time I argue that there is another axis that we should explore to planned by computation that I think is much more efficient and can also be less risky is to do local training. So I'm going to first go back to the parallel to explain how we can go from data parallel to local training. So data parallel um as I said before you start with a model randomly in size you put the same parameters a copy the same parameter on each GPU you do one step so that's a forward a backward and you synchronize so that's the reduce that's your bottleneck of communication and then you do again again a forward a backward on each GPU with a different subset of the data and then you synchronize again so the problem is I said before you have this communication that is very costly. So the cost the amount of communication that we do you can reduce it as I show before with cocktail po demo there is way to make it less costly but it's still a blocking communication that you have to block for it and you have to wait that is happen so you can compress as much as you want but if the bandwidth start to get way too low at some point you still have to wait for the communication and you have eel GPU waiting for communication and just wasting the time to do another computation. Um so there's way to do local computation local training. uh there is a lot of literature up to the 1983 I think it's come under many name such as local HD uh federated learning although it's slightly different and I can add later why um and fed here I'm showing in my framework which was made explicitly for LLM DLCO so where the idea where we still have we start with the same model randomly you put the same copy of the model randomly on different GPU and then each GPU is going to see a different subset of the data. The difference is here like we don't communicate at every steps but we do multiple forward backward pass on each GPU and then we apply we do an optimizer apply so we update our parameters but each GPU with each replica which has different subset of the data is going to be optimized in a different way because they see different data so now you don't have a single version of your parameters but you have multiple version that are slightly different because they are training on slightly different data so they're going to all multiple steps. So that's between dozen to hundreds of steps they're going to do differently. So the model are going to be slightly different but there is no communication in between and then every for example 100 steps you're going to communicate all the parameters together synchronize again them together and then you do another round of local training. So the idea of block training is like you do multiple forward backward steps and optimize apply without communicating. Of course like you're going to say that if we are communicating much less so here we're only communicating every hundred of steps and we do local computation we are bound to get lower performance because intuitively you want to communicate as much as possible to get the per best performance and that's true. In order to recover the loss performance because of the less communication we need to do a step of outer optimization. So it's inspired from the local HDL literature. There's many small variants uh that all quite different. I'm showing here like the var which was the best for LM training. Was there a question? We can take questions at the end. >> Okay. Um so I'm talking the variance of DLCO which is a variant of fedopet local but that's the one who works the best for LM training. So you have multip you have two step of um optimization. You have first in red the inner optimization which is the local training where each GPU. So each GPU with a worker with a index of I is going to see a different subset of data. So that subset di they're going to compute their loss here on line six and then on line eight they're going to update the parameters with their inner optimizer. So the in optimizer is going to be Adam and they're going to do multiple steps each independently to optimize the parameters and then once in a while so every for example edge steps which could be for example one or steps they're going to compute a delta here it's a delta between for given parameter for given a worker GPU their current parameters minus the parameter they add one of those steps in the past. So what you can see is basically this delta in the parameter space is a trajectory is what's the path you took across hundreds of optimization steps. this trajectory uh then we can consider it as a form of outer gradient although it's not a proper gradient you have one out grad per GPU per worker and then you do an average so here this average in line 12 uh trigger on all reduce operation so that's the communication that communication only happen every hundreds of steps so while this communication is very costly you amortize it and then you do now step of out optimization so you actually consider that this outer gradient which is as I said not a proper gradient you actually act as it was a proper gradient and you apply an out optimizer um in the fed literature there's many kind of optimizer they use sometime Adam and everything but we actually show in zilco that form training you actually need to apply a much simpler optimizer so ne so it's led with a neester of momentum uh that actually work much better and if you combine basically as in optimizer powerful optimizer such as Adam or mun for example and out optimizer using neester you will be able to train uh for local training so not communicating for dozen 200 of steps and yet at the same performance so the problem is like with the loco is like you do less frequent communication so that's good you can amortize it but when you have to communicate uh that's still a very costly communication of because uh you communicate the world parameters and it's also blocking operation so you only amortize it but you don't fully resolve the problem So you can do uh another solution which is a streaming vlo you can consider that your model is actually split across multiple layers. All the layers could be on the same GPU or across multiple GPU it doesn't matter. Uh and you have like the world model on different GPU still as before they're going to do multiple steps of local training as before but when you synchronize you don't synchronize so with auto varants you don't synchronize out variance for the wall parameters but only for one layer. So you have the layer green of the parameters. So if you have three layers here, we're only synchronizing every 100 of steps, one/ird of the parameters. So that's a massive reduction of memory of communication and the next outer steps. So after another 100 of step of local training, we recommunicate the red layers. So the good thing is like as before, it's less frequent communication because it's only once in a while. It's also less costly communication. That's a way to actually do a form of compression as we only communicate a subset of parameters but without any loss of ML performance that we found in practice but that's still blocking operation. So then we can do an overlapping where the idea is like when we communicate uh that outer optimization we don't actually block. So once in a while we do 10 local training steps we compute the outer gradients for the layer green. we communicate it but then we don't actually wait for the communication to finish but we still continue to do local training and then we replace it. So you can play on this algorithm. The way we do it is very similar. Every H steps we're going to send the communication in line eight of my parameters or my green parameters for example. And then uh in two steps in the future we're just going to get to receive that communication meaning we can fully overlap the communication. The good thing is like if you can overlap the communication with computation, it mean that if your model is actually very long, very slow to take a step when you do the forward and backward very slow to make the computation, you have actually more time to do the communication. So the slower your model is, the more you can overlap computation communication. Uh luckily larger model also usually have longer step time. they just have more parameters, more flops, more data to do and thus even though larger model have more data to communicate, they also have slower step time and thus is easier to overlap more communication. So we show here for example the bandwidth required to achieve 95% computation. Computation is how much time is dedicated to fully doing computation versus doing communication. You want that number as high as possible for data parallel with a 1B model. is like in the 200 gigabyte seconds while is like 200 of less 200 gigabyte seconds. But as you go to a 10B model or 100B model, 100B model is going to be 100 time more data to communicate and you need more time actually more gigabytes per second for data parallel to communicate. But for the locos streaming the loco with overlapping it will require less uh bandwidth only one gigabytes because you can fully overlap the communication and the model is actually much slower. So we have more time to overlap. Um so here like in this example for example this communication this outer gradient that was only a single layers so streaming and overlap was also contized to float for and with one practice there was no loss of performance but you can further reduce that communication for example mulu and do x to paper made by not my group uh they apply for powerd or demo. So the compression method we saw work well for data parallel. You can also apply it on the outer regions in streaming the local. So you can further reduce that thing. As I said before all those methods are not mathematically equivalent. So data par and pipeline par are mathematically equivalent to just train on a single GPU and they are safe. You can trust their results. While our methods they're going to change the learning dynamics. You do an outer optimization steps. You only communicate a subset of the parameter extra. meaning that you don't have the same exact learning dynamics, the same model optimization. So you can be scared that actually doesn't work as well. And yet we see that for example the streaming zoku if you train for example for 200 billion tokens on data parallel versus streaming the loco uh if you were to only have one gigabit seconds communication between your GPUs uh you will take 1,000 hours for data parallel while only 8 hours for the logo. And yet you can find that the performance on both the loss on some downstream evaluation such as tis are often similar. So you can be not mathematically equivalent use much communication and yet be still performance equivalent. So no like I'm showing you like methods uh that I know work well at scale for example doo x that was published by another group in China show that we can make it work this model at 100 b parameters. So we know that we can make uh no communicate much less. So does that mean we can do planetary computation where you have GPU all across the world and they can all train the same big AI just decentraliz? So not exactly. So while the B width is I think a solved problem you see more paper that try to compress even more those outer gradients and everything but it's very incremental in the sense that we already solve most I think of the bandwidth problem. We have another problem which is the hardware failure. uh lama tree from Facebook. Uh so that was like the previous generation last year. They reported that the error failure rate over 16k GPU and 45 days was quite high. So they had um one failure every 3 hours they were using that par when you have a failure in that case uh one GPU fall it mean like you are actually blocking the whole cluster is falling down and you have to reconfigure it to not use the DPU. So a failure on single GPU is very costly and the problem is like if you have like the same infrastructure if you have twice more GPU you're just going to have twice more failure potential and it's there so it's going to become exponential and the more GPU you have the more failure you have and here Lama was using enterprise grade GPU and they have a whole infra team to actually make sure the GPU are trying to not fail. If you want to do planetary computation using consumer grade GPU GPU on your bedroom, it's probably not well taken care of and your GPU is going to fail very often. Maybe every every weeks or every month your GPU is going to fail. But if you use hundreds or thousands of GPUs, it mean that basically every hours or every 30 minutes, one GPU is going to fail and bring the wall training done. It's going to bring the wall training done only maybe for a few minutes the time you configure on a cluster. But if you have too much failure, it's going to take a very big heat on how much GPU and computation you can use. So you want to be resilient to to those failure in streaming loc and I think in local training in general local training is quite attractive method because each um you can replicate your model across multiple location and each of them are going to be uh training independently. So if you have a failure it's localized put that local training bubble and you can avoid it uh you can avoid to put down the whole cluster. Furthermore you still need to do communication but that communication is very rarely every 100 of steps. So maybe you can recover that failure uh qu fast enough to then come back and do the synchronization and because we overlap we can have like some kind of a graceful period where we can try to recover from that failure during that time. Uh but there's also another problem is like if a GPU especially if you use it setting so if you use for example H100 and A100 to kind of different GPU they don't go the same speed uh so even in that setting you still at some point have to blocking communication in streaming the loco at some point you still need to all wait for each GPU to communicate their data and basically everything is going to be slower we all going to wait for the slowest GPU which is not really what I wanted because if you have a better GPU but you want to still use it and not wait for the slowest. So then you need to go to other methods. We had a paper a few years ago or two years ago, one year ago as the loco where basically we do a delayed outer update. I'm not going to go too much into details. Uh the idea is like we saw that we don't want to wait for the other replicas. So basically we're going to synchronize as soon as the replica finish is dozen of steps of local training. It can communicate with this parameter server and never wait for the other replicas. The problem is mean that different replica will synchronize the parameter server at different points in time and they don't really bring the same information. One replica maybe several hundred of steps in the future compared to the other ones and you have like a problem of consistent in between different replica. The way to do it is basically we don't update our outer optimizer state at every synchronization point but we try to have a buffer where we accumulate multiple outer gradients from different complicated moments in time and we only update it once in a while. So this kind of delayed update I'm not going to go too much in detail allow us at a small scale only 150 million parameters to show that we could do a singo where we don't have to wait for the slowest GPU which is a very desirable property. So the question is like no can we do planetary computation training across the world? Not really yet because uh I think bandwidth as I said is not really a problem anymore with those methods. Uh I think the loco might handle most of the hardware failure although we will need to publish or someone need to show it that it can work at a very large scale which is always a risk of course. Um as I said like zlo has been proven by prime intellect and then by um uh China telecom I think the company at 20 billion respectively and 100b that we show we could scale the loco it was synchronous version no one has proved externally on open source that I think could still scale at a larger scale but we have also another problem is how many replica we have so while in data parallel uh which is a mathematically equivalent method to a centralized training if in data parallel you split your batch size across one GPU, two GPU, 4 GPU, you have the exact same results. So you're completely agnostic ML wise to a many GPU, you do distribute your computation. So that's good. If you have more GPU, you just use more GPU and your model would still be the same, just faster. However, uh in a method like local training, Fedopete or DLCO, you actually very sensitive to how many replica you have. So how many different subset in parallel they do local training and uh that's I think something I'm quite frustrated about federated learning which was a field before the uh they did hundreds or thousand of replica but they actually never really care that if you have thousand of replica you're going to have to use much more flops in order to be ML wise performant against a centralized baseline. However, when you go at a very large scale, when you train LLM for example like KI which is one trillion parameters flops it become very expensive. We're talking about millions or billion of dollars. In that case, you want to be your method to be as flop efficient. Meaning that if I give you a certain amount of flops, you need to be able to produce the same performance as a centralized baseline. In our case, if we have DLCO with two DLC replica, so you split your world training into two replica and you use as much flops as a centralized baseline just you communicate less and you're asynchronous, you will be able to be as good I mean wise as a centralized version while not being mathematically equivalent. So that's good. But then if you go to actually more replica to for example to four replica, eight replica or even more. So as we show in this paper the scaling low for that the more you have in parallel that do in parallel local training while being importantly isoflops so you're using as much flops in the overall system you will be worse than a centralized baseline and that's actually a problem uh because if you want to do isolate the failure as much as possible uh you want to have as much replica as possible if you have only two replica and there is a failure you isolate the failure only in half of the cluster if you have eight replica it's only 1/8 e of your cluster that's going to be affected by the failure. Moreover, if you want to do actually plant eye computation, so using all the GPU in all the bedrooms uh in the world, that's going to be between a few thousand maybe millions of GPU that are all different at different location, all very low bandwidth in between and can have failure. So you really want to have like millions of replica in parallel. But if you do that, you can still train your model. It's just going to be extremely inefficient and you will require to put much more flops in it to get as good performance as a centralized bed line and which I believe you really need to be isoflops to use as much flops because to train a very large scale model flops become very costly as you need order of less than it was before in the past. So can we do plan computation? Um so with local training paradigm which I think is the future to do pure distributed/deentralized learning uh and the do family of model which has proved its work quite well. We are no competitive eoflops against a centralized baseline when we have two applica. We use several orderless bandwidth and then centralized baseline which is disabled property. We can fully overlap the communication with computation which is even more. We can overlap even more computation as the model is slower which is nice. And we we show a paper external to us show that we can scale the loco up to 100 billion parameters and still being as good as a centralized baseline. There is some work we did on asynchronous training and other work also in f learning but that's less on LM training large scale. So there is not really yet a published work on a large scale LLM that show that we can do as sync doo as synink local training very well and furthermore there is not yet a work that show that we can scale to eight or 16 replica per lm while being isoflops and as good as a centralized baseline so that's something to explore but I believe that is b to open people ask me oh can we do planetary computation right now not really because of the problem I said before But I think we are very close. A lot of progress has been made uh recently. Uh and it's bound to happen that at some point we will be able to do planetary computation. Um I'm putting here reading list. So I will send the slide and we can probably put on the website if you want to have the links. Uh I recommend the first two books which are how to scale your model using classical distributed learning method. Classical I mean because they are mathematically equivalent to centralized baseline. they are not as good to actually use less bandwidth or be asynchronous but they are safe method and then I'm showing here like a few relevant paper for local training for large scale language model in particular um PyTorch no implemented streaming DLCO in their own codebase so you can reuse that if you want to experiment and likewise um TISM on Twitter has made a nano GPT version for DLCO so nano GPT is a very small GPT lm training that is very easy to tune that's very easy to train you can train it I think in a few hours and you can quickly experiment new idea on top of nano GBT and vlo it for now uh so presentation how to do planetary computation I first show existing classical method which are mathematically equivalent to a distributed to a centralized method but they use a lot of bandwidth they are log step each GPU has to be synchronized to each other at the same moment in time therefore I believe that to do plan computation you need to go beyond that but you will have to take risk. You cannot be mathematically bent to centralized baseline and we have to prove that we can still uh train your model across the world while being is of flops meaning using as many flops as baseline and be as good while bringing much more benefit such as less bandwidth and resilience to failure. If you have any question I can take it. >> Uh so there are a couple of questions in the chat. Uh if you see the bottom >> let me see and go to Q&A >> I cannot can you where the in the message of the column I don't see the chat >> in the bottom right there is nine dots column with meeting tools >> okay I never use screen uh okay uh when data goes from one GPU to a second GPU does it need intermediate CPU can you go directly Oh, I lost my like. Uh, when data goes from one GPU to a second GPU, when data goes from one GPU to a second GPU, does it need any intermediary CPU or can it go directly and is it fast? Um, it's kind of depend of what kind of bandwidth you have. If you were to go from one GPU to another GPU and they are in the same uh cluster, usually you can connect them interconnected. So you don't actually to transfer the data from the GPU to the CPU then to the other CPU then back to the GPU. You can go directly there is interconnection. For example, if you use TPU, you can simply do using the ICI communication to go straight. However, if you want to communicate over the internet like across from different countries your data from one GPU to another GPU, you will have to go to the CPU memory and then to actually transfer it. So it depend to how how far you are going. >> Um Abraham do we use multiple GPU if a model is uh too big? Yes, if your model is too big you have to use multiple GPU. Uh in the case of local training here I'm only showing a way to actually distribute the data axis but each uh local replica in local which different data can use internally for example classical tensor parallel or pip parallel. It does require those method that you need to have the wall model in a cluster that is highly well connected. However, then each cluster can be poorly connected in between. Uh, next question. When training process is broken down for multiple GPUs, do all GPU need to be a similar configuration compute power? Um, yes. Yes. uh when you have like multiple GPUs especially in a like a classical methods which are like not as sync uh you need to have all the GPU to have the same configuration and same amount of power basically they need to go as fast as possible and at the same speed at each other otherwise we're going to have to wait for the slowest so you need to have the same configuration and the same amount of power GPU that's also why in if you use on classical distributed method you want to use it hardware which has h100 versus B100 which have different configur uration, different power, different speed. Uh, you're going to be very bottom leaked by the slowest GPU. Uh, next question from Vibro. Does overtraining not lead to overfitting of the model in case of like small LM tree where the rio is 3661? No. So, um, the problem is like we you probably learned about overfitting, underfitting and everything which was I think very important on generalization where you do the trend validation test set. What's very important in the previous setting, the first vague of deep learning when it was a small model. Now we are in a new kind of setting where basically you are training a model that is so big and you on so much data that it's actually very hard to work. You could work it but even at a small scale scale the model is only three billion parameters. So when you see 15 trillion tokens but that are very diverse, they're not coming from a single domain but from all over the internet. you have so much diversity of data that actually you don't overfeit on that kind of setting. Uh and the hope is actually you actually can just generalize and use as much flop as possible. So it's quite different setting from what we used to learn in classical deep learning. A question from Aon Margoliss regarding 20 to1 scaling law do regulation techniques such as drop out and pruning reduce the number of needed token? Um not really because like if you want to do regularization like drop out or pruning. So in the case of overfitting uh if you actually use less or that it's kind of I think the same question as before we don't really overfeit right now. Uh you could overfeit if you have like for example let's say a 20 trillion parameter parameter model and you have as much data or something here. you could overfeit but because we we just increase massively the amount of data while the model is still quite small comparatively it's actually very hard to overfeit. So you could use a relation technique in case of a footing but that's not about reducing the number of need token. The idea is you still want to see as much token as possible always more and you can combine it with the regulation technique to avoid overfeitting but you still want to see more tokens. That's not a solution to see less token uh anonymously said pipeline parallelism is not good. Do you have any insight for on that? Uh I think uh it's kind of a question of vibe. Some people like to use pipeline parallelism. I personally I hate pipeline parallelism. So I would agree with Ilia. Um the problem with pipeline paralism is like first you have the bubble of time. So you can just Google this thing is like how you actually have to communicate. Um basically you do one computation of one layer per GPU for the forward pass and for the backward pass. But then basically you always have some GPU which are idle. That's quite annoying about that. uh so they wasting computation and there is some tricks but it's very annoying to actually use as much computation as possible in pipeline parallel another problem also with pipeline parallelism is like they are not very great I think for fault resiliency if you lose one GPU then basically you do a forward and then you want to do the backward but one GPU has crash so you cannot do the backward anymore because you lost uh one node in your backward pass so it's actually very poor for the resiliency of course that's my view and maybe But other people such as for example pluralist AI so it's a company in Australia who try to do distributed pipeline parallelism they believe it could be working so I don't think there's a proper consensus on that question from archang is there ever an upper cap on how many GPU should be used or particular size model do we ever see a drop in training efficiency of effectiveness by using too many GPUs uh yes um so first of If you want in less in classical discriminative method you will do scaling loads you will show that I have that amount of flops budget. So how many computation I need to do that or I can do in a given time. Uh I will then choose based on my scaling load. You can find example in the caplan paper or the transa paper. Oh actually need to train a model of 10 billion parameters. I need to train on one trillion tokens for example. And then you can actually use how many as much GPU as possible to be as fast as possible. If you use too little GPU, you're going to be slow because you cannot distribute as much as possible. If you use too much GPU, you could also be slowed down uh because your GPU are not fully working. So if you use too much GPU, you are not you don't have enough for example data paral you don't have enough token per GPU. So they're not using their full uh power of computation and they're actually spending more time on communication and your MFU is going to be quite low. So that's a world trade off and it's actually quite complicated to find the right configuration. And if you look for at the lama paper their MFU so basically how much percentage you actually using computation of your model was very low I think like in the 10 or 20%. uh anonymous your PhD was on control learning and computer vision but you're currently working on distributed modular LLM are those two areas of research connected in some way if not how you move toward distributed modular LLM research so that's good question so I my case was on learning computer vision so like for example you take imaget you learn one of the classes then you learn one one of the new classes and etc you keep adding new knowledge I think learning is still critical for area that's probably the most critical thing. uh however when I joined uh deep mind we thought that we wanted to do modularity so actually how to have a modular system where you have different component learning different kind of knowledge so that could be mix of expert but that could be also something else we believe is very useful and probably needed for content learning because you can compartmentalize your knowledge on different location explicitly of your model that's probably a desirable property however um we also wanted to be collaborative where everyone across the world can contribute their own module inside a bigger AI but then I realized that you need to do distributed because all those modules will be across the world so poor connection and if I contribute to my module you contribute to your module and extra we don't contribute at the same speed as many problem of infrastructure you will have to distribute it so I saw that we will first need to solve proper distributed uh learning and then we will be able to do modular control research so I'm not doing learning anymore but I hope to come back one day when distributed is fully solved. Uh Naf as how do we answer the security and integrity of data over such a large network of GPU? Uh that's actually a very big question. Um so if you if you own all the GPU they're just let's say like I'm just renting GPU on Google cloud. Some of them are in the US some of them are in France some of them are in in China or whatever. here like I'm owning or at least I'm leasing the GPU and I own the data. So there's no security risk. If I do planetary computation where I'm using all of your GPU in your bedrooms and everything, I need to be sure that no one is going to to falsify the data or the gradients doing that plant computation. Uh and that's actually a problem. There's people who try to make some solution. uh it's not really clear how to do it and I think that's a big challenge um and it will probably will come with an efficiency in your training so I don't really have a solution for that um Shamima [Music] is there any rule of thumb for knowing when we might need data model and how many GPU we might need um So I think like it's it's a lot of trial and error you can have like so first like I think in the you should look at those two ML book especially the first one is quite nice it give you like some kind of rule tones like how to actually in your head like estimate what are the communication how much you can fit in a single GPU so that's a very good link uh but otherwise you want to minimize communication as as much as possible um there is a different amount of as I say hierarchy of distributed method based on the communication. Uh data parallel is going to be to be using the less least amount of communication then you're going to have like maybe seconds parallel and then maybe model parallel. So like transfer parallel. So ideally if you can fit your whole model on a single GPU and then you just do data parallel so you replicate the model across multiple GPU that's perfect. You want to avoid doing model parallel if you can. if you really have to, but I try to minimize as much as possible. So, in general, it's a lot of trial and error until finding the right setting for your model given your configuration and seeing whether first it can fit and second if it can be as fast as possible. Um, Timis, are there any efficient implementation for doing ring b on sparse compressor gradients? Uh I'm not sure exactly about that. Uh there is probably variance. I think you probably need to look at the PCCL I think like intellect released recently a library to do a new protocol to do I think a better in that case. So maybe that could be a good direction. So just Google prime intellect. I think that could be a solution. Unfortunately in in Google we have our own implementation which may not be public. So I'm not exactly sure what's exist outside here. Uh can we add slide after the class? Yeah, I will send the the slide to and you will maybe put on the website or the YouTube link seminar can you recommend some paper for beginner that describe the internal parallelism so we can implement them. So I will strongly recommend to go over the two books uh the GDM autos to scale your model and the RF ultra scale playbook that show like classical parallelism method. Uh you can also look at the picotron so from face picot they have a small toy code base and also a YouTube channel where they do from scratch from scratch distributed methods and you can learn that way. So Google pun from face amit praash what kind of errors are this GPU error for the lama tree run GPU fail in sum so has many errors that's a good question what's the errors um if you look here uh basically they show like a list of the failure sometime your GPU fail and sometime it's it's memory will fail sometime there software bug so there's always software bug let's Sometimes so far bug like that maybe the PyTorch has a bug but it could be also maybe somewhere the networking stack or the CUDA kernel or something there's a bug uh there's sometime like a problem um on the actual connection so network switch cable the connection between GPU there's a fail it's hardware it can fail uh sometime it can be uh too old the GPU sometimes actually like the SSD so like the the the CPU you can have problem everywhere uh that's such a complicated system where you have so many moving parts uh that you will always have a failure and it's always will be new failures. So you basically you cannot completely eliminate the failure. So luckily Facebook has a lot of software engineer and they can try to make it as good as possible but you will always have new failure for any win you run. >> Uh and thing so the current best planetary computation is still limited to 100B parameters. Um, so I guess the biggest one that is public is probably this one, I think. Yeah, I'm not sure there's a bigger one. Uh, but it's not limited to one of the parameters there. So like to do planetary computation, but you still need a lot of GPU whether it's even if it's a planetary wise, you still need a lot of them and no one has ex has shown publicly that you can do more than one. But if you can make it work at one, I'm pretty confident you can make it work at bigger scale. Uh anonymous will there strictly be a need for planetary computation? Do you think there's a possibility of algorithmic innovation that is more data and more efficient? Um so there's always like new innovation. For example, if you look at the deep paper, they made a lot of innovation on the transformer mix of expert architecture. um you every like big labs also make a lot of improvement algorithm innovation so you can be much more efficient meaning that if for the same amount of flops you put in your training you get a better model with those algorithm innovation but then you also want to use more computation because it's also will be better so I think it's like it's an it's a mistake to put in a position the algorithm innovation and the computation those go in pair you want to be as algorithmic if innov to be more innov pair flops, but you also want to use much more computation. Maybe you could actually solve a by just making work on one GPU, but we are not really in that paring. The current padding is still to use more flops. Uh Amitra, in the final reading list, you show the JML book does all the same for Python from GPU in general, mostly Jackson TPU. Uh the book was quite um so this book Uh the book GGM auto to scale your model is mostly targeted for jacks and TPU of course but a lot of thing are very similar to GPU is like you do the data parallel or pipeline parallel in GPU or TPU is the same thing you still have to do all gather all reduce scatter those operation are very agnostic to the hardware a lot of are the same so I think you can still read the book and it will still be very relevant for GPU you will have a bit more detail about TPU but that's okay and you can just go to the Next link. So the arch open face ultra scale playbook which is more targeted to pytor and GPU anonymous any python libraries to start parallelism for beginner. Um yeah use picot I should have put it in the link I'm putting here in the chat picing [Music] face. So pick from face is um is a good way to to learn parallelism on a very to code base. Um Vor Kumar maybe not a very answerable question. Okay. But what's the intuition behind training with distributed training which is not very mathematically accurate but still under with similar result as collected matically accurate training. So basically why why like methods such as like for example streaming vlo which communicate much less often communicate only once in a while a subset of the parameters that is overlap and everything or ties and everything why does work as well with two complic um that's there's multiple factors so when you communicate the outer gradients the tractor is actually quite well behaved it's like much more smooth and less nosy than the grad. So when you apply now optimization step is very critical. You can actually do a big jump in the parameter space that can recover most of the mistake. The question like why in literature of fed learning people keep doing inner optimizer HGD and then out optimizer maybe Adam that was a mistake. So you need to have a powerful local optimizer to train a transformer such as like Adam or M. And uh in the outer optimizer we actually show that we need to have a nest. We don't fully know yet why uh that works better. We have some intuition but nothing concrete yet which as many thing in deep learning also it's very hard to pinpoint exactly the reason why. Uh Louisa Corpass what's estimate of planetary compute power of the world as of today? Very good question. Um so I cannot reveal the numbers of Google but you can look online probably of what's how many GPU are owned by open eye Microsoft entropic exteri uh you can probably look at semi analysis that's a blog about those kind of data center explanation that give how much power is for each thing then we have the personal GPU I believe that right now most of the computation is still located into the corporate entropy grade data centers and not in the personal computer. Um so I don't really have a number to give you both because I'm not sure I can give my numbers and I don't also know the other people numbers nest map system for training across GPU distributed across the world is data parallelism the only option or can we have other parallelism as well you can have other like I show for example um pipeline parallelism you people like um Max rai who did the swarm and then you have the purely startups who pick up a swarm and make a compression in between the pipeline stage across the world. You can use pipeline parallelism across the world. Uh personally I don't think that's the way to go but there is research in the direction and they have pretty good results. So that's not necessarily data access what's your foot on crime intellect one and intellect two. Uh so crime intellect is very strong team. uh intellect one intellect 2 are very different thing. Intellect one is basically open DLCO which is open source version of DLCO that they trained I think on 30ish uh and they content in F4. Uh that was good repolution at 10D scale with one token. So pretty good model. Uh unfortunately they didn't compare against a centralized baseline. So it's hard to know how much uh inefficiency they add in their system but also costly to train a model at that scale. Intellect 2 is very different. Intellect 2 is actually a way to do offline policy training for the post training. So here I said in this talk I'm only talking about the pre training because the post training or you do a and you basically you do the VR so enforcement learning from verifiable reward that's very different kind of distributed system where you have like sampler who are sampling and then are sending back to a learner that actually batching it. It's completely different way. Uh both are very interesting and I think both of the prep training and post training are needed. So my thought is that intellect one intellectual are very cool and are the two part of the training. One for the training, one for the post training. Uh was what are the pros and cons of using pipeline parallelism technique for distributed training in machine learning? uh pin parallelism can use less bandwidth potentially than data parallel under the under the right setting. Uh it however required to have like quite a large patch size because you need to do microbatching um to avoid as much possible a bubble of time. The bubble of time means that as it's very hard to not have a bubble of time. There is a paper from I last year I think called near zero bubble parallelism. We try to show that you can reduce massively the bubble parism but you will still have a little bit of um bubble of time meaning like an efficiency in the pipeline parallel where your GPU will be idle and close not doing computation. So that's the main con of pipeline parallelism. The second con I said before is that I think it's less easy to deal with under fault uh and hardware failure. Uh Tatam Tatra sorry in your experience what are the most robust approaches to distributed learning setup that remain effective under set condition like intermittent network failures not out and CM GPU failure. I believe that you need to do local training meaning that um you need to fit your W model on a on a single location. So your model may be on multiple GPU but they're all well connected and then you have multiple cluster across the world that do local training. So they are completely independent and once in a while communicate. I think local training is like is the way to go if you want to do a worldwide plan computation. Uh ten do you think there is a specific algorithm oral choice that help distributed training be more resilient under this kind of issues? Uh so if we talk about bond width you if you have like a smaller model for sure it's going to be better. For example if you use um for mix of expert are great because like you only activate maybe like if you look at the ki that was released yesterday for a given token they only activate 30 billion parameters. So only but the total amount of parameters is one trillion parameters. Meaning that yet they activate little parameters. So they are very efficient but they also have a lot of parameter that when you do all reduce on that parallel you have to communicate much more. So if you have a very low bandwidth using an expert is going to be much more harder than using a dense parameter that is much less parameter hungry. So there is some kind of architecture of choice like this you can deal um also m of expert there's communication you have to do when you do expert charging and etc then but then for the hardware failure you will always have hardware failure regardless of what you use hardware failure will happen anyway it's both of which kind of hardware you're using and how many chip you're using I'm not sure I can ask some more this question it's a bit uh too vast Louisa Kopashim, are you aware of a tool that estimate GPU requirement based on data and model tag and shape usage test? No. No, I don't. I'm not aware. It's not that hard. You should look at like the first first thing I think like using rule of thumb to terms you if you know a model is like the small entry is three billion parameters. So you can estimate very quickly how much is it actually cost to store it in on the GPU. Um so it's going to be three billion parameters multiply by four to store them to have them the amount of parameters in float 32 that you have to store in the in the GPU. Then you actually multiply this by three to get the amount of memory you need for the model and optimizer such as Adam. And then you can estimate like the activation and the gradient based on what's your second length your batch size. So you can make like ru of toss to very quickly estimate whether you can actually fit the model or not on a single GPU and then if you do sharding for example transfer parallel by trans if you do trans across two GPU you just divide the amount of um memory per GPU per two. So it's quite easy. uh PM Salman cam how about a accelerate help in doing distributed training in a node of 6H1 [Music] I'm not familiar enough with face accelerate uh but I think they provide like utilities to do data/pipeline/ transfer parallel so yes it will help you to do distributed ting on on n of six h100 which are well connected in between if it was six GPU that were like across the world the poorly connected then HF accelerate won't be any use of anonymous a little off topic question can you give some guidance on how to study physibility of a research topic for graduate school like if a topic is something that will be relevant for coming years or is it just a head topic um I think it's quite hard I think it's okay to go you know action that not necessarily pan um not easy to answer question I think like reading quite a lot of literature try to read every a bit of every papers on every different sub field so pre training post training distributed uh diffusion extra reading a bit of everything to try to kind of see where the field is going I think it's quite important so does be I guess will be well read and be willing to also switch topics If you believe your topic is not relevant anymore, um I guess you probably should read um there is a manuscript written probably 50 years ago by Risha Aming called you and your research uh which is very relevant on how to do proper research. So it's a you and your research. Um anonymous is there a relation between distributed learning and fed learning? Yes, fed learning is distributed learning. uh is a way learning is basically doing a local training. So that's what I show before. Uh you can argu there's many words basically in academia like everyone name it differently. So you could argued learning or loco is the same thing it's local training I think there's some differences fed learning was mostly so there was a lot of emphasis on privacy that each device must not link the data to the others. Um also fed learning do a lot of replica so sometime in the thousand multiple thousand of replica but then they don't care about flop efficiency uh which was fine because at the time they were training maybe for medical data or small data sets if you want to train a big LLM fed learning don't care about flop efficiency but that's a problem you need to care about fl efficiency fed learning also do a lot of like replica sampling and they also care whether if the replica is actually hostile So whether you can trust that replica or not. Um so it's similar and yet also different in some way. Um can you comment on using distributed learning in vision model? Uh more specifically in biical image domains. Um so if you use classical methods of distributed learning yes you can use distributed learning for vision model for biomedical stuff. Um if you use stuff like dlo will work well on transformer I think it will work quite poorly on convulsion neural networks but probably you still use transformer right now nowadays even for computer vision uh I don't know enough whether it would be very different for biomedical domains I cannot tell you for sure I'm sure if you look in the federated learning literature there's probably tons of paper about that because they care a lot of privacy so you can probably look in that action. Uh Rafael Man, do you see any nongradian model training that can compute? Uh no. Uh that's kind of the dream that everyone would like to train your model without graduing other like neuroscience inspired method. It's kind of a joke that every P students I did the same like everyone we want to find a way to maybe use non method and find other method like based on our science that can train a model. So far no one has managed to make something that is very competitive. If you could make it work that's be great but that's also very very very risky. Uh second question, how will ultra ethernet potentially affect which direction are the most optimal and um affect how the distribution mixture can look like? Uh I'm not familiar with ultraet but even if you have better connection that connection is always like limited to your inside your cluster. If you want to be across countries for example I don't think you will have a super great internet anyway. So you will even for plenty computation you won't be able to just improve the hardware you will have to change the methods uh if you're inside the cluster I think it's still you still need to use a combination of all the classical distributed learning method so data parallel and transfer parallel and maybe pipelines on uh what is contest parallelism in 4D parallelism in pra um I'm not exactly sure because I'm not I don't use that notation internally uh in Google. I think context parallelism is relatively similar to what I call second parallelism where you split the across the second axis. So when you have like multiple token in a sentence for example you can actually split the computation across the second axis uh because um um because when you do the fit forward network in the transformer uh the token are processed independently. So here you can split the computation across the seconds across multiple GPU. But then you need to gather back the token embeddings when you do self attention because they all communicate. But then you can do distributed method such as the ring self attention. Uh Sashima Osain very interesting to collaborate with you on your experiments. Can I reach out you to you? So you can reach out to you. Uh you can reach out to me. Uh however I cannot collaborate uh with external people. Yeah. So I cannot collaborate externally but there is many people who are doing open source of DCO. So I'm put a link for example for the nano GPT and DLCO on internet. You can try to reach out to those or the prime intellect and others and collaborate with them. Unescadar um thank you. You're welcome. Um I was wondering what are your thought about fest not supporting jacks anymore especially with respect to the GDM jacks book you chair is it worth trying to do multiple GPU training with jacks for people who are not at Google so um bing phase don't support jacks anymore because I think most of the pens source community is using pytor so that's fair reasoning from their side uh however jack is still very well used uh so google use js entropic use js hai do js Um so I think many people use DAX it's very I think it's great DAX I love DAX I think the book is still relevant because the book is like not he told a bit about DAX but it's also like a lot of ideas that are very transposable to Pytor so I think it's still worth to read the books um and whether you should do GPU training with Jacks or not if you're not at Google that's kind of depend of your setting it's true that if you do PyTorch it will be easier to have uh collaboration because more people are familiar with PyTorch than DAX. >> Okay, I think that's all the question or did I miss any? >> Uh no, that's thank you so much Artur for uh taking out so much time and answering all the questions. [Music]