GPU VRAM Calculation for LLM Inference and Training

Key Takeaways

hello everyone welcome to AI anytime channel in this video we're going to explore that how we can calculate the amount of vram is required to you know perform llm inference so most of the time you know you are not sure that how much of vram you know the GPU Ram that you need is required in order to run or inference a large language model so for example if you talk about Lama 3.1 8B model how much of GPU I need do I need if I have a single GPU how much of vram which is video random access memory is required to inference so I'm going to show you a basic calculation and different tools that you can use to make an estimation you know of the vram is required for both inference and sometimes training as well so let's jump in and see how we can do that now we have to understand uh how do llms basically use vram okay uh the uh let me just write it over here so I'm just writing vram uh and that for video random access memory now it's say special type of memory guys that used in gpus uh almost 25 years old were in invented in late 90s 1990s you know for uh image and Graphics rendering so this basically for uh high computations so high computations mainly mainly for accelerating accelerated compute that's what were required you know for image and mostly the graphics rendering you know to deal with the graphics rendering things uh the purpose it doesn't it's very evident that we also have CPUs memory you know we have CPU and we have memories but we we cannot exactly use it I'll tell you why you know when it comes to uh get a better performance when we are influencing large language models the purpose of this V Rams was to upload uh you know the tasks like image image or Graphics ring or anything from CPU for efficient and faster uh you know rendering of if you talk about image then complex visual scenes because if you have played some game you will understand the games like FIFA and all where they have high quality uh visuals that they have right now and that's where it came in so it basically offers A high bandwidth compared to the standard system Ram that we have and it allows faster data transfer so we should keep this in mind that's that's the that's the main thing so I'm just writing faster data transfer faster data transfer between the gpus if you have multiple gpus as well or if it was single GPU and it's memory so GPU and its memory the data transfer will be faster which enables processing large amounts of graphical data and nowadays you know different formats of data that we are dealing with so that's it started guys and we're going to you know see how we going calculate it a few things so if we talk about why it is important let me just write it over here for you so you can see Let Me Maybe we can make it bigger okay so how do llms use vram so that's that's the question now here okay so let's just how do llms use vram that's the thing first is talk about model parameters so let me just write it over here model params second is your activations and third batch size and then fourth four is your Precision these four are the uh important topics or categories when you categorize the use of vram when llm in when you make an llm inference so inference is a very fancy word you know to basically generate something out of an llm okay basically we're running an llm to uh do a next token prediction or or check or instructions whatever now in model parameter what happened you have millions and billions of params and now trillions as well so millions and billions of params that you have now these are stored in vram during the inference basically we store them so storage now in the activations was what happen each layer of an llm generates substantial amount of activation data which is temporarily stored in a V Ram so basically you know you do a temporary storage of you know all the activation data that you have know uh so temporary storage of activation data and then you have batch size you know which is pretty much self explaned to you so if you have a larger batch size more vram is required because then the model has to handle uh you know a very large batch size having inputs and that the do it simultaneously so it's it's a uh you know it's you would have seen this in deep learning as well when you used to work with NE networks earlier so in the batch size what what we see if you have a larger batch size then more vram okay so it depends what kind of GP have Precision you know uh basically the floating Point how do you want to how do you want to use it so there are different so you have FP 16 floating Point 16 32 int 8bit int know 4bit blah blah blah so you have different uh things over here that you can look at so this this is how the V Rams are distributed so it starts from model params activations batch size and precision so that's how it works now the formula that we see is pretty much the if you talk about a simple formula it is number of params and then you multiply this with precision and this Precision is divided by you know uh 8 and then you just multiply all with 1.2 now This 1.2 is nothing but the 20% that we are and there are some assumptions of it and I'll show you the different tools that you can you probably don't have to you know use this formula because a lot of tools that you can use it 20% activations and other things like other uh and other memory that you have in the process of inference now number of parents for example if you want to talk about about M let's talk about M 7B or any model if we talk about M 7B the number of params becomes 7 billion so it has seven and it always has to be in billion if you talk about this formula so seven and then imagine if I'm loading the model or I'm going to use in FP 16 so if I have 16 uh floating Point as a precision and then I multiply all with 1.2 so this total become I don't know how right it becomes two and 1.2 it becomes 4 14 1.2 so basically you need around 16.8 GB of vram 16.8 GB of vram is required if you look at this formula now there are some assumptions the assumption is Max sequence so max sequence is 512 for this assumption so you have Max sequence 512 you know then you have batch size which is eight and GPU is one only one GPU we talking you can have multiple GPU and then you can offload and distribute a you can do a parallel uh parallelism over there when it talk talks about different gpus that you have now this is a pretty simple and straightforward formula nowadays we have better optimization techniques uh there lot of Frameworks or libraries that you have you're starting from Onslow VM blah blah blah based on uh paged attention uh rope scaling and whatnot so there are different techniques that you can use to basically optimize the need need of vram you know in an llm inference now but this is how a very simple one works now I'll show you different tools that you can use and I'll give all those Link in description so you see this let me just refresh this we have different tools one is called vram estimator it says estimate GPU vram uses of Transformer based model so I'm going to focus more on Transformer based model only talking about op source and Transformer based model now here you can see it says inference and training so if I click on inference running params then you have fp16 bf16 fp32 so for example if whatever is default let's select that f is 16 Max sequence length 512 batch size if I increase that to 8 number of GPU 1 and if I select a model from here you can see you can find out some model and if you if you see and you can again you can change the you can see here what is what it says it says model parameters could be taken from configuation on hugging face or directly from model V model. config now if I select for example if you want to do mistal 7B okay in this case you can see number of params number of layers vocabulary size hidden size blah blah blah so these are the if I make this in GB as you can see very close as we said 16.8 GB of vram here you can see 16.2 one9 GB of total vram and also shows you and how much of his uh distributed you can see the cudak kernels we can look at the parameters and we look at the activations and output so mainly the distribution is on parameters so it gives you a distribution uh that how these have been calculated so if we talk about the parameters used number of parameters is 7.51 billion then it multiplies that with number of bytes per parameter so two number of bytes per so two bytes per parameter you can see it over here then it becomes 13 almost 14 GB of vram and then some activations and then output 10 so okay and then Cuda kernel over here so you can use this tool you can also use it for training if you want it gives you how you want to train it on mixed Precision if you want to use an Optimizer for example Adam Optimizer and keep this you know and for example this looks good hold on for a second total V is 152 first moment second moments now if just make B size two if you if you if you change the bat size over here if you look at here if when you when you made it two it drastically reduce the uh GPU need number of layers 32 okay if you do it on these configurations of model parameters if you do it on full you do on SGD sastic you can find out the chain uh in the GPU vram over here so you can use this tool if you want I'll give this link in description the other tool that you can use which is by HF XEL rate hugging face Excel rate that's called Model memory calculator now using this tool you can calculate so for example you know if I just want to show you something here so you change model name or URL so for example if you want to use whisper an open source model by open AI for speech to text tasks now you can select Transformers and if this is fine and you can see the need of this so basically it shows you the memory uses for open AI whisper large V3 and training also using adom so it also explains that below now here it shows the memory uses you can find out all the details that you need the the D type the largest layer which is whatever is there in the model the total size and training using Adam blah blah blah you also use model memory calculator to also test it out you can also use this can you run it uh llm version you can find this is a bit more uh comprehensive and based on uer AI Transformer if you guys don't know I will recommend you read this Transformer math one1 this is a fantastic blog written on Uther AI is very old you know it's like almost more than a year old and you can read that now you can find out you can also use this tool know pretty much straightforward you can you know bring up your own model here if the model if the model is sitting beside behind a GED repo or then you have to bring up your access token you have to change your model name you have your GPU vendor over here and then you have how many G like which GPU you have A10 a00 blah blah blah so you can also use this bit more comprehensive this is the simplest one that you have and this is where you can read it and you also use this this is what I wanted to do in this video guys I wanted to show you that how you can you know calculate the amount of vram is required GPU is required to you know do llm inference and there are other tools as well but the these are the tools that I refer to and I'll give all of these links in description if you know any other tools please comment it down for our audience guys that will be beneficial for us that's all for this uh video guys if you like the content please hit the like icon if you haven't subscribed the channel yet please do subscribe the channel That motivates me to create more such videos in your future thank you so much for watching see you in the next one