LLM Quantization with llama.cpp on Free Google Colab | Llama 3.1 | GGUF

Key Takeaways

welcome fellow Learners in this video we will learn how we can quantize any open-source large language model available on hugging face in few easy steps specifically we are going to quantize Lama 3.1 Model A recently released open source llm provided by meta we are going to utilize Lama do CPP to quantize this Lama 3.1 model the main purpose of llama do CPP is to enable Alm inference with minimal steps and provide state-of-the-art performance on wide variety of devices locally or in the cloud the specific reason why I'm using l. CPP quation method and not any other quation method like gptq or bits and bytes is to make sure that uh the quantise model will be able to run on both CPU and GPU uh the main purpose of quantizing these models are to make sure that uh we can reduce the sizes of these models without hampering the accuracy such that we can run it in low compute resources devices also like usually people have uh their devices of having 16 GB of RAM or 8 GB of RAM with like 4 GB of GPU or 6gb of GPU or sometimes they do not also have any GPU uh for those kind of devices uh with less Computer Resources or low Computer Resources uh the quation of these large language models is necessary so let's get started with the implementation part so in the first step we are going to download this uh meta Lama 3.1 model we are going to quantise this 8 billion size of this Lama 3.1 model so before downloading this we need to log into this hugging face Hub to log into this hugging face Hub we are going to utilize this notebook unor login module so let's run this cell and it will prompt us to enter the token uh as you can see I have already written down this token over here and if you want to create uh your token in hugging phase you can uh go to this profile uh section of this uh hugging phase you can click on this settings and then over here you will be seeing this access token tab you can click on this access token tab then you can click on this create new token and then select this read Tab and provide the name of the token and create token then copy that token and uh over here over in this particular section you can paste your token and uh once you have done that and clicked on this login button you will be uh logged into this hugging face Hub in this particular notebook now once the login has been successful we can download this model so to download this model we are going to to utilize this snapshot download module provided by hugging face Hub so let's run this cell uh one more thing you will notice over here I have provided the base model uh this is the basically the path of the folder where I am going to download this Lama 3.1 model uh other thing is this ignore patterns parameter so if you see I'm ignoring this pts type of uh files over here so the reason is if you go to this hugging phub and then go to that Lama 3.18 billion page you will see uh over here in this files tab uh that there is a folder named original over there a file named Consolidated .o. pts which is of around 16.1 GB and we do not need that particular file as we are going to utilize this safe tensors file and directly going to utilize the safe tensors files and then we are going to quantize this particular Lama 3.18 billion model so let's wait for a while until this particular uh model is being downloaded and it will be downloaded over here I have created a different folder this original underscore model over here this model will be downloaded now you can see all these files have been downloaded in this particular original underscore model folder so let's uh go to the next step of quantising this particular Lama 3.18 billion model so in the next step we are going to clone this sl. CPP uh GitHub repo so you can go over here this sl. CPP uh repo and you can uh see in its description how to uh use this Lama CPP to quantize this particular any uh large engage model available on hugging pH and those steps I'm following over here so like I need to first clone this l. CPP in this particular uh notebook now once this l. CPP is cloned in this particular collapse space uh we can quantize our uh that Lama 3.18 billion model but before quantizing that Lama 3.18 billion model we need to convert uh that model those safe ters file in particular ggf format which is basically accepted by this particular l. CPP so these are kind of file format which are being required to uh use this ll. CPP so uh let's first create a models folder where I'm going to save this particular uh ggf formatted model so let's run this cell it will take few minutes to convert this llama 3.1 8 billion model to particular this uh ggf format and uh one more thing is like you will be seeing this fp16 so uh this particular uh conversion will convert this complete model into half Precision of size floating 16 so let's wait for a while now you can see this hugging face model which was in do save tensor format has been converted to this uh ggf format so uh if you see over here uh in this particular models folder this Lama 3.1 fp16 dogf file has been created uh to convert this particular model this uh Lama 3.1 of do save tenses format to this particular uh ggf format I have utilized this convert uh HF to gf. py file that is provided by l. CPP so uh if you see uh we have cloned this l. CPP over here and under this folder if you scroll a little down you will see this file is available over here so you can utilize this particular convert HF to ggf py file to convert these hugging face models into uh ggf format so uh now we have converted this the next step we are going to build this Lama CPP such that we are able to use this llama quantise uh tool to quantize this particular model you can see I have created uh this build folder under this Lama CPP folder and then use C command to build this Lama CPP so let's run this cell it will take few minutes like 5 seven minutes to complete this particular command and build this Lama CPP so let's wait for a while so now this build process has been completed so in the next step we can utilize this uh llama CPP to quantize our Lama 3.1 8 billion model so for that part we are going to utilize this particular command so if you see we first need to change directory to this particular Lama CPP uh build and then bin so this is the build folder that we have created in the previous step we need to change directory uh we need to change our current directory to this particular path uh then we can utilize this Lama hyen quantize tool to quantize this particular Lama 3.1 8 billion model uh in the next few parameters you can see first I have passed that model path that ggf model path and then I have passed the uh output model file path which I want to create and after that I have passed the quantization method type I have utilized this Q4 km this will quti this model to 4bit and then it will utilize less memory and provide more efficiency so let's run this cell and wait for a while to uh complete this quation and if you see this will basically save this particular quantise model uh to that same folder that models uh folder this one you can already see this file has been created as uh now it might be having less size so you can see over here it is only having 7.47 uh MB size so let's wait for a while now this quantisation process has been completed uh it took around 40 minutes to quantize this Lama 3.18 billion model and over here you can see the size of the quantized model so it is of around 4.58 GB next uh we will be utilizing this quantized model for the inference part and for the inference part we are going to utilize a rapper around l. CPP we are going to utilize a python wrapper uh such that we can use a python library to inference these quantied models so let's run this cell uh one more thing you will be seeing is like I'm using this particular version to install this Lama CPP python Library the reason is uh I was using the latest version also but those versions were not compatible with this uh Google collab that's why I'm using this particular uh version of this Lama cpv python so let's wait for a while to install this particular library and once it's been done we can uh use this library for the inference part now once this library has been installed this Lama CPP python Library uh we can import this Lama uh module from this llama CPP now we can simply use this Lama uh module and pass the model path uh for this particular model path where we have saved this model this Lama 3.1 Q4 km. GF model and uh we will be able to use this uh model to inference so let's run this cell now now this is loading the model you can see all the parameters over here so you can see uh the cap type number of uh vocabulary size uh the context and all those all parameters are uh written over here one more thing uh like you can see over here currently it is using only CPU uh it is not using any GPU uh because you can see this Bas parameter is uh zero which means that it is not using any GPU uh for now now this model has been loaded so let's uh use this model for the inference part you can see I passed few simple parameters uh for the generation over here go uh as false it will make sure that uh the uh users query will not be printed in the output and the top K uh as one which will make sure that uh prediction is greedy so let's run this cell and uh let's see how does it respond so now we can see uh it has provided us this result uh the influencing with this conted model has been done and the question that I have asked to it is like which country hosted 2018 FIFA World Cup so like over here you can see it has given us this result like 2018 FIFA World Cup was the 21st FIFA World Cup and over here we can see like uh it has provided us the result like it took place in Russia from 14th June to uh 15th July 2018 uh the only thing that concerns over here is like it took around 2 minute to run this particular query in this uh CPU version but that's okay for now as we are able to quantize this model and we are able to successfully uh do inference using this Quan model now how we can utilize this model at a later time so like uh once this session with this Google collab space uh is ended how we can utilize it in our local system or how we can save it somewhere such that we can utilize it at later time so there are two ways one is we can uh push this model to our hugging face uh space and another one is uh we can save it to our Google Drive so like uh saving to Google Drive will make sure that uh your model is private and this will be only accessible to you and if you want to push it to hugging face up that model will be uh publicly available and if you want to uh save it privately in hugging face up you might need to subscribe to their pro version so to save this quantise model in Google uh drive we first need to mount uh our Google Drive to this particular Google collab notebook for that part we are going to utilize this drive module from this google. collab Library so let's uh run this cell over here we can see it has asked to connect to Google Drive let's click on that and let's access uh our particular uh Gmail account where we will be giving this particular notebook access to our Google Drive uh once we have done that uh we will be able to access that particular Google Drive where we can save this model now to save that model uh over here you can see I have created a folder in our Google drive under this my drive I have created this llama models folder and after that I have used this uh subtle library to uh basically save that particular uh quantied model to that particular folder this llama models folder I have already done that that's why I will be not running this cell again but if you uh run this cell you will be saving your this quantised model to your Google Drive now to save it to your hugging face workspace you need to first have a access token which has right access so you can go to your uh hugging face web page and over here in your profile section you can go to this settings and in this particular access token tab you can create new token and while creating new token make sure that you select this right tab once you selected this right tab you can create your token and then copy that token and then utilize it over here so like you can see this particular token I have used to log to this particular hugging face Hub now after logging into hugging face Hub uh with right access uh we will be utilizing this HF API uh module to basically push our this particular Quant model to hugging face Hub now to save this particular Quant model to your hugging face Hub workspace you need to First create a model ID so first part in this model ID is my username and then second part is basically the name of that particular uh quantise model workspace that I want to create uh the other few things that I have passed over here is like path of that particular file uh from where we need to copy that particular model and the path at which we want to save our model in our particular hugging phase Hub so these few things we need to pass I have already done that and if we copy this particular model ID and search it in this particular hugging pH subub over here we can see that uh this model is available and If we go to this file step this Lama 3.1 Q4 km ggf file has been stored in this particular hugging face workspace now in the next step we will be utilizing this quantied model uh with GPU such that we get better inference time and also reduce uh model load time so for that part first we need to change this runtime type and uh we can do it from here we can change this to T4 GPU and save it then it will be connected to a T4 GPU now as we have connected to that T4 GPU workspace we will will not be having that particular uh model that we have conted over here in our that content uh folder that's why we need to F that model from where we have saved that model so as you have seen that I have saved it in uh that hugging phe of workspace I will be downloading it directly from there I will be utilizing this snapshot download module to download this particular uh contage model that I have saved it in hugging face workspace so let's run this cell and uh we will be downloaded that particular quantized model in this quantied models folder so over here you can see this quantized models folder has been created and this uh particular file will be written under this particular folder so now you can see uh this particular model has been downloaded so let's utilize that Lama CPP python to basically uh do inference using this quantied model now to run this uh quantied model in GPU first we need to install that Lama cppb python uh with g GPU extension such that we can utilize GPU to inference it so for that part uh we are going to utilize this command like we will be installing this Lama CPP python with extras like we will be utilizing this particular extension to install this Lama CPP python where I have mentioned like I need to use Cuda 12.2 so like you need to see which particular Cuda version is available that part you can run nvi and over here you can see Cuda version 12.2 so similarly I have used cu1 22 which specifies that I need to use Lama CPP python which can utilize Cuda version 12.2 so let's run this cell and this will install our Lama CPP python in a GPU version so this library has been installed now uh we are going to utilize the same steps that we have used earlier so like we will be using this llama module from this Lama cppp and we will be passing the path of that particular uh model that quanti model now one more thing that we need to mention over here is the number of GPU layers that we want to be offloaded to GPU so I have passed minus one which makes sure that all the layers has been uh offloaded to GPU so let's run this particular cell let's first remove these lines uh let's initialize our model then uh after that then once model has been loaded we can run inference on that particular model and let's see how much time it will take this time so now this model has been loaded uh now we can run our inference part so let's copy the same arguments also that we have passed earlier so let's uh copy that directly so we can copy this one and we will be running the same over here so that we can see that how much time it will take this time so let's copy paste over here and instead of llm let's change it to model and let's run this cell and see how much time it will take this time so here you can see it ran within 6 seconds and over here we can see the result it provided the similar result like 2018 FIFA World Cup was the 21st FIFA World Cup and it took place in Russia from 14 June to 15 July 2018 so it's great we can see that by utilizing GPU it's really faster than when we were using uh CPU to run inference over this quantise model so now you have got clear understanding of how you can quantise an OP Source llm by utilizing Lama CPP and then you can save it to either your Google drive or hugging face Hub and then you can utilize that same uh saved model for inference part at later time and the major benefit of using this l. CPP is that we can use this qu model and do the inference uh either in CPU or GPU both so like you can uh do inference in your local uh local system also which are having less Computer Resources as you can already see like this model is having only size of around 4.58 GB the original model was taking around 16 GB of his space so like you can uh load this model in your local system also where you are having less Computer Resources so great thanks for making till the end of this video goodbye until the next time