How To CONVERT LLMs into GPTQ Models in 10 Mins - Tutorial with 🤗 Transformers

Key Takeaways

to convert a large language model that is in a pytorch file format or safe tensors format to a gptq quantized format this video tutorial is going to directly help you to do that typically people who do not have large gpus want to run large language models prefer a quantized model because quantized model would fit in less compute resource machines if you know the block is one of the most prolific quantized model releaser I've ever seen so in this video we're going to see a new feature that came with hugging face Transformers where the GPT queue has been integrated with Transformers making it easier for you to use gptq models and also making it easier for you to export gptq models or quantize the regular llm into gptq models let's go ahead and start with this entire code that I'm going to use in this video is from this blog post making llms lighter with Auto gptq and Transformers all credited to the team that put together this Google collab notebook I'll link the Google collab notebook in the YouTube description to make it easier for you to get started once you reach the Google collab notebook the first thing that you need to do is you need to click connect make sure that you have got GPU already available in this case we are going to use an Nvidia GPU a T4 machine that has approximately 15 gigs of vram the GPU Graphics memory once you have connected this then you have to install certain libraries I'm going to install Transformers pift accelerate and Optimum so install all the libraries click run away because it says this is the author who created this do not share any details that's fine click and install all the libraries the next thing that you have to do is you have to install auto gbtq until the next release of Auto gbtq you can you can build the library from The Source but after the latest release of Auto gbtq comes in you don't have to do that you can directly do pip install or gptq so let's click install this as well once you do that then the next step is for us to know realize what to do so one we can either quantize the model like a language model llm from scratch or the second option is for us to load the model that has been already quantized this video is particularly focused on quantizing a large language model but in the future I have plans to make a fine tuning as well fine tuning of a GPT EQ model for you to quantize a large language model from scratch the first thing that you need to know is the gptq quantization method is a post Training quantization Method so if you go here you can click this and then see the paper it's called post training quantization that means you are going to train you are going to quantize the model after the training has been already finished so there are certain nuances about how do you do that and that is why for gptq quantization you need data set so the gptq quantization happens by doing inferencing over the data set so you need to pass a data set that the quantizer will use to quantize the model in and itself so you can use any of these default data sets or you can you know just hard code the data as well but the quality of the data set that you give also reflects in the quality of The quantization Happening that's something that you need to keep in mind so here what we are going to do is we're going to use the C4 data set as the data set that we are going to pass into the quantization configuration you can pause in any other data set that you have but just keep in mind if you hard code as well like you can have like a single line as well but then it has its own impact in the way the quantization has been happening so the better the data set is better the quantization or the lesser the accuracy degradation will be okay so how are we going to do the quantization the first step is from Transformers input Auto model for causal Element Auto tokenizer gptq config this is the new feature then you have to specify the model in itself in this case we are going to use the Facebook opt 125 million parameter model next we have to specify the quantization configuration what is the quantization that you want to do what is the Precision to which you want to bring down the quantization to it supports two bit quantization 4 bit quantization six bit quantization and 8-bit quantization and then you have to specify the data set in itself and then you can start the quantization let me run this the first thing is it's going to download the model after it downloads the model it is going to use the quantization configuration and it is going to create the quantized model as part of this tutorial we are also going to learn how to push this quantization quantized model into your hugging phase model Hub which means you can use it whenever you want to it to be used once the model has been downloaded once the quantization has been finished which would take approximately less than 5 minutes as you can see trained split has been done and the quantization has processed has been started at this point the quantization has been successfully done as you can see now let's run this and then see if the quantization has been proper so this is just to make sure whether the quantization as a process has been successful it does not validate whether the quantization inference process which means the model is performing good or not just to make sure whether the quantization has been done properly what we can do is we can look at the decoder layers and we can look at Q weight and Q zeros and then make sure they are of torch in 32 data type so if you see Q8 of torch in 32 data type and q0 of torch int 32 data type which is just here and you know that the quantization has been done properly once again just to click verify this does not make sure whether the quantization is of high quality this just ensures the process called quantization has been successfully done after this is done you can now do the inference on the quantities model we can use the same API so you can just call the tokenizer and you can give a text and then you can just expect it to return something in this case you're going to use the quantized model instead of the base model that we have downloaded the start so hello my name is this and then it gives you the response so that means the quantization has been successfully done you can go ahead they have also given you example of how you can hard code the quantization data set and then use it but if you happen to do that then the quality of the quantization may not be highly good so let's try one more thing question write a joke about Mac computers answer so this is if you know this is not a chat model that's why I'm giving this in this particular format and you can see that it did not give us anything in there anyways the point here is that the model works so the next thing that we are going to do is after the quantization is done we are going to upload the quantized model into our hugging phase modeler for us to use it later on whenever we want the first thing we need to do is we need to authenticate our notebook which means we need to go here and then use notebook login click this this will open up a box and here you have to add your hugging phase token to get your hugging phase token go to your hugging phase account and click the profile icon and click settings once you click settings you can go here and find the token access tokens and copy one of the tokens that are available here come back to the Google collab notebook add it here and then click login once you click login it says login is successful token is valid the permission is right now all you have to do is you have to push the model to the hub so the model name whatever name you want to give you can give and then just run it and this will push the quantized gptq 4-bit quantized model into hugging phase model of which you can use later on whenever we want to use it so model has been pushed here we can click this that will open the link here and you can see the model has been pushed so this is opt 125 million parameter gpdq 4-bit model under one little coder and that is where the model is available now to quickly use the model that we just created so you can go here and then select the model that you want like for example in this case if this is the model that we have used for example you can go here copy this link come back to the Google collab notebook select that model here or you can use the block if you want or in in our case if you want to use the model that we pushed then you can just simply load it nothing else you can simply use it with model tokenizer and just simply load it using the same way you load any large language model from hugging phase modeler once the model has been loaded now all you have to do is do some text generation just check the model in itself you can check the model and the next thing that you can do is you can just start load creating the text let me run this hello my name is something something and then run this and this is going to create okay hello my name is Kari I'm a student of University of California San Diego and it just repeats a bunch of things because we have specified to create a lot of tokens and again this is not the best model that we have got um if you happen to use one of the latest state of the art model then this might have a much better result they've got the code for quantizing or you train fine-tuning a contest model as well I'm going to make a separate tutorial but in case if you want to use it before I make the tutorial if you do not need a tutorial in itself you can go ahead and then start using it in the same Google collab notebook which I'll link it in the YouTube description so to quickly summarize in this video we learned how to quantize a large language model into gptq models and we also learned how to make it into different precisions like two bit four bit six bit eight bit and we also learned about how to push the model finally to the hugging phase model Hub and finally we also learned how to use the model for inference if we have already pushed the model to hogging phase model I hope this tutorial was helpful to you if you have any question let me know in the comment section otherwise Happy prompting