CodeGemma 2B & 7B LLM by Google: Open Source Coding LLMs

Key Takeaways

hello everyone welcome to AI anytime channel in this video we're going to look at code Jemma so code JMA is a new code based llm you know that has been created by Google it's based on uh JMA model of course uh the earlier you would have seen that a couple of months back Google came up with the open source llm you know after a while they they put their you know feet in open source landscape and they come they came up with a Gemma model now they have released two models code Gemma recurrent Gemma and the model but we going to focus this we want to focus uh code GMA in this video uh they have three different variants in this so variants are two 7B and 2B but there are three different models weights that they have released in 7B there are two models code GMA 7B code GMA 7B it which is the instruction team model uh and then we have code Gemma 2B so there's a 2B variant as well that we have now the underlying know the concept Remains the Same going to generate something for the 7B and going to complete for 2B so if you want to you know uh perform natural language uh like task using the code Gemma 2B probably that's not the model for you okay because it's it has been created for code completion not generating code based on natural languages it does not perform that well but I will show you with that that model only that I'm going to perform for ask questions uh you know generate python codes and things like that using 2B only not going to rely on the 7B okay but we can try 7B as well that's not a problem now when you talk about how they have been trained the 7B model has been trained on 80% of the code Corpus and 20% of the natural language Corpus the Text corpus that's how the data distribution have been for the training when the pre-training part and then have been the instruction tuned as well for the code GMA 7B it for the 2B model the only uh Data Corpus is 100% code that's what they have done okay and you know they have been they of course it has been done at probably deep mind you know there they would have have been created this model and this is what we going to show it has uh performed good I have tested it out and I'm going to show you for a few of the questions and how you can do that now if you look at here on my screen uh let me just wear this glasses now code JMA 7B by Google it's a GED model so if you want to if you want to test this out you have to acknowledge the license okay uh so as I said about the pre-training and the instruction tune so the pre-training part so guys right so you have to understand first there should be pre-trained to guess a base model then that there should be further fine-tuning for you know Downstream task like if you want to train a instruction tune you will not immediately do that okay so you'll first create a pre-train model and then you will uh probably Alig it or further fine-tune it enhancing the models models logical reasoning and capability and that's how the instruction tun have been done okay now they have performed well on some of the coding capabilities I have checked for python they have the language capability for the 7B model and this is what it is okay now let's go back and see how we can write out it's available on Google suing face repository you can see it over here it says code JMA 7B uh and you can find out uh different things over here so this is the table that you should go through it so code completion is available on code Gemma 2B generation from natural language is supported in both the 7B model not the code Gemma chat and instruction following but I want to show you a few example with code Gemma 2B only that it it does the job they have given you some code completion some prefixes some some suffix to put that in a prom template kind of a thing okay that basically differ different parts of the input that basically creates a separation now I'm already have installed Transformers and accelerate I'm using Google collab probably you need a basic GPU T4 GPU will be enough for you know to be model so feel free to use that now after doing that I'm going to do from Transformers import Gemma tokenizer so it has a Gemma tokenizer you know the part of Transformers class class if you're working with the Llama model probably you'll be using a llama tokenizer now this is done I'm going to have Auto model for Cal LM because we're going to generate something that's why I'm going to have this and the other thing that you need is for that let me go to my hugging phas yeah this is what we going to do we're going to do a notebook login guys because the model has uh a gated it's a gated model it has it sits behind a it behind a kind of a wall where you have to first go and knowledge the uh model licenses and things like that then that's why we need a access tokens let's get the access tokens from here and I'm just going to copy any one of these these are all these are all having read accesses so I am able to inference that let me just do that here and I'm going to pass this and this will validate okay now you can see login successful and after that let's get the tokenizer so I'm going to do token toer equals and then I'm going to pass Gemma tokenizer GMA tokenizer do from pre-train let me just get rid of model as of now okay we don't need let's focus on this J tokenizer do from pre-train and this is not the model that we have it should be it should be Google so let me just go back here okay and let me just copy this and this will get the tokenizer so let's get the tokenizer and you can see we are done with our tokenizer it gets the it gets the config Json and tokenizer Json with the special token map Json files and downloads the tokenizer excuse me I have a sore thr now you have a tokenizer model which is 4.24m you can see it over here so we are done with that now the model and the model is again it it should come from Auto model for Cle LM so we not doing a mask language modeling we are generating text that's why we're using Auto model for casual LM from Transformers and then I'm going to pass my code GMA let's pass oh I pass 7B let's pass 2B I'm I'm going to try with the 2B model so I'm going to pass 2B over here and you can see then uh this is fine this is fine this is fine let me just refresh this and this goes Tob device map Auto is fine and I'm going to remove this by the way and I'm going to pass torch D type okay so let's just get torch D type equals and this is going to be Auto and let's run this now it's going to download the model and we're going to test this out for a few you can probably take the code from hugging fish repository as well the idea here is to test some prompt and see how the model is performing because the size 2 be size you'll be able to run it on CPU as well so if if you have compute limitations I'll recommend that you should check this model out right on some of the use cases that you'll be probably working with and that's a great uh that's a great effort by Google I will say to release this code based model we earlier have had stable code and 52 also generating good Cod code so we have a stable code which is around 3 billion and also there was a 1.5 billion model as well and this 2 billion model it performs well then we can bit on this right for some the coding huge cases like in Code test cases Generations code refactoring code review and whatever we do in the hdlc life cycle you know software development life cycle you'll be working with different hug cases and if this can work on CPU you know in tandem with some other smaller models in an Enterprise that's what you need right so let's test it out so I'm going to do a prompt here so prompt equals and I'm going to say let's try it out with a fast API framework first so write write a python fast API uh function for calling of wether API okay this is and I'm I'm I'm using a natural language guys here it's a bit of natural language uh understanding here that model SW has okay that's why probably it will be able to generate otherwise it will fail and let's have inputs and then I'm going to write inputs and tokenizer so tokenizer and then pass the prompt so prompt has been fast and then return 10 or PT and then I'm going to bind that with model. Device okay so whatever it because we have that device thingy done let's run this and if you just say inputs you can find out all the tensors okay you can again decode this out if you want to do that okay now we'll not do it yet okay we're going to decode the output that once we generate so I'm going to say prompt prompt length is equals I'm going to pass input and then have an input IDs you know all of these had IDs if you look at over here it has IDs over here and that's what we're going to pass here input IDs and then just save that was right but it should be minus one okay now save that is done method object is not Sub sub subscriptable what did I made a mistake promt length input ah it should be inputs not input and prompt length it says input IDs uh let me just run this again again inputs and that should be input I'm making a mistake here that should be input ideas and now if you do a prompt L again yeah 13 as you can see fantastic right these are all we used to to uh when earlier we started our career guys we used to focus on all of these what is T or we started learning all of those uh intricacies and all the deep deep dive you know into tensors finding out loss functions or whatever not going into this but these are these are the fundamentals that you should understand now let's use model. generate so outputs equal I'm writing it right output and then I'm going to use model. generate so model. generate and then I'm going to pass inputs and Max new tokens let's keep it is probably 150 or something okay so as Max new tokens it will take a bit of time guys to generate and then we're going to print outputs and let's print the outputs once you print the output you can see you get a set of 10 Sol over here now we have to decode this so we're going to decode this so let's do that so I'm going to do print and tokenizer do decode output zero and then we don't probably need that we don't need to skip ah so this becomes this output zero and then I'm going to write prompt Len and colon this should make sense to tokenizer do decode outputs and you can find out fantastic we and of course you could have used SK special tokens too then to get the better output now if you look at this I'll make it a bit bigger this is the right answer at least for this kind of model I accept this answer guys you know for this uh let's see it gives you define get weather and we have of course import request Define get weather URL this is the open weather map so this is able to understand what kind of question I asking so it got that then data response Json and return data and then City input enter weather data get WEA fantastic I mean I I I liked it you know this this is the right response for this question that we asked let's ask some other questions as well okay so this is the question that I have asked okay and probably I'll just copy this here prompt would have created a function just uh let me make it a bit small 110% this is fine after prompt let's get the prompt lent here so PR uh not this tokenizer input tokenizer here goes the tokenizer and then we going to use prompt L and this makes sense guys okay let's change the question now so what I'm going to do is what question we can ask here uh let's ask uh I I want to ask a tough question okay uh the question uh let me think how to write the code write the code in python or or let me try it out one thing okay I'm increase the max new tokens to 250 and I'm going to take a a doc string I'm going to C copy this just just this piece of code that it has generated and let's trite out an interesting task okay I I have copied this okay this is fine I mean that is fine so let me just get this rid of this let me just get rid of this and then what I'm going to do now is ask the question uh write the test cases for the function let's generate this now let's see if it able to it's able to get that nothing okay so didn't get that okay so just a 2B model it's not good for you know probably generating test cases out of it on natural language on this large Cod s because that's the c s that you are putting it's more of generating code of the code completion but yeah what 7B model would have got that okay and uh let me see what's wrong in the code it's not getting anything now that we have prompt we passing the prompt now let me ask to generate a question and see if it's able to get that uh a python fast API not that question now I'm going to ask a simple question write uh write a python code to generate 1,000 random numbers or random integers rather in integers and it's it is given you so that's that's the idea right so it's good for generating code the 2B model so with the 2B model as it has said here as well that probably will not work out you probably need a 7B model to uh work with that uh 7B model to does the inferencing and all those things that we wanted to do it here okay now you can you can make a change here as well okay if you make that as 7B you get the 7B thingy let's try it out probably I should not get a Cuda out of memory error because 7B model I'll be able to use on b00 even if I have something uh in the volume so when I say in the volume the 2B model is it has been has been persisted in the volume on Google collab so it should probably not throw me error because that does not uh required a lot of gpus probably a lot of space it's not about GPU it's about the space as well first that it should not say that no not no space left on device and then we'll try to load this on v00 and see if it's able to run it if not then probably I have to delete the runtime disconnect and delete runtime and start again for the 7nb but I just wanted to check this out okay and that's why let's try it out and see required okay you can see it's getting all the models and I want I want to ask the same question guys okay so let me just copy it here write the test case right the test cases for the function and then below goes my function yeah and this is fine if if it's not able to understand this piece of code then it's not a good model I will not recommend to even use that model if it ask this question it's not able to understand what I'm asking the SB model then don't use it okay uh let's try it out so you can see that it's downloading the SS two of it I'll just close this I don't need it now so let me just get rid of this this this this this this this and this and then we have the prompt thingy and the inputs and then we'll able to run the model let's try it out if we don't get a Cuda error we are okay with it uh it's now loading and that's the main thing so let's see if it's able to load in the memory ah ah it says some parameters on are on the metor device because they were uploaded to the CPU so sad I think yes but yeah again you can delete and disconnect the runtime ah and you can see EOS probably I'm not doing the uh what they recommend to do if you look at here they say this prompt import dat time blah blah blah let me check this code okay copy it come back here I'm just going to paste it what's wrong calculate uh I mean I I can get rid of this we don't need this here okay and oh let me just get this entire prompt thingy okay I'll get rid of this prompt here this is good UT dat time what next okay if I am let's let me first run this guys okay we're going to run this and try to let me first run this and see what happens but I am more interested to find out the uh capability of to see if it's able to handle complex software development use cases otherwise we have better model like Star Chat star coder wiget coder code Lama and the other llms for coding that we have the coding purposes so if it's not able to understand these kind of questions it's very difficult for you know to go and use it because it gives me the same thing okay uh calculates a person's Age based on the birth year blah blah blah uh import dat time if I am prefix if I am middle okay this is fine now you can try it out and let me know your views on I don't want to extend this video any further okay you can try it out and let's uh let me know your views on uh code JMA model what do you think of the performance on use cases I liked it the 2B model for generating code the smaller code snipp it probably those are helpful if you are building a a Plugin or extension kind of a thing you know for any code editor or the Chrome extension or whatever or to generate simple code bases you know for a given use cases you can try it out now this will be available on GI projectory if you want to use this notebook I'll I'll paste that link in the description and that's all for this video guys if you like the video please hit the like icon and if you haven't subscribed the channel yet please do subscribe the channel that basically motivates me to create more such videos in near future uh share the video and Channel with your friends and to peer thank you so much for watching see you in the next one