Discover Salesforce's Game-Changing Code T5 Plus Model

Key Takeaways

ladies and gentlemen brace yourself you're about to dive head first into the swirling vertex of the Transformer architecture and code language models and their full multimodal Glory Days ain't your grandma's knitting tutorial folks this is the digital equivalent of juggling flaming chainsaws while riding a unicycle on a tightrope it is like trying to solve a Rubik's Cube in the middle of a roller coaster ride it's like well you get the idea so if you're an absolute beginner you might feel like you've accidentally wandered into a quantum physics convention but don't worry hey don't worry we'll try to keep the mood light and even as your brain does somersaults trying to keep up never mind buckle up grab your popcorn and let's dive into the deep end of the coding pool and then I would like to show you an absolute outlier and this absolute outlier is here by a company I didn't expect it it is Salesforce Salesforce research area started in September 2021 with code T5 very interesting model but you know what they came up now in May 20 2023 with code T5 Plus an open code large language model for code understanding and generation so this is now the advanced model May 20 brand new for me when I'm recording this and I would say have a look at the training data and the architecture of their Transformer architecture so here we go what is the data now in the pre-training data set they enlarged here this specific set it used with a recently released GitHub code data set this is now available hugging phase this is fantastic they select nine programming language python Java rule with JavaScript go PHP CC plus and filter the data set only by permissively allowed licenses and files with 50 to 2000 token they cleared the data set yes yes yes and and this is interesting they also have instruction tuning and they apply over 20 000 instruction tuning data in a specific code and they use here this self-instruct mechanism I have a video on they use here another llm the generates here to the the instruction fine-tuning data for them this is open AIS text DaVinci 03 one of the best models currently for this task and I include here the task instruction synthetically the inputs and of course the expected output so the input output pairs plus the instruction is their synthetic in self-instruction fine-tuning data set and they call this instruct code T5 plus model so whenever you read instruct code P5 plus you know this is here from openai synthetic data set generation to show you how this looks very familiar you notice if it's SQL pipe never mind you having instruction let's take this here on the right side optimize the given Python program to improve the speed of execution and this is the training data set so you have an input that is not optimized and then you have an output of a code sequence that is optimized for Speed of execution and you have hundreds and thousands of examples so the system learns how to optimize code that is given and right present to you the code sequence of hundreds or thousands of programmers that optimize the specific code sequence and SQL queries yes you know this it doesn't depend anymore where your data is everything becomes interconnected in a data mesh great now the task that they pre-trained code T5 Plus on is interesting because they have now a lot of tasks and I would like to show you the beauty of this system what makes it so different to the other system so the train code T5 plus a mixture of pre-training tasks they have this causal language modeling I just showed you before the contrast if learning the classical way and they have a text code matching this is the very first beginning of the video where I showed you the comparison gpd4 to store chat that is based on the star code base llm so this text code matching is interesting for them and they do this modal code and a bimodal code X data set great now I will show you in a second that this is a model that has an inherent beauty and a powerful structure no other model has no other GPT model can have because the way they built their llm if they take now the complete architectural side of a transformer you have an encoder and a decoder model so the code T5 plus can support a code generation and the code summarization you remember on the left side you have to encode a stagger on the right stat you have the decoder stack and you can easily use it as a decoder only model this is exactly what the GPT models do for auto regressive generation tasks such as code completion in code completion this is the strength of the GPT models because there is absolutely the focused task of Auto regressive generation of code and finally now if you have both and here only the decoder you have here the option of only an encoder model which they of course use here for the neural search ability or if you look here at Asia for Microsoft at the retrieval task as they call it so you can run this code T5 Plus in three different model configuration and this is not even it there's even a better way because you can use here retrieval augmented code generation I have a specific video on this you know this but just to show you this how it works and what is the benefit of it because of the way we build our Transformer about the architecture that is now different to a pure GPT system let's have an example so our query to the system is download an HTTP resource from URL and save to a destination capable of dealing with gzip compressed content so write this code for this particular query of mine normally if you have an ultra aggressive GPT system it generates an answer and it says hey yeah download and hear a compressed they look now for a python function definition that has for example in its name download and compress and they said hey this is it this could be that just the name was chosen by some coder who has no idea that his code will be part here and this is not the right answer because if you want the solution space this was searching and it was so huge that they came up with the wrong solution now if you have now an encoder stack in your Transformer you can have now that you first activate here the encoder stack and retrieve here relevant information you remember in the human language case it was this cosine similarity calculation that we did with Vector representation and now we can make a same methodology we apply here we reduce the available space of all the different possible solution applying at first the birth structure we say hey look I found this here with a high cosine similarity and then even my query and here additional data I have here now retrieved here this and then retrieve then generate the solution the coding sequence I'm looking for and you see now here in the definition of the Python function is some code chosen here as a name download file so only from the definition you would not be able to guess that this is the right one but as they write here they were looking for him the URL Libs 3 command and if you add here to this the URL lib3 command they found I there is exactly what we are looking for so this methodology that is based on the architecture of the Transformer that they use here in code T5 plus with this step they get a better performance here on smaller models and smaller models in the open source Community are really important ladies and gentlemen welcome here to our coding session you're not gonna believe it we gonna have an hour interference run on a free Google collab notebook at first we will install pytorch then we will install the latest version of the Transformers we will go for a data set and accelerate we will install up to bits and bytes and wnb and this is it and then the only program we're gonna execute is this little cutie here Transformer we have a checkpoint from Salesforce the code t5p model unfortunately the maximum size we do not we cannot use the 16 billion parameter not a 6 billion parameter not a 2 billion parameter model we have to go on a free call up notebook for this 0.77 billion parameter model and we use a good Accord and I think we start now so here we go torch Transformer we don't need theft UDF little is required okay there seems to be some problem disconnect delete everything please delete everything now this is horror clear output runtime yeah let's go one two I need accelerate I need bits and bytes I do not need any login does this Define hearing last time now we go for the Salesforce code T5 plus yes ladies and gentlemen so we have here the baby version and you will see it will be able to run this on a free code notebook it is exactly as you know it here we have a tokenizer auto tokenizer from pre-trained we have a checkpoint the checkpoint is from Salesforce entity model we have a auto model for sequence 2 secrets language model it is a T5 model you know it you'll love it absolutely clear from pre-trained we go with float 16 because we have this old Nvidia GPU brain fluid is not possible forget about 8-bit this version is this model is not available in an 8-bit version yet I hope maybe when you see this video please try it out and this is it this is it uh then we have a look at the footprint of the model and then easy we have a tokenizer we have a python function a python function is here what I have prepared here four examples for us we go with pi torch to device we got a GPU and then we just go with the encoding of our input IDs we have here the model generate command I have the output set to a maximum length of 750 characters and we decode the output skip our special token and this is it ladies and gentlemen so let's go for the first thing and the first thing is very easy a python function that I defined here and I say here in a text to code remember we have the bird and the GPT so here we have write the python code for linear regression and here we go here we jump yeah the other thing is it finished yeah everything's finished torch audio is finished hugging face Transformer is finished accelerate and data set is finished bits and bytes and W and B is finished yeah everything is fine so here we are now so we are downloading here now my tokens are token map are configuration Json and then we have here with 1.48 gigabyte our pytwatch model this little baby model our code key five piece 0.77 B okay we have to take what we get for free so ban him this is the fighter watch model and now as you can see we have a text to code ability we have a code completion possibility I will show you next and then we can even go for a little bit more advanced thing of course we're running now with a model less than one billion parameters I'm really interested what result we will get back here from the free call up notebook but I wanted you were able to see this you can do this on an iPad wherever collab is great for experience to get here the first learning experience and I'm still waiting for my first learning experience and here we are so you see this is not the output right python code for linear regression so input numpy oh matte plot lip so we have here upload then we have here our python function linear regression initialize the parameter initialize the weight intercept it's a little bit complicated I would say prediction update the weights what linear regression with okay this is a little bit strange generate some random data fit the model with linear regression uh okay yeah I mean yeah circuit would be available but okay and then we plot the data but you know what if we do it now the second time you know it's not always that the result that you are given is identical so the model is now loaded so it should be really fast to see this it is real time and then I can prepare already the next one because next one I show you now here what here the classical models exceed here if we give it here now a code sequence itself here for a python function we say here the name of the Python function should be fit a linear regression model this will then be our next exercise for the code completion and now I have told you everything I know about these four lines of code so now I hope we got a result in here we are and initial parameter get some data okay yes maybe another possibility let's go here to the pure python code completion exercise this is where all the models shine they do not have a bird model attached or have not been trained on the chatty functions so this is here more or less hardcore code llm you have to give them here a specific uh python function that you want and you hope you cross your fingers that the name that you're asking for somebody has used yes here it is unbelievable here we go it's color and beautiful so here we see you have linear fit linear regression X and Y we have a nump array of shape Y is a number area of this shape from a scalar linear model import linear regression yes exactly if it regressor return regress or easy as it can be and the next one here is an example I would like to show you this is a little bit more advanced write a python code for an expert Vector embedding of a sentence and then I think you got it you see this is all that you need yeah you can forget about the memory footprint because you see it you have here those four lines forget it it is really just you define here the Marlin and we use you define if you go CPU GPU you have your tokenizer your model you have your encoding and the output it can't be easier than that so beautiful yeah if you have here quantization issues there is a hugging phase here main classes quantization Advanced use cases for quantization if you want to have a deep dive into quantization topic but we stay here in the version that the model is available for us so let's see what he knows about expert Vector embedding of a sentence and here we go hey not so bad torch torch functional F yeah expert oh wow it defines now a class expert Heaven's Sake okay we have our embeddings we have to wait lstm wow this is an old stuff a linear debate initialize the weight initial the feed forward Loop wow okay I mean for for a model for a baby model this size 0.7 I mean it it hardly can see below I don't know above something so yeah okay I we have to be satisfied but yeah I would highly recommend that the you have a PC infrastructure with an Nvidia GPU that is able to handle this because as I told you just to show you yeah just show you if I just change here this and I put here my beloved 2 billion parameter modeling you will see yeah at first the model is of course bigger we have 6.45 gigabyte to download here a pytorch model but it takes about two minutes okay we'll be back in two minutes you will see we will crash with gporam so you can see it took us two minutes for 6.5 gigabytes so Dion's the aunt Colony carrot single-handedly carry here every single bite over to me Jesus okay and now we are waiting so our prompt is a right python code for an expert Vector embedding of a sentence and I wanted to show you that maybe we are crashing oh we are crashing on system Ram interesting so let's see where we are crashing for what reason and as you can see we are the 11 gigabyte system RAM available at 12. so I expect we will crash now any second with a 2 billion parameter model in a free Google collab notebook as you see 12.0 12.7 my goodness this is your session crashed restart after crash after using up all available Ram yeah come on pay for collab Pro no I'm not gonna pay but I'm sure that you understand how to code it have fun with this experience this on your machine so here you have them more or less if you want to compare tier the code llms now here we finally finished here our selection with the code T5 plus a completely different architecture that was also instruction tuned the main model is the 16 billion parameter models are really comparable here to our star coder 15.5 from Salesforce via the open Community from hugging face but the nice thing is they give you here if you want to try it out a 6 billion parameter Model A 2 billion parameter model and there's even a model below 1 billion free trainable parameter from code T5 Plus now to be a realistic everything that isn't technical preview we have to exclude and we have seen the results from the encoder from the 6 billion parameter model from Facebook does not match at all with star coder for whatever reasons so if you are interested and this was the request by the community about star Coda but I would like to show you that there are other models that can be very very important that you know about them and they can even have a better Performance Based on the way they were constructed and Salesforce provide here also an open license and when I looked at all of these models side by side and I wanted to show you another comparison I noticed some sometimes those models create just blah blah blah nonsense you've shown this in the introduction so what we do the first step is now to understand why it happens this the model do not perform and this is easy because constantly we update our code data understanding stack Overflow is typical where I learned a hundred years ago my python coding and you see just it was money ask a question 21 days ago modified 14 days ago and just some days ago they came up with a new code sequence that is much better and we should use this now now this is great but your model has been trained for a particular day I don't know may 2023 and this is it and maybe the next official update of the model will be in some months or maybe in 2024 when there is a complete new run to fine tune a complete model with the new code updates it is very expensive here to run near the complete model on the new data so the question is what can we do to update here at the code knowledge the code book if you want that is available on the internet yeah if you look at it you know with the human language the NLP task it was easy because the work from Shakespeare I'm not changed at all this is a static block of information or if you go for some French literature or English literature whatever you want now compare this to the dynamic development here in Python so you see not only is it not as clear as a human language but the speed that the python code base changes is amazing and this brings new problems to an update now here we go second step now that we understand that the code elements work differently they also learn differently let's have a look at this in detail so we have to dive a little bit deeper I already told you we have here a Transformer architecture we have here the encoder stack that looks at the complete sequence and calculates the relation between all the different tokens and this is our bird system and for the GPT system we have to Pure decoder stack the decoder Stacks looks left and predicts the next token beautiful you know GPT has also the reinforcement learning from Human feedback the templates and this is now an interesting path forward for us because these templates here could be something that we could utilize remember that I told you the GPT utilizes a left to right causal autoregressive language modeling objectives this is beautiful and they now try to integrate a feature of bird and this is here in bird the mosque language modeling objective if you think back two years with word and expert you had this most language modeling objective that was at first a little bit difficult to understand but then will come really familiar with it so because mask language model can condition on both the left and the right context to infill here the mask region and this is exactly what they apply now from the big GPT models their models that they have trained for millions of dollars to have now exactly this functionality of the bird system so let's play with this let's play now at the GPT part of the decode stack how can we create these templates for ourselves or how can you do it if you're a company now creating multiple code templates is easy cheap ID you can do it with humans or you can automize it you can publish like 5000 micro jobs on the internet and every coder under micro under macro work platform that has access to the internet should produce one instruction of the Python code as a training examples two examples per day for a week and you see easy in a week you get 70 000 code instruction that form now your new training data set that are based on templates so whenever you have a task like save a panda data frame in a CSV file format or open a Json file in Python those 5000 macro jobs create in a week already seventy thousand instruction based python code for example if you focus on python training data sets so the combination the combinatorial set now of those seventy thousand how you add together how you stack different commands together for different tasks is amazing it creates our modular stack and this feeds back here in the template structure so we have here a perfect update you can do this I don't know every month every two months if there whenever there's a new version coming out from I don't know python itself or whatever so this is one solution beautiful or interestingly enough if you go with the code T5 plus you can vary now the size of the encoder on the decoder stack particular for your task and this is another beauty of the code P5 plus model so how can we learn here our llms that are not performing so beautifully that are smaller this is not a gpt4 system but this system's about 15 16 billion free trainable parameters but we have we have bug fixes we have python updates we have new functionalities are added to all our libraries some code is now out of date we have new libraries coming in we have updates of libraries daily basis weekly basis they are now better ways to code some some gorgeous coder published here very intelligent way of coding new sequences you have new models you have unbuilt the dynamic is breathtaking so good news we have solutions for this yeah yep and we have two solutions for you so let's have a look at first those small 16 billion parameter malls you can fine tune them and I showed you at the beginning how to fine-tune system and the good thing is it's a Transformer system we know how to fine-tune Transformer if you have a high quality data set and it would say this is the most crucial part you can instruct fine tune it you can fine tune it in a supervised way this is the way to go I will find you in my 16 billion parameter models for my specific code preferences and you remember that I told you in one of my last video this fine-tuning way the way we build instruction fine tuning templates we can use now also and this huge monolithic gpd4 models or this copilot X models that are based on gbt4 in the prompt design in the prompt Engineering in the templates we will use because this this augmented additional information we provide in the prompt for the system based on in context learning and we have the instruction prompt tablets that focus more or less exactly on the same fine-tuning technique so this is great you might say hey all this talking you had about Jacks and flags and is there anything to it well if you are interested this fine tuning is of course faster with functional programming and you can get some you can significant speed improvements if you want to enter here functional programming but it comes here together in a beautiful way and if you look at the current code examples they all go with the functional programming approach and if you go here with the prompt base or the open API phase with a monster gpd4 models you have to ask yourself hey when is our prompted we input strong enough to override the inherent knowledge of the system because the GPT system has been free trained and fine-tuned on millions and billions of code sequences so when if I have now a code update I have to override the inherent system memory of gpd4 when will it happen and we have an answer for this in one of my last videos yes exactly this I showed you when it happens just let me go here 4K gee remember I was here with the Learned content a inherent to gbd4 and I said hey when does The Prompt content override it if the system content is our sun is a star and I want to contact our sun is a stone when will this happen and how do I have to design The Prompt but now my I see alcant prompt content will override it when is my prompt strong enough to overwrite the system and the solution you'll find here in this video I don't want to go into detail no not at all you know it but we have no two solutions either you go with the fine tune your code llm model with fine-tuning or instruction fine tuning once every week every month whatever you want on those beautiful open source 16 billion free trainable parameter models this can be done nicely this is within the budget that we can afford in our Cloud compute infrastructure or if you want to use the power the real power the coding power of gbd4 models and you want somehow to overwrite their knowledge and you are not a system administrator in Microsoft Azure or the responsible for for running the gpd4 models on Microsoft supercomputer Center then as a simple user you can in every session that you start with gbd4 you can have a prompt design to learn the lmm specific code sequences you want to use you want that the system responds with when you want help for your coding exercise so this updated code sequences we now know how to insert with a prompt in our huge monolithic gbd4 models so we have two solutions fine tune it the smaller ones and the same technique is working here with our prompt injection isn't that beautiful we have two solution to optimize our models and I'm looking forward what would happen in the next weeks in the next month to come because this is a area of research that is evolving with lightning speed I hope it was informative I hope you have some new information for your particular task you want to code and maybe I see you in my next video foreign