StableLM: An Opensource large language model by Stability AI team

Key Takeaways

hello everyone hope you all are doing very well I'm still traveling and I am currently at Japan and I'm meeting with great organizations and I'm really working with some great organizations at the their place and we are working on some very cool generative AI projects unfortunately I can really talk a lot about it but I'm telling you that this field is booming and as an engineer it is great and golden age for all of us to start taking the advantage of this model the reason I created this video is that because very recently the stable uh stable AI team behind the very popular and very successful stable diffusion model for images they have actually released a new large language model named stable LM that fully open source model only few first two models 3 billion as well as the 7 in parameters is publicly available so I decided to create this video give you some of the crude details or the Deep details which are available at immediate for you to take advantage of it dig more look the code more and if you have opportunity try it and learn from it so let's get ourselves started stability dot AI is not something very new in the field of generative AI research their stable diffusion model has already been very popular and one of the top image generation model used by very large Community all over the world so very recently just a few days back is stable a stability AI team behind the stable diffusion generative AI model has released a new model which is a large language model named stable LM by as shown its name it's a large language model which is created by the team behind the stable diffusion model and the company or the organization and under the name of stability dot AI so this large language model is released based on Creative Commons license and if you look into the plan the large language model which is created by stability AI team is just two models are released with 3 billion and the 7 billion parameters but the 15 30 65 and 170 billion parameters models are already on its way to release incoming days or weeks or months so what is available right now so right now the 3 billion and 7 billion parameters models are available so we can take a look those uh models at the hugging phase repo so here is the hugging phase repo if you would want to test the 7 billion model parameter so we are opening the hugging phase and that's where you could actually uh ask or you can start chatting with the 7 billion parameter which is the tuned one so what is coin on today let's see what happens and you can get a very quick idea about the interaction or the conversational model which is the Sable large language model 7 billion parameters and like you know is is pretty much what the standards you are going to get in terms of these models they are trying to reflect that they are machine they are model they are not human let's come back if you are interested to see the model itself where they are available how the size are we can actually go look into these stable the AI hugging face repo here you are going to see that the this is the a space which we were interacting here so you have to understand the idea behind the hugging phase data sets spaces and model so we need to look into the model so here are the models here you could see that there is a uh the stable language base and tuned base and tune 3B billion and the 7 billion parameters so you can actually look into here these are the tuned in the base models for the three billion you can look into the file version here 10 plus 4 about 15 gig parameter 15 gig model and here you can also look into this one is about 15 gig parameter so the difference between is they tuned as well as the base one if you look into the 7 billion ones here and you open them also you will see here that 7 billion tuned version is about 30 gig model here is the 7 billion the base model which is also around the 30 gig so you can get an idea that these two models are already available in the public domain as described by here so the web demo is available for the 7 billion one but these two models are available the base and Alpha and here is the link to direct so the reason I wanted to show you this way so you can understand that how you can search the larger language or any model which are publicly available at the hugging phase location next we can also look into the little more information about the model itself so first we need to understand that all these large language models which are publicly available mostly are are based on the large language data set which is called the file and the file is actually created by the team behind the eluder so they is about 850 gig open source language modeling data set which is combination of 22 other data set but if you are interested in building your own model by scratch and you have resources that's where most of the organizations so most of the open source folks are really go here second you will see that the stable LM Alpha this model is actually using this model and it has about 100 1.5 trillion tokens we already know about three characters lump sum create a token so this table LM Alpha is actually using the file but is a new Rotator set which has about 1.5 trillion tokens which is the three term size of this pile and as you could see that they are going to train a lot even the models which are released just are the very small one so besides that the theme behind this table AI they actually what they did they use the very popular open source model training processes so one is Stanford so just as a proof of concept they actually train the model then they fine-tune the model with the Stanford alpaca procedure which is the com procedure using the combination of five recent data sets for the conversational agent so if you look into this the file the pile is actually the open source language modeling data set so the training for the training tile was used but to fine tune the model so that it can become or act as a conversational agent this model is fine-tuned combination of five recent data sets Al taka GPT for all and these are all open source and then a share GPT 52k databricks jolly and finally the anthropic h So based on the model was created with the pile data set training and fine tune all these five data sets this is what you are seeing as a result of this overall process very important to understand if you are interested to start training by your own if you have resources this is where people start going and start training the model you can also look into the GPT for all training process there is a lot to see and there is besides all of this the company in my previous video I talked about are the servers where they have full data set as well as the fully trained models as seven models are already available from 111 million all the way to few billion parameters so something uh to take from this next we also need to look into that uh what this one the context lens for these models is 4096 token I think if I go a little more below here it may not be described here okay so what I wanted to check is that the the the tokens for the these both models are 4096 tokens it means the contacts lens it means in order to predict the next word the minimum 4096 tokens required for both 3 billion as well as the 7 billion parameter ah models and there was a place where I was able to get little more information about these models if I go into here let me open this page backside and I was looking into the files his face here is the files here I was interested into so that's the code so if you look into the code you will see that this is the code which is actually used for uh you to start interacting with the models here you can learn that which model has been used so here is the model which is really being used the tokenizer is created based on that so here is the model so model and tokenizer so you can look into the application itself which is actually running here this application is actually running on a 10g instances and looking into the code you can actually build this application by yourself and if you want to build your own application this is pretty much code you really needed in order to build your application basically using the Transformers pipeline available at hugging phase you are importing these methods and after that just loading your desired model and then finally setting up of your token size and then finally passing the prompt and generating the result in in between you need to use the tokenizer to first take your input prompt is split it properly and then finally you generate the tokens and then decode the results one another aspect around this uh stable language model Alpha is that it can actually use a personalized impersonation not impressionation but the best way to say that you can use a particular discussion mode it can use the chit chat mode it can use the formal writing and you can say hey can you write this letter for me can you do uh uh can you take this resume and can you create the job description fit for this particular resume or something related to that creative writing and finally you can also use this model to writing code and comparative to the open AI models you or your results going to be very which has been also explained in the uh bottom here is that there you might not be seeing certain results for big reason because the open AI models are basically used with rlhf the reinforcement learning with human feedback here human feedback costs a lot really that's highly expensive to doing those models that's why most of these models are still using the reinforcement learning but they may not have that much capabilities or the performance or the up to marked results related to the open AI models because these are open source models so that's all I had for you in this very short video I hope as an engineer you were able to get a lot from this video where to dig what else you could see and how you can enhance your knowledge by following this video I do appreciate your time and I'm looking forward to seeing in my next video Until then thank you so much