Red Pajama - Operation: Freeing LLaMA

Key Takeaways

alright so this video is going to be a quick one I just wanted to talk about this project called red pajam so this is from the group called together computer I think is also their name and this is basically the start of a whole project to reproduce a fully open source version of the Llama models and they've kicked it off by first releasing the data set so it's pretty impressive their plan is to basically create a a set of open models of the Llama models and to do that they actually have to train the foundation models on over a trillion tokens so here's the data set based on what the original llama actually used so this is over a trillion tokens they're saying it's 1.2 trillion tokens if we remember back llama the 7 billion and the 13 billion parameter models were trained on one trillion tokens and the two bigger models going up to the 65 billion parameter model was trained on 1.4 trillion tokens so while it might seem perhaps not a big deal that oh they've released this data set because it's just scraped from the internet it is definitely a big deal in regards to the pre-processing and all the things that have been done for that so they've managed to put that all together in a way that can actually go through and make a nice cleaned high quality data set on par with what llama was trained on now in theory that should mean that we can get a model out that will be as good as llama and so they point out in here that basically this has been uh sort of a takeoff moment for AI and certainly for large language models that these open source models have come along but unfortunately a lot of the models like llama alpaca vicuna koala are not really fully open there are some that like pythia open chat kit open assistant and Dolly which are fully open but a lot of the others are not so this is a way of them kicking it off and getting started to make fully open Llama model so the group there's quite a number of groups together about this we've got together themselves there's also people from Stanford from eth in Switzerland from Mila in Canada it's definitely a big International effort to make this thing happen and so they talk about the three main components of this being the pre-training data which needs to be both high quality and have broad coverage the that's what they were releasing now the base models which is apparently their training at the moment and then third will be the instruction tuning data sets which we'll probably see a variety of those come out over then so anyway they go on a little bit about the different reasons why and some things about llama in there and then they break down the actual data set so the data set is made of five dumps of common crawl which is basically looking just scraping the internet of pages and then they've got a number of different filters that they're using to clean that they've got the C4 standard C4 data set which came out of the T5 model back in 2019 they've got GitHub they've got archive papers they've got the Books Corpus which I'm pretty sure was used in the original GPT 2 model Wikipedia been used in many models and stack exchange there as well so this is quite impressive the number of tokens that they've got here that are putting all together to create something that's sort of 1.2 trillion tokens this is definitely in the ballpark of what where llama was so they put this up on hugging face if you wanted to go and train your own llama model now and you had the money and the compute you would certainly be able to do that the data set is on hanging face it's a trillion tokens it will take you probably quite a long time to download it and they've also got in here a smaller version of this which is the sample data set so this one you can actually go through and have a look at it this is only a billion tokens a subset from the main one so anyway just the main thing to sort of keep you informed here is that we've got a full open source llama model that sounds like it's well on the way to coming out which will mean that a lot of things that people were doing with fukuna with koala with a lot of these models there's probably going to be versions of these that are going to be fully open source in the not too distant future anyway on that note as always if you've got questions please put them in the comments if you found this useful please click like And subscribe I will see you in the next video