Llama 3 is here! | First impressions and thoughts

Key Takeaways

hi everyone so today I'm going to do a little bit of a different video here on my YouTube channel so we have really exciting news um so L Tree by meta just dropped and it's this release includes 8 billion and 70 billion pre-train and instruction tune models so those are already made available and you can download them basically and start to use them so I just tweeted this um a couple of minutes ago and I provided some details here in the tread but what I want to do in this video I want to go over some of the details and kind of share some of my thoughts on where things are headed it's we've been waiting for lat tree in the space for a long time you know working with language moldes for some time now and working with some of these people that are involved in this Lama Tree Project as well has really built up a lot of excitement for me and especially because I do a lot of work around large languish models I do have a lot to share so if you are interested in those uh this will be the video for it so just to cover kind of an overview of this model here so the official release is here they made it available here at lm.com Lama tree I'll share all the links and they'll be made available in the description and basically the news here is that it's a 8 billion model so and a 70 billion model right so so L Tre involves two of these models and you can see here from the results and I will show you here and that's usually one of the things that we look at when we're doing model selection as developers so that's one of the more important pieces of information to tell you more or less uh what are the capabilities and and what this model can do so the good news here is that Lama tree um the instru model because there is an instru model as well as the instruction tune model and we will talk about how did this instruction tuning in a bit you can see here that this meta tree metal Lama tree 8 billion model outperforms the GMA 7 billion so this one is from Google uh this one is the open model from Google and obviously this is Mel the 7B instruct model as well so these are all both instruct models you can see how it Compares um not sure about the 8 billion I'm not sure why this wasn't the 7 billion and I'm very curious about that to hear more from the as well but that's really interesting that this is an 8 billion model and you can see how it obviously outperform both of these models um I'm very interested in the human eval especially because human eval has a lot of like code generation and so on and obviously MML use for reasoning and so forth it measures reasoning capabilities and also you have like matte reasoning capabilities which is good you can see for these two uh different um like benchmarks right so and then obviously is question question answering uh but you can see how it op performs both of these models that's really great to see this is this looks like a very strong model it's probably the strongest open model yet and I may be missing another model but you can see that this is already showing very strong results one thing that'll be interesting here is to compare with GPD 3.5 that's something that I might end up doing in a follow-up video so how does it compare with some of the apis as well that's obviously of interest because when you're doing model selection you want the best model right and sometimes for experimentation you want to try out these models and they're usually made available a via API so you know how how might it compare with something like GPD 2.5 turbo it's a good question to ask as a developer here um so this one is a 70 billion parameter model right so we have an 8B and we have a 70b now the community has asked also for the 30 billion there's no 30 billion here unfortunately but that's something that's really interesting to see that there's only a 8 billion and a 70 billion but you can see that 70 billion model definitely does a lot better on these benchmarks you can see all the Improvement here and interesting enough they compared it with the latest Cloud tree sunet model and also the Gemini Pro 1.5 model and you can see how it all performs now I can see that there is a this meta tree metal Lama tree 70b uh Falls a little bit behind in the mat reasoning I'm not sure why why that is the case I would expect some kind of improvement uh maybe it has to do with how they evaluate it and so forth but uh this is really interesting and also like Gemini one pro 1.5 uh there is a little bit of a a text here that says manura prompt so the manura prompt which is manura is is just another mold that's used for like math reasoning and so forth um and you can see how this one obviously performs really well and this only tells you how competitive these mods are becoming you can you can tell that these mods right this the comparisons or the performance is is quite similar um so that's interesting to keep track of so that's a little bit about the instruct model okay so this is more about like responsibility I know meta has done a lot of responsibility work if you're interested in that there's some interesting um details in the blog post so there's also a blog post I'll put this link in the description as well and you can read about their approach you know how to kind of integrate more responsible use of these models and how to ensure that these models are safe in their outputs and so on there's a lot of tooling that they're doing for instance the Lama garu is really interesting you can take a look at that if you're doing some evaluation around safety or working with these models and you really care about safety which you should but you you can find more information about that there um okay so that's it for the kind of announcement here very high level and what we can do now is we can jump into the blog post which has more details um these are all the takeaways uh pretty much what I posted on Twitter here um I can read here on my Twitter um some of the takeaways and just kind of sort of summarize it for you now something I didn't cover here when I was covering the instruct model one interesting aspect of these models is you know how strong are your pre-train models as well because you know some of these instruct model we might not want to use them you know they might be too tuned to the point where we can not use them for our use cases so we might be interested to use like the base models right the pre-train models um and that's obviously what I said here great news for AI developers we can see that this let me let me go back to the blog post here and I'll show you here the results you can see here that these models which are also released um are the top performing models broadly speaking here on all the different benchmarks that were tested right so there's also an AGI of all here uh big bench The Arc challenge drop all of these common benchmarks and you will see how strong this model actually is this pre train model and that's very good because that means you you have you know these very capable open model that you can tune for your use cases right um You don't really have to follow the instruction tune model so that's really amazing to see and it's incredible that you can get this amount of performance today with these pre-train models uh so I'm very curious of going into the details about what actually enabled these models to reach to this point now something that's interesting and I've seen it in the community is sort of what this model entails and so on so here are some technical details or at least a summary of it um obviously a standard decoder only Transformer was used there's uh this is a typo here so the vocabulary is 128 tokens right so more tokens I think it was like 32k Lama 2 something like that and now they have increased it um it's trained on sequences of 8K tokens I know a lot of people will complain about it however I know the Lama tree team is working on extending cont uh the context window L because that unlocks different applications with these models right so it's very important that we have um longer context window uh you know we want to use these models for like rag systems or doing large scale analysis on bigger documents and so on so we really need that kind of uh support as well um I'm pretty sure other folks might want to also extend some of these models and that always happens with these releases so we'll be curious to see what what happens there and I will probably be tweeting about that if if I see anything interesting from the community they also applied group cor attention this is not surprising I think most of the like state of the models the open models uh that share details obviously are using something like this which is you know like a really improved version of cor attention this is are for efficiency and also for effectiveness of of these systems to model sequences um there's also pre-training on over 15 trillion tokens not so surprising here I don't I don't think this is surprising pricing but it's it's a lot of tokens obviousy and a lot of data that was used but I can see that in terms of the data they used mostly what publicly available sources right obviously this is an open model and we would love more details about what exactly that data entails but most of these releases don't really have that right so that's something to note um I don't really have the expectation that they would share a lot of that but at least they're claiming that it's um publicly available sort really interesting here something I've been noticing in terms of like trainings of these large language models is the combination of ideas right so this one actually for postering right includes a combination of the supervised fine tuning rejection sampling po this is like the typical things that you see with these models today um and and then DPO as well which has been quite fascinating to see some research papers result uh report really successful uh results with these techniques and so I think what they do is although not much is shared about how they do it um they do share that you know they have combined basically to do the instruction tune models or instruction fine-tuning that they have combined a combination of these things right so that's that's remarkable to see and I think we we will see this continues to be a trend where you are combining all these ideas and techniques um it's not that the field has saturated I think it's just you know combining things to see what we what more efficiency or what more optimization we can get from these models you can see here here from this Tex some of our biggest improvements in model quality came from carefully creating this data and Performing multiple rounds of quality assurance on annotations provided by human annotators so obviously human annotators still very key to get performance from these models and also the idea of having highly created data which is something that you know we don't we cannot IGN there is a lot of focus on not only collecting big or large data sets right but also ensuring that they are higher they high quality and don't contain the information that you don't want there's a lot of like filtering techniques data cation techniques that I see from all the research that I've been following recently so that's great there's not a lot of details obviously and I know that there's like a technical report that's coming soon so when that happens I might do a followup to go through all the like the technical details but here it's more like high level and just to give you some ideas on on and and thoughts basically on what has been announced and what we know from the release okay um there's a bunch of other things uh you know what what it took to build Lama tree um other like partnership stuff and so on you can read that in the blog post on your own and there's also something about deploying llama tree some llama recipes something to also check out as well um right you can see here they also mention group uh group group query attention uh which has been added to lat tree so this is about production right making these molds uh available and make sure that we can put them in production for different applications so those efficiencies or optimizations are really key um okay you can see it here GP gqa contribute to maintaining the inference efficiency on power with LMA 27b exactly right as we scale these models we want to make sure that you can also right make them effici at inference time it doesn't make sense we release a bigger model or you know a model of some some size like this and it's unusable because the difference is pretty bad although there are some other companies that are focusing mostly on INF efficiency and that's something I'm tracking as well and I share a lot about that on on my Twitter um okay so so what's next this is also very exciting I think this is this release um I I don't think it's premature I would hate to say that it's premature um I love what meta does um I know the community has been pressuring there were some rumors that this was going to come out in June or July um but I think the results are great it's great that the community has access to these models as well to start experimenting and kind of comparing with with all the status quo models and so on uh but I'm excited to see that there is like more efforts around latry and more future releases that are going to come as well so there is a 400 billion parameter model on in the works you can see that it's still training um and you can see that the checkpoint as of April 15 right 3 days ago you can see the performance and this is insane performance already for a 400 billion plus parameter model you can see how ridiculous these these numbers are I mean if you have tracked performance on mlu and human eval on these uh different benchmarks they're kind of really hard benchmarks but it's really interesting to see this right there's also the question of like data contamination and and so for it as well but hopefully the team is actually focusing on that and ensuring that there's none of that during these kind of experiments but this is amazing I mean I'm excited about these results for a 400 billion parameter model the expectation is that as you scale right you you're going to get better results those are kind of their sort of like scaling laws um and so there's a lot of interest in that and and I'm really excited for like the new developments there's also something here about supporting multi modality that's something that I actually expected latry to have natively support right so you can see that this gp4 has a vision and other models they also have Vision capabilities and also there's interested to have like multilingual capabilities that's very interesting too to support use cases in different languages there a lot of complaint about longer context I know a lot of people probably will be a bit disappointed that it's 8K but I'm I'm pretty sure that they're working on this and we will will have longer context window to be able to use these models for like knowledge intensive use cases or trying to develop like these rack systems um and even these agentic workflows right so this is what everyone is building and everyone is doing today so including myself and I and I want to see that there's more support for longer context window um okay and yeah we will see that there is a research paper that will be released eventually um I know the research papers on Lama the Lama team and meta overall they usually include a lot more details so I'm anticipating that there will be more details about training data more details about like um you know some some of the architectural uh decisions and so on so there we will know more about some of the um reasons why these models are performing really well on some of these benchmarks all right so that's uh roughly speaking the the results there is also something here about um usually when you do evaluation right there's also human evaluation and this has to do with like you know real world capabilities I'm huge advocate for this when I talk about models and when there's always a release something I always call out is like okay I don't really just want to see mlu performance or human performance um because cheating can be happening there as well I'm not saying they are cheating but I just say it can happen and sometimes it happens unintentionally um I love to see more from the community where they test things like creative writing instruction things that you we really use these mods for in the real world and you can see that they also did some a little bit of effort here on doing some human evaluation and how they compare with Cloud onit michell medium and gbd 2.5 as well so you can see the results here and it seems that a tree instru model 70 billion uh it's seems to be U more favorable or at least more preferred by humans from from the win percentage here so that's awesome to see um again all of these things right something that evaluation is something you do on your own like when you're building use cases um so you know take this with a gr of salt you you really need to do your own evaluation but it's great that they did it and they can show here how um how how these models are performing and how they compare so that's something to look into as well okay so one last thing I want to call out actually too is the model card so before I wrap up here there is a model card you can go through all the details are here um the license as well there's a community license I believe um you know latri is intended for commercial and research use in English you can see it here um know when you use these models I always take a look at the license I don't know if this is an improved version of the previous license I still need to go into the details of the actual license but you can read it yourself here and that's something that the community also expected some changes on so take a look at that and decide for yourself whether this makes sense to use for your use cases um okay it's a little bit more information about hardware and software I might do a little bit more um overview on this once the technical uh report comes out I guess there will have more details on that there um CO2 emissions um training data I guess there's more going to be more on that in the technical report I assume but you can see that the pre-training data cut off was March 2023 right so that's really important for the 7 billion model and for the 70 billion it was December 2023 so this is more about benchmarks again that's already we went through that um there's something on safety here and and and more details I was look at the um and you can see all the contributors here uh really an amazing set of people working hard to release these models right so it's a huge list here um so thank you throughout the team and and meta for really pushing themselves and know pushing this model out there for our community to kind of Leverage and and and start a use so thank you to the team there you know there was some conversation um on Twitter about whether we're going to have msture of experts you know msture of experts is what all these other companies are using um on their latest models so we have like I believe Gemini and we have like the um what was it called the other model uh Mel models as well mixell models that they use mixture of experts and a couple of other companies are using mixture of experts and leveraging that right um and I like really this quote from one of the contributors of this of this work Lama tree so if you look you can get more details uh from Asen Jang here and you can follow and you can read over some of the details some of these are already kind of summarized uh but here is a little bit of a clue on you know why these models are so performant and again it has to do with better skinning laws and infrastructure um and and so forth right talking about homage gpus and so forth but I guess we will know a lot more about these models when the technical report does come out and we will have a better idea an understanding of where these mods um what's the secret sauce what where are these mods performing so well and so on so I hope I hope we have more details very soon that we can share so if you want to use this model you can see from the announcement here right you can experience Lama Tree on meta so it seems that meta AI is meta's intelligent assistant it looks a bit like chat GP right and you can have conversations I guess you can try it um I believe from my understanding that this one is supported by Lama tree so if you want to get a feel for whole Lama tree I believe is the instruct model how it performs on on different tasks um you you know you can go to meta a and try it out so there's a couple of like templates here that you can use so let's say let's try some of these help me with an assignment so here you have help me with an assignment start by asking me two questions about the assignment okay you can see it's obviously this is conversational agent right it helps you to like solve task and generate whatever you want to generate um so I will be testing this out more I might do followup just leave a comment if you want me to go through this I mean I do experiment a lot with these moldes but if I start to do experiments here it might take a long time and they will extend this video I just wanted to focus on the overview of the announcement and give you my thoughts but I think this could be a good follow-up video if you want me to test this and um we can have like even a live stream if that's necessary but I can test some of the capabilities right like reasoning and so forth so let me know um that's going to be it um do try it out um leave some comments if you want in in on this video uh about what more you want to see I'm going to do more of these type of videos um you know in the coming weeks as well so I am posting more regularly on YouTube as you can notice um and that's something that I kind of uh challenge myself this year to do and to keep up to date with all these kind of changes everything is going so fast but I think once we do this in community and kind of communicate and share ideas and so forth I think you know this is a good way to kind of keep up so that'll be for today thank you so much um if you enjoyed the video please leave a like there and also it helps if you subscribe as well um that tells me if you're really interested in this content and it tells me if you want to see more of this in the future so have a good one bye