2024 Predictions - Models, Companies, Techniques

Key Takeaways

okay it's 2024 happy New Year everybody I thought I'd make a quick video just talking about some of the predictions Trends some of the things I think that are going to be interesting this year around Ai and machine learning and this whole sort of world of large language models and multimod models Etc so I'm just going to go through these pretty quickly I don't want to make this video too long I'm going to break it down into a few different sections to talk about models companies the players that are go on I want to talk a little bit about some of the different kinds of models that we're probably going to see this year and then also some of the challenges that we're facing with these kind of models and with AI in general not from a safety perspective I don't really want to get into the whole safety stuff here but more some of the challenges that if we're going to sort of Advance these models what do we need to think about so first off let's look at the actual models I think 2024 is going to be a fantastic year for models I think we're going to see a variety of different models coming out definitely we're going to see multimodal takeoff a lot more than we did in 2023 I think in 2023 a lot of the models were ready the challenge was serving these models for multimodal we saw that very clearly with the GPT 4V model that even though they announced it quite early on in the year it wasn't till sort of November December when people actually got access to it and this is one of the biggest challenges that all the big players have at the moment is that how can you serve these models at scale so a lot of people tend to think that when these announcements are made by open AI or Google or something like that that they're just Vape aware that the things don't exist I can assure you that the things exist the models exist the challenge is getting them into production where you can actually serve it at scale to a large number of people that want to use use it so one of the biggest secrets in 2023 I think was that a lot of the models that were in production were much smaller than people think and I think this is true for chat G PT as it is for the Palm models and a number of other models out there while the research version of these models may have been hundreds of billions of parameters the actual versions in production were much smaller were sort of in the range of 20 to 50 billion parameters and I think this is one of the key things that we're seeing people put a lot of effort into over the past year is better distillation and serving of inference models so basically how do you get a very big model to a size that you can basically serve it at scale to a large number of people and this has been a challenge I think for all the big players in here in that they understand that once they put something into production and give it to customers companies are going to want to use it and they're going to want to use it at a pretty decent scale where they have to be able to provide this model with reasonably low latency with enough instances of the model to be able to serve it to many customers and this is why all the models when they come out generally are just given to a very few select customers while the companies are then working out how to improve serving it at scale or doing distillation so I think in 2024 we're going to see a lot of advancements in this we're going to see a lot more better distillation of getting large language models distilling them down to smaller models that you can then put into production interestingly in the open source accumul we haven't seen people go after distillation very strongly we've seen a lot of effort around quantization of models and making smaller versions of models via quantization but I think in 2024 we will see a lot more distillation of bigger models down to smaller models which then can be used in production easy whether that's really small and being used at the edge again quantization will be a key factor there or whether it's deploying them in the cloud but wanting to be able to serve many customers at the same time in regards to new big models I think we will probably see GPT 5 this year I think we'll also see Gemini 2 this year coming out I think for the time being a lot of the architectures are probably going to stay the same around these Transformer mixture of experts style models that we're seeing here in many ways it's in the big players interest and when I'm talking about the big players here I guess I'm talking about open AI Google Nvidia perhaps to a lesser extent Microsoft it's in their interest to be able to stay with this kind of architecture because they've optimized a lot of their Hardware around this so there's a great resarch paper called the hardware Lottery which talks about this sort of concept a lot more but basically if you think about it that the big players have got the large amounts of compute which allow them to uh not only train these models but then serve them at some kind of scale for this so I think one of the big things that we'll see is NVIDIA will probably get more into being some kind of cloud provider themselves that's quite possible that may be very limited to just you know providing compute to certain startups or something at the start I think that's what they're doing currently and we're also going to see open AI Microsoft versus Google trying to ramp up their compute for training these really large models so I I think it's very interesting to look at something like Gemini Ultra which was actually trained on TPU version fours across multiple data centers and it's quite amazing to think you know that Google can actually train these model models with the latency being low enough that they can actually put them across multiple data centers as they're going through this since then Google has publicly released TPU V5 uh PS which are about 2 to 2.8 times faster than the TPU 4S for training these models so you're getting to the point now where if Gemini took a few months to train Google's sort of gearing up their hardware systems now to probably would be able to do that in a little over a month which makes me reasonably confident that probably Gemini 2 is already training or already being prepared to be trained and to be announced sometime in the middle of this year I think a similar sort of thing is going on at open AI with GPT 5 so we heard a lot of Fantastical rumors around qar and it seemed like everyone on the internet had an opinion about what qar was it seems to me that what that sort of points at and around that whole sort of incident points at is that perhaps open AI have worked out some new ways of improving the training of improving their use of data while there are a number of scares around that they may have discovered AGI already I think that probably more pointed to the fact that when they were doing sort of test models or sort of mini ablation models they discovered some ways of improving these models at a very low scale and now they're at the point of where just scaling that up for GPT 5 so I think one of the big questions is going to be is open AI going to sit on GPD 5 like they did with GPD 4 for sort of six months before they actually announce it to people let alone actually release it to you know a large number of people so the base model of GPT for finished training in the summer of 2022 but it was only in 2023 that people actually heard about it when they announced it and then for many people it was a couple of months after that before people got access to it and we can see that the compute challenges around that made it very expensive to use if we're going even bigger for things like GPT 5 or you know other large models then you've got to wonder okay at what point does it get to be too expensive per token for people to actually use these for almost anything I think this is where the installation really will come in is that we will see these models used more to distill smaller models than to be actually used a lot themselves for everyday uses so on the open source front I I think things look really great here where I I think in 2024 we'll probably see an open source model that gets to be on par with GPT 4 at least for most things I think while there's a lot of Benchmark hacking going on in the community people are working on different ideas of how to actually make these models better I think there's some really interesting projects perhaps not open source but like uh f 2 is certainly an interesting project and pushes that back onto the open source Community to go back and look at playing with smaller models a lot more and then I think we're also going to see a number of new architectures so the whole sort of S4 hyena Mamba style architecture looks very promising the big challenge at at the moment is that we don't know if how well this thing is going to scale it looks extremely promising and I hope that it you know does work out I'm reluctant to say that this is a Transformer killer just because we've had other things come along and it's turned out that a lot of these things when you actually scaled them up didn't work as well as people had thought they would in the first place another thing that I think we're going to see in 2024 is the whole concept of reasoning models so models that are not really just built for language in language out but built more for doing reasoning tasks at a dense Vector embedding layer and then passing those out to a language model so I think that there may be some interesting things happening in in that space of course with the open source stuff one of the biggest players has been meta with the Llama models I think you know what happened with llama 1 and then later on llama 2 in 2023 really Chang the game I don't know if it was intentional that people leak the Lama one weights but it certainly you know the fact of of the way that meta responded to that and realized that they could actually benefit from the ability to have these weights out there and have people hacking away on these weights certainly changed the game I think the whole no Moes paper that came out of Google was very interesting in regards to this I don't agree with everything everything in that memo but I certainly think there was some really interesting points that meta benefited a lot from having these people work on llama and work on ways to fine-tune llama work on different data sets for fine tuning llama that allowed them to basically fold a lot of those things back into internal projects at meta I think one of the the classic examples of that was the whole llama CPP project and having quantized versions of llama that was something that perhaps the big companies weren't paying as much attention to because people at the big companies tended to think that the quality of quanti models was not going to be good enough and you know I would certainly count myself in that camp that originally I wasn't behind the quanti models as much as I thought that you'd be losing too much quality in the output it turns out that with a lot of the quantization systems now they're getting still very good outputs for this I think it's really really interesting to see models like the E34 billion model be released in multiple versions with as well and then being benchmarked against the original one so that's certainly interesting in there so I I expect that we will see llama 3 very soon you know over the next month or two perhaps even sooner than that and the big question is going to be how good is that llama 3 Model going to be is it going to be on par with gp4 if so then when people go to find tuneit we're going to see some amazing models come out of that whole process there I think another thing with language models is that the whole sort of second tier companies like anthropic mistal inflection some of these companies that have released very interesting models sometimes proprietary models and in some cases like mistel open source models as well as proprietary models these players are going to become really interesting so I I definitely think probably mistel is one of the really interesting ones there their whole the mistel 7B was and is still an awesome model their mixtur model is also extremely impressive you know for what it is and you've got to think that now that they've raised a lot more money they're going to be making bigger mistal models and in fact I think they've already got one on their API and you could imagine that they will have something quite large on the way reasonably soon as as well so I think the whole sort of model space is looking extremely good for this kind of stuff another kind of area that I think we're going to see take off is the whole sort of Music models I'm pretty confident that both Google and open AI have been sitting on models that can make music and full songs with singers with everything like that there's a very interesting interview with bjor from ABBA on the Rick biato Channel where he talks about seeing a demo of one of these models being given a private demo of one of these models and it seems he was approached because he also has a role in one of the music licensing organizations and so I think the the reason why that Google and openai haven't released these models is just they want to sort out the whole sort of copyright things and this before these things actually get released but I think we will see that those come out in and 2024 I think people are sort of working behind the scenes to try and work out some kind of deal where a certain portion will go to the people whose music it was trained on and stuff like that I think that's going to be a really hard deal to do and and certainly not everyone's going to come out of that as being happy I think the same will be true for movies so we're seeing some amazing and cool new models for making mini little movie clips uh from companies like pick laabs you know I think stability has got one there are a number of these things out there and I think this is just a start I think in 2024 we're going to see this takeoff even more and I think it's going to you know just like we saw the diffusion image generator models take off at the end of 2022 and then 2023 they advance so much I think we'll see the same thing happening with the video models for this so Google has already shown off a number of the video models that they have picker Labs has actually got products out there that people are using and I do think this is going to be a really interesting space because I think over 2024 probably the second half and then even the next couple of years after that I think we're going to see people be able to make first off being movie shorts but then probably whole movies of where they're able to take these things and turn them into some kind of movie and I think that's going to really open up things just like you YouTube has killed a lot of Television for a lot of people because people can Niche down into exactly what they want to see I think we're going to see the same thing happen to places like Netflix as people come out with just really cool content that they've made so you know I expect to see some teenager make a full movie in 2024 with these tools and while it won't be perfect I think it's going to be interesting enough that people are going to watch this going to see the whole idea of creative people being able to use these tools to make interesting art and interesting products that people are going to want to check out so I think that space is really awesome to check out and look at another one that I haven't talked about which I think is really on point as well is the whole sort of Agents thing so I definitely think we will see advances in agents this year the biggest issue is going to be getting agents just to be more reliable and then slowly things like speed and other things like that but certainly getting agents to be more reliable there are a lot of people AR working on different ideas for how to do that and I think that we will see these things cross over from being ideas like Auto GPT and baby AGI which were trying to make an agent that could do anything to being much more specialized agents where the agent will be sort of crafted to do a certain type of class of tasks and be able to get really good at handling those kind of tasks and I think this is one of the areas that we'll see things take off with with perhaps a Revival in the RPA or the robot processing automation area where that a lot of companies came onto that field a few years ago and people started to adopt their products but then it didn't really fulfill a lot of perhaps what they had promised or what the whole sort of movement had promised there and I think this this is where agents may come in and actually improve a lot of that stuff a lot another thing that I think is really interesting to think about and I'd love to hear people's views of this in the comments is that the whole sort of UI and ux for AI hasn't been properly worked out I think over the last year we've seen obviously the biggest product being the chat interface with chat GPT with character AI Etc opening I opened up their whole voice interface towards the end of the year but I think this area is an area that hasn't been explored as much as it should be and I think this is one of the things that we want to think about is that okay as we're creating AI applications as we're creating these things what does UI look like for AI what is ux like for AI if you don't have a chat interface how are these things going to be able to respond I think this is an area that people are going to do a lot of interesting research this year and also I think just the community building products are going to try out new things of where people start to realize that H maybe people don't always want to just have a chat interface and while the voice interfaces are great they're not always suited for every particular task so I think that's an area that we will see a lot of things happening as well as we go forward so overall i' would say that there's a lot of cool things happening I haven't really even touched on the whole orchestration Frameworks things like Lang chain llama index a bunch of these things that are coming up I'll be certainly making videos about these things as the year goes along and I would love to hear from you guys of what sort of things you''ll be most interested to see as we progress this year with the different uh things that are out there so overall I think 2024 is going to be a fantastic year for AI I don't see the whole sort of AI winter that people are talking about I think some of the big things are going to be around ideas that were on The Fringe last year now becoming much more mainstream with the things like agents with some of these particular uses of models and stuff like that and I think it's a really exciting time to be involved in all this stuff so as always if you've got you know comments or questions please put them in the comments below I'm certainly interested to hear what other people think are going to be the big trends this year and what things are you know going to happen I I will certainly read and check out the comments for this and as always I will talk to you in the next video bye for now