Comparing LLMs with LangChain

Key Takeaways

in this video we're going to be looking at comparing and evaluating large language models using blank chain so one of the issues that a number of people have asked me questions about is how do I know if this model is going to be good for production Etc basically if I compare it to chat gbt it's nowhere near as good the obvious thing is that of course A lot of these models are not going to be anywhere near chat GPT because they don't have the same fine-tuning they haven't been trained on the same data they're probably not the same size Etc what we can do though is we can set up a bunch of models and we can then basically ping them with the same prompts and see okay what do we get back are we getting back for similar things and you will find that some of the models will be really good for maybe classification or fact extraction much more than they are for other tasks that are maybe creative writing or something like that so this allows you then to see that okay maybe on certain tasks on certain chains I can get away with using a much cheaper model or a model that I'm just pinging from the hugging face Hub rather than having to spend money on tokens from the chat GPT API so let's just have a look at this I've just got it set up in here as always the collab to this is in the description all of my videos I've put the collab in the description and first off we're gonna basically once we've set up our keys so I'm going to be using openai cohere and hugging face we're going to compare about seven different models in here and we're going to set up the models and in this case I'm actually going to use the same temperature for each model so remember the temperature is what causes the randomness I'm not going to set it totally to zero I'm going to set it very low in this case now I encourage you to experiment with this and go through and do these yourself so the first model that we're setting up is the flan 20 billion model so this is the model that I've done I think a couple of videos on already came out recently this is a fine-tuned version of with the instruction tuning of the ul2 20 billion model on top of that we've also got the flan T5 XXL model so this is an 11 billion parameter model and you see both of these we are using from hugging face Hub rather than locally we could load some of these up locally I've just chosen in this one to make it simple for people so that you would just go for the ones in Hub if you're interested to do it locally have a look at my video on the hugging face and serving those models like locally okay so I wanted to try and compare the GPT Neo XT 20 billion and I've got this working in a code lab locally but unfortunately it doesn't seem to be working when I ping the hugging face up for this which is not surprising right it's quite a big model the same is true for the bloom 7 billion parameter model again these models cost Fair bit of money to serve my guess is that they're not serving it unless you're paying for it one of the ones that they are serving which seems pretty cool though is the GPT j6b model so this is a 6 billion parameter model this is actually reasonably it's not a totally new model that's come out in the last month so this is a while back so the the models that we're getting from hugging face hub from open AI I'm going to get two models we're going to get the chat gbt turbo model right so GPT 3.5 turbo and we're going to get the text DaVinci zero zero three model which is the old model that everyone used to use before the chat GPT API came from coher we're gonna get two models from them so we're gonna get the command XL so cohere command models are the same as the instructor models so if you think about the text DaVinci this is a retraining of the GPT instruct model this is the same kind of thing they call them command models and this is their Excel model and then they have a Excel nightly which apparently is up just updated as it's training so they've got this thing constantly training and they do a checkpoint and this is you're getting the the last checkpoint from that in the past I've looked at you when I've looked at cohere I've tended to just use this one this whole sort of comparison thing is changed my mind about this as you'll see when we go through this so we've got our models up right or we've got our language model chains up for those once we've got those set up we need to basically set up a model laboratory so link chain now has this ability in here to run a variety of tests in here and we can basically just do this by importing the model laboratory and then I'm just going to set up a simple prompt template and then we're going to set up a model laboratory passing in all the models that we want to test and passing in the prompt that I've got here so this is going to be called lab so we've got the basics of those and we've got each of these set up and now as long as we're sticking to this kind of prompt we can just pass in lab dot compare and pass in our input and it will then go through it so you can see first up we've got the chat gbt API so here I'm asking what is the opposite of up ideally I think most humans would just say down but it's interesting that the chat gbt goes into a long sort of of evaluation most likely because we've got this Chain of Thought prompting going on here let's think step by step right so that's the chain of idea here so if we wanted to we could turn this off and just see okay what does it give us without any of that well if we look it shows eventually it does get to the right answer I think therefore the opposite of Up Is Down okay So eventually it does get to the right model and we can see therefore the opposite of up is down if we come back over here and we can see the old GPT 3 Model actually wasn't trained I don't think with Chain of Thought prompting in it you can see that it just goes straight for the answer the opposite of up is down so it's important that you understand the reasons why these are giving different answers because we've asked for the Chain of Thought prompt we use Chain of Thought prompting we're asking for the reasoning or let's think step by step here we come to the GPT j6 this gets it right what is the opposite of up the opposite of Up Is Down what is the opposite but then it goes on just an on and on step two it's very verbose there and maybe not ideal for what we want if we come to the flan 20 billion model we can see that okay we're getting down is the opposite of that that's pretty good we've got a bit of the reasoning and we've got the answer down so that's probably one of the better ones that we've had so far if we look at the T5 XXL again the flan model we can see that it's it's doing pretty good but then it goes off on this direction of the sun the sun is all in the sky before it eventually gets to the answer is down so you can sort of evaluate and what I suggest is your particular use case you should make a set of maybe 10 or 20 different prompts and then be able to see for different tasks which one is better for you okay the cohere one this one we're seeing fewer up you are above something if you're below something you are down the opposite of up is down again it gets to it with maybe a little bit of weird reasoning and then finally we've got the coher nightly model it looks like it might get there oh but then it says oh the opposite of up is not back the opposite of up is not down the opposite of up is not under so and it just goes on and on for that and this is one of the things that I found doing this was actually the coher nightly model which I thought would be better it actually turns out to not be generally for me looking at this next we're going to take one of the questions from the flan paper and this is the answer to the following question by reasoning step by step the cafeteria had 23 apples if they use 20 for lunch and bought six more how many apples do they have the answer should be nine but we should get step by step okay the Chachi BT says cafeteria had 23 apples they used 20 which means 23 minus 20 equals three three plus six equals nine therefore and this is giving us a really good answer here right it's got the explanation Etc the GPT 3 Model again this wasn't trained on Chain of Thought prompting and we really see this here right that we see okay it gets these bits right so it's extracted that from the question correctly but then it's all 23 plus six it equals 29 and that's not correct right it should be 23 minus 20 plus six so it skipped a step there if we look at the elutha model the gbtj model and like I said this is I wouldn't say a super old model but it's a reasonably old model you can see it's just trying to do completions and it's not really I don't think this is a fine tune on any instructions so it's not getting the great results for this the flan model this handles it perfectly it gives us the exact answer that we want the smaller flan model the T5 one doesn't do a great job it says they bought six plus three equals seven apples so it's getting its logic wrong and that's probably just because it's a smaller model but this is 11 billion this is 20 billion okay the cohere command model the command XL large or extra large model they had 23 gets extracts the key bits of information right but it does an awful job at the math and basically says they have seven if we look at the nightly model it just tends to go on it's going on and on for this next one up again another question from the flan paper and I think it's even in the Palm paper this is definitely suited to to a much bigger kind of model right because we're asking at specific facts about people so you can see that in here when we're asking can Jeffrey Hinton have a conversation with George Washington give rationale before answering okay so open chechi BT basically starts off with a very long thing about okay could they travel together is this scenario possible goes through and it's kind of off base though right it doesn't seem to know who Jeffrey Hinton is it doesn't seem to have any dates for him there this one actually the the GPT 3 Model kind of seems to do better here the first Jeffrey Hinton is living person and George Washington's deceased person second a conversation requires two living people to communicate with each other therefore no Jeffrey Hinton cannot have a conversation with George Washington so actually the logic in this one is actually much better than chat gbt for this so bad thing to look at a Luther again not so great Jeffrey Hinton oh George Washington is dead so it got that right Jeffrey Hinton is not dead got that right Jeff again is not a ghost not a zombie he's not a vampire okay it's just doing completions right at this stage we would be going okay this model is not great for unless this is going to be fine-tuned more for instruction we're probably not going to use this model what about the flan ones okay the 20 billion plan George Washington died in 1799 Hinton was born in 59 he actually was born a lot earlier than that I think it's 1947. so the final answer is no so it gets the answer wrong even though it gets a fact wrong and with the T5 we also see the same thing gets the answer right but gets a fact in there wrong the command models he's hinting a real person yes is Washington a real person are they both alive no are they both dead no do they live in the same time period no do they live in the same country no do they live on the same planet yes so this is interesting it's reasoning here right this is showing that maybe you could prompt this to do very specific things can they have a conversation over the phone yes they could have a conversation on the phone no I don't think they can I think if George Washington is dead I think it would be very difficult to have a conversation phone the nightly model again it's doing things like this but it does actually come to the right decision of saying that they cannot have a conversation but it's interesting because it seems to decide can they both speak the same language which I think the language probably wouldn't have been a problem the fact that one's alive and one's dead would have been a problem all right let's look at something a bit more creative so this is where we're asking it now to tell a story and these ones we were definitely out of the box expect the bigger chat chibi team kind of model tools and gpt3 models to do well by the looks of it they do pretty well I'll let you just go through this this is interesting though this is where the elutha one probably does better than it has on a lot of the other things so it's actually got a story going it's got parts of it but then it just goes into an endless loop so this could be because we're not setting a penalty we maybe need to play around with the penalties for repetition and stuff like that in there but the idea here is I'm sending them all to be the same okay the flan models that Jason was a professional carrot he was an athlete he was a great basketball player he was a great football player great basketball player with a great swimmer so they get that bit they just go into all the things that he's great at and we don't get the actual Story coming out there on that one I think this is you would run it a few times maybe to see okay again on that one also this could be a thing to do with repetition penalty the command x large model this seems to do much better though this time you can see that it's got the whole thing about him being a sports fan and then one day so he's got this girlfriend Jessica cheated on him she had an affair with his best friend Jacobson was heartbroken he started to working at a farm growing carrots he was good at it he made a lot of friends this is interesting and unfortunately he meets another girl who also cheats on she was the professional athlete so he doesn't get all the facts right but it gets perhaps more than the other ones did with this and then the the nightly model here is doing a similar thing to the normal one by the looks of it so I'll let you go through and read that one the next one I thought was an interesting one that I took from the flan paper this is I'm riding a bicycle the pedals are moving fast I look into the mirror and I'm not moving why is this so this is a little bit like Common Sense reasoning here so again we would expect the bigger models to have to show more of this ability than smaller models so we can see here that okay you're likely on a stationary bicycle or train so chechi BT does pretty well right at doing this the old GPT you are not moving because you are coasting the bike is still in motion maybe this is slightly possible but this is certainly different a Luther totally wrong the bicycle is moving because the bicycle is moving the flan ul2 actually gets it that I'm stationary but it doesn't really explain that we're on a stationary bike and then the T5 flan is a smaller model I'm looking at the wrong angle so this is definitely the hardest one of the harder questions I would say to some of the other things this is definitely going to be more suited to the big models so the coher models do one of them does quite well the mirror is on a stationary bike so that's interesting that nightly one gets that this one maybe not not as well okay the last one I want to look at was fact extraction so I've edited down the article from the last video where I did something like this just to make it so that it'll fit in with the tokens but basically we've got a sort of edited article about the recent Mobile World Congress something we've got some facts in there about a 6G we've got some quotes from different people and then we've got this referring to the OnePlus coo and so we're asking it to basically extract the OnePlus coo and this is the task that you probably don't need a huge model for right so if we go through and we look at this okay chat GPT has no problems with this it gets it straight away gpg3 has no problems with this it gets it straight away looking at this even the a Luther one extracts out the right thing it just puts in a lot of other stuff as well the flan model it gets it both flan models get it and both cohere models get it so it shows you that this task is probably not as hard as the previous ones so if you had some kind of fact extraction maybe that you would try a smaller model than paying for the open AI one all the time doing the same context in here and now trying to ex we're just trying to get out what is supply chain Innovation and what I was looking for is this foldables supply chain Innovation so let's see do any of them get this it's a little bit vague in there too the chat gbt ones seem to give us a much longer for this so does it get two foldables in the end no it does talk about things like flexible LED screens though so it's relying a lot on a lot of the weights a lot of the information that's coming from its weight so it's hallucinating an answer in some ways and then the old gpt3 is basically just giving us a definition of what's Supply driven Innovation is rather than looking for what's in the actual yeah I'm not saying anything about foldables there rather than what's actually in the context that we gave it the elutha one looks like it gets foldables but it's got so much other stuff around it it's not really useful the flan models both seem to get it the flan models I definitely think are very interesting to be used fact extraction that's one of the things that you could use them for passing across documents and stuff and extract acting various facts that you need out of this and finally the cohere ones that again they seem to be more interested in giving you a definition of this than actually looking at the context of this so the idea here is just to give you a rough idea of going through these testing out various models you can plug any models into this and try this out if you've got the access to them and you could also load a model and just set it up locally with that if people are really confused about that I can do another a small one of these showing how we would do that anyway as always if you have questions please put them in the comments below I'm happy to answer them if you found this useful please click the Subscribe and I will see you in the next video bye for now