Mistral Large with Function Calling - Review and Code

Key Takeaways

okay so we have a new model from mistol out and in this video I want to go through some of the really interesting things about this model about mistal AI the company and then also we'll have a play with what it can do both for just normal prompting but also for function calling and seeing how we can actually sort of use this as an alternative to some of the open AI models that are doing function calling Etc so first off I'm not going to Hype up the model like a lot of people are doing this is a good model right if you came to watch the video just to find out if it's a good model it's a good model that's not the most interesting thing about this I think one of the things that the sort of second point which is much more interesting is that this model hasn't been dumbed down by overly rlf perhaps like some of the big companies have done out there and then the third point I think that's really interesting about this model is that they've done all of this in under 10 months so M itself hasn't even been around for a year and yet they've released probably one of of the best open-source 7 billion models both base model and a very good fine tune instruct model and they've released a number of proprietary models and now they've gone and released their large model and my guess is this is probably not even the biggest one coming one of the things that I find fascinating about this is that Arthur mench the founder and CEO of the company has basically said that this model only costs 20 million to make which basically means it's way cheaper than what open AI is spending on making these models and so there's a whole interesting sort of dynamic here of if you can make these models more often and cheaper than other people are doing you're going to have multiple bites at the Cherry meaning that you can then learn from what people like about the model what people didn't like about the model and then be able to use that when you're doing new fine tunes of that model but also when you're doing whole new models as well going through this so if we come in here and we look at the announcement so this is mistal orage I think means to go offshore and so that would fit in with the theme of the mistal being the sort of winds blowing and this is sort of their Flagship model that is going out into the world and I must say as a smaller side I find it very cool that they're basically launching this model and then in the headlight it just mentions it's also available on Azure so rather than make a big faner about these things they're just putting these things out without the hype without the sort of teasing and stuff around it so it's like we're going to serve this model we know that a lot of companies probably are going to already be doing deals with Azure so if you want to use it on Azure you can do it on Azure so I should stress that this is a proprietary model right this is not an open weights model it's not an open- Source model like their 7B model where people can basically do anything with it they're serving it themselves on their servers the versions I'm going to show you with code in here are using the mistal AI API which actually I found to be very fast so I think that's quite good but it is definitely a proprietary model one another thing that I find to be to be really interesting though as well is that they're also open for people to basically run this model on Prem so if you compare this to Google you compare this to open AI they're very reluctant to do anything on Prem anthropic has been open to doing stuff on Prem and there are a lot of users out there that are big companies that are hedge funds that are people that just can't put their data out into the world and they really need to run these things on Prem so I think that M's going to have a huge win here just from this alone that they're prepared to basically talk to customers and and work out how people could use these in sort of most sensitive use cases to access model weights and stuff like that like I mentioned already the model is going to be available both from their platform which is LA plateform and Azure anyway let's jump in have a look at the model I'm going to talk a bit about the benchmarks I'm going to talk a little bit about where like all large language model companies they're not always comparing Apples to Apples sort of comparisons in here all right so the main figure that they show in here is this Benchmark of MML U now I'm not really sure why people still keep hanging on to mmu as being like the prime Benchmark I don't think is a great Benchmark at all for this so we can see that they're comparing themselves second to uh gp4 here yet they very conveniently left out some models perhaps rightly so in that some of those models are not yet publicly available but if we help them a little bit with their comparison here we get something uh like this and we can see that actually when you make this comparison and you include Gemini Ultra and you include Gemini Pro 1.5 in reported benchmarks they're actually now in fourth place in here really I feel that the benchmarks are ending up becoming like a a big distraction with these kind of things like I said earlier this model is very good and I kind of feel it's definitely a flagship model on par with a gbd4 or on par with a unnerfed Gemini model some other interesting things about the model is that it's a multimodal model unfortunately it's multimodal in the sense of Western European languages there's no ceric languages in here there's no Asian languages in here but that's fine right a lot of companies are just focused on English so it is good that it goes beyond English and is definitely usable for other European languages next up we've got that it's a 32k context window in here so 32k is very respectable context window for these kind of things yes it would be nicer if they go out to 128k or further and my guess is that they're probably working on that so I wouldn't be surprised to see a future version of this that actually has a much longer context window they've also talked about that it's pce in instruction following so this is one of the things that I find to be really big advantage of this model is that it does pay attention to your instructions and doesn't always just say no or things like that they've also done that in a way that it basically allows developers to set their own moderation policies so by having it to follow instructions it's going to be really interesting to see can people jailbreak the prompts on this quite easily or not and then the last one that I really want to cover in this video as well is the whole idea of that this is natively doing function calling so I think after having played with this model for a while one of its real strength is around reasoning I find it to be very good on things like GSM 8K I find it very good on things that require decision making that kind of thing so I think this is one of the advantages of this and my personal belief is that a lot of the big models from other tech companies are actually compromising their reasoning and compromising some of these skills by trying to make it overly safe and overly rlh fing or doing some kind of alignment training which is basically reducing uh the ability for these things to be able to reason as well lastly along with releasing this model they've also released their own chat platform that they're calling Le chat this is basically just a chat interface where you can select different models you can try some of these things out Etc let's jump in and have a look at some of the outputs and then we'll have a look at the function calling for this stuff as well all right so let's have a look by starting with the mistal large the the latest model that they've basically released so I'm actually going to use Lang chain to actually do the calls here you will need to basically get a mistal key and set this up so if you just go to the API docs you can basically sign up get a key Etc you can go through it I've basically put my key over here in the collab Secrets just like normal and then I'm just bringing in some standard stuff from Lang chain so that we can call this so you'll see that the model that I'm using here is basically mral large largest we can do the sort of standard just invoke chain to get a response back I'm just going to print it out in in markdown here we can do streaming chains if I've got that in here we can see we can get something like that and then we can also do you know batch chains if you want to try doing that and you can also then use the L chain expression language in here so this is just taken from the L chain examples all right so for testing the actual model I've basically just set up a generate function and made a little function where we can basically pass in a an instruction a system prompt and a max length and so basically just like we test all the other models and see how it goes I'm going to go through this quite quickly the responses are very impressive right they definitely have their own kind of field that is different than both open Ai and Google and anthropic right it's definitely their own kind of feel and I think this is one of the whole things it's interesting mistal 7B was like that also in that it had its own kind of way of outputting things for me this actually feels a bit different than the 7B one my guess is that probably because it's trained on a lot more data those sorts of things okay so right a detailed analogy between mathematics and a lighthouse we've got sh be happy to create an analogy let's break it down step by step so obviously in my system prompt there I'm asking it to do the step-by-step thing it does it quite nicely does it quite different than some of the Google models how they're sort of seem to be heavily relying on Chain of Thought this is doing it quite differently here mathematics and music again we get some nice results out you'll notice that the response times here are actually nice and snappy on this big model as well the standard question that I've asked for going on a year now coming up on a year now what's the difference between a llama vuna and alpaca again we get a sort of structured response because we're asking for the step by step it is quite different than the structure of the responses in the mistal 7B though so that that's interesting to to notice so I I think this is clearly paying attention to the the system prompt in here we asked for an email to Sam ultman we put out write out your reasoning step by step so it does a better job than something like the Gemma model last week which in this question it just gave us reasoning it didn't give us an email when we asked for the reasoning here it's still giving us an email but it's putting the reasoning in the the context of the email again quite Snappy generation here 5-year-old the 5-year-old is probably using some words that I'm not sure I VI would use is in transparency maybe it hasn't adopted the personality as much as some of the other models have out there and then the one from the vice president again we've got quite a a well structured argument here the question is it adopting the personality as much as we might want in this all right things like this so what is the capital of England you're Mr Large write out your answer short and succinct it hasn't been succinct here right it's been anything but succinct probably even more than some of the other models out here the question about Jeffrey Hinton and I encourage you to come in and put your own prompts in here I always tend to stick to the same ones so we can compare against other models that we've seen sure enough it gets to that Hinton is living George Washington is no longer alive he go through sort of the reasoning and then comes back with the answer that Jeffrey Hinton cannot have a conversation it's perhaps a little bit verose I would say compared to you know some of the other models that we've seen here for creative writing I think this is interesting you might want to play with the prompts to get better results out of this I don't think this is the best creative writing that we' we've seen although it's clearly doing the task it's clearly going through this code gen seems to be very good for some basic ones and the ones where it really shines are these gsmk ones so out of all the GSM K ones that I gave it it got all of them which is very impressive that's definitely up there with GPT 4 level quality model most other models will fall down with one or two of these questions in here but even this one where it was able to basically work out that we're solving for x 7x = 847 therefore we're basically 121 people and then when we give it the math version of the same thing it's able to do it quite nicely as well definitely a good model and worth playing with what I thought I would also do is have a look at the mistal medium so I haven't made a video about that in the past so I took the same notebook and just ran it through with the m medium latest model in here and this is also a good model so we get different you know results out here but they're kind of similar right they're kind of similar maybe the logic and the reasoning is not quite as good but I encourage you to go through and sort of look at these yourself and sort of see how they are we can see that this one is relying more on the step by step perhaps a little bit more like the mistro 7B does rather than the the bigger model the young child email to samman actually seems to have done a nice job of capturing the character maybe a little bit better than even the big model here so I guess really you want to try these out and see okay how well do they go looks like we had a failed run on this one that one I should just run again meanwhile we're looking at that what is capital of England this is more succinct here can Jeffrey you can have a conversation again we're getting to the right answer we're getting different sorts of reasoning along the way the creative writing again similar name similar kind of thing to the big model so it does seem that a lot of the stuff is trained on uh similar data here and then finally the GSM 8ks with this mediumsized model actually does pretty nicely it seems to be getting all of these right with this same model here now the reasoning is different than the large model but it does seem to be in some ways for some of them it's more succinct for some of them it's more verose but again all the GSM questions are correct here so overall I'd say take the notebooks have a play with it yourself see what you can get out of this and then decide if you like this model I can definitely see that for some people this may be an alternative to the open AI models and to some of the anthropic models and it will be very interesting with the large model to get it to try and do some of the sort of evaluations along the lines that people do with uh ragas and other sorts of llm evaluations where you're getting one llm to evaluate another llm so it' be really interesting to see where people have always used GPD 4 for evaluating the open source models now I do think we've maybe got another model that with the Mist large here that can do that as well it definitely seems to have you know a good sense of reasoning in this model so I'll be interested to see what people do with it for that as well okay so one of the big features of mistal large is its ability to do function calling this is something that we've seen open AI Implement extensively and people are using it for a variety of different tasks especially things like agents and just getting the model to return instructions of how to run a specific tool or a specific function in here so with Mr Large it's not hugely different I'm just going to walk through some examples of this so the first thing you want to do is set up your tools so here I'm basically just making this up as if we like a a restaurant and this is like an online booking ordering platform something like that there are two tools here you can either order some food to pick up for a takeaway order or you can make an online booking for a certain time now you'll notice here that I'm returning things back all I'm doing is just taking these in and returning them back you could imagine in here I'd have some logic that actually sends it to a point of sale or does something else with it in here one of the key things here is the what you're sending back is what's going to be sent to the language model to confirm the last part so you'll see later on I've sort of played with this second one a little bit to get the the language model not to say things like oh okay I'll email you or call to check to confirm the time or something like that here we're basically taking in one variable for food items so it's just going to be a string of food items and in this one we're going to take in two variables one for day and one for time in here all right so once you've got those tools so these are the two tools that we've got and you could imagine that these could do API calls they could do a variety of different things we then want to make a Jason schema for each tool so the Json schema is pretty simple here I'm basically having because I've got multiple tools you're going to have a list with two objects in it the first one is going to be called takeaway takeaway order and the Really key part to this is the descriptions so put an order in for the food that you want to pick up and take away and then food items the food items you want to order right so these are passed into the model so that the model knows what arguments it needs to give back to the function in there and you can see that the required arguments back are the food items in this case the second one and if I was making it a little bit fancier I'd probably make this a list of food items but here I've just put it all in one string the second one is again quite simple this is an online booking place a booking at the restaurant for lunch or dinner the two properties here are basically day and time the day that you want a book to come in and eat at the restaurant and then time the time you want to book for lunch again if I was making this a little bit more sophisticated I might have something some sort of checks about date and and that kind of thing as well the idea here though is that we're going to get two variables out and I want you to see that if one variable is given but not the other one is given the good thing with this is that the language model can actually get that variable out of the person so that you return that back as an argument so you can see that the function requires two arguments the day and the time going in all right once you've got that set up you're going to need a way to call the function so you set up a dictionary where it can just take in a string that matches what we've got here so you can see here this is online booking so this this is online booking here and then that's going to basically call the function online booking and it will pass in the arguments that we've given it from here if you wanted to pass in other arguments you could hardwire some of those in there as well for doing that all right so now I come down to start the conversation so you can see I've basically got a simple chat message it says hi can I put in order to take away pick up in 30 minutes please and you see that we pass this in to the model and we're going to pass in the messages so this is our basically our list of messages which is going to obviously be roll user roll assistant or roll tool for this if it's a tool response very similar to open AI in that sense we then pass in the actual tools so it knows that it's got tools to pick and we we're giving it the Choice Auto so it can choose to basically pick a tool or not pick a tool we're not forcing it to pick a tool in this case all right you can see that it it goes through we get a response back saying yes of course I can help with that could you please tell me what food items you would like to order for takeaway so that's just a normal response back so what we do is we append that assistant response back to our messages and now we adding a new message for the user can I get fish and chips and a suaki okay we then basically can see our messages now we've got the user message we've got the response from the assistant and now we've got our user message again we send this back in again with the messages and the tools as well and now we get back a a function call right now we get back something where it's telling us that okay you're going to call the function or the tool takeaway order and the arguments for food items are going to be fish and chips and a suaki in there and so you can see now we can basically append that and we've got our messages now where we've we've got the user assistant user assistant but this time the assistant's giving you know instructions no content for it saying anything but instructions for a tool call and this is where the tool calling happens so at the tool call basically we get that response back for the tool call we can then basically get the function name out for that and we can then run it through so you can see here we've now got what functions going to be called what are the parameters are going to be called for that and then because we had our dictionary that we made our names to functions we can just pass in that function name in here and then we can pass in the function parameters so you can see so that's showing the ones I ran at the end which you'll see in a second but you can see that we would pass in basically in this case tiway order and then food items would be the various things there and then we get the response back from the tool to basically say your fishing chips and of likely is on the way we then append that as a role tool because that's a response that didn't come from the user it didn't come from the assistant it came from the tool so that goes back to the we we append that to the messages we send that back and you can see that now we've got our message list is obviously getting longer we send that back and sure enough now we get a response back that is just a standard assistant response that says great your order of fishion chips and suaki will be ready in 30 minutes we'll see you there herei basically Incorporated the early messages of saying that the user wants it in 30 minutes kind of thing so in that particular case there was only one argument what if you've got multiple arguments and they don't give all the arguments at once so here we've got the same sort of thing but now we're going to use the booking tool rather than the takeaway order tool hi can I make a booking for dinner on Friday night and we can see yeah of course what time would you like to book for dinner on Friday night so it's kind of because we passed in the tools knows that it can't just automatically call the tool because it doesn't have all the arguments for the tool so we then append basically the assistant response our user response that says can I get a table at 8:00 P.M please we then get the the response back which is the tool calling response basically telling it hey you need to call this function online booking and the arguments are going to be day Friday time 8:00 p.m. we then go through the tool call all get the details out of that we can see that your booking is set for Friday 8:00 p.m. no need to see now this is I've done it as if the tool is kind of talking to the customer but really you could just have it come back and say booking confirmed Friday 800 p.m and then anything else you want to pass to the large language model so in this case there's no need to reconfirm please arrive on time I added that in after you know trying it out a few times because the model kept saying oh okay we'll send you an email to confirm or something like that and I didn't want that I wanted to basically be able to say straight away no need to reconfirm just come along so you would want to play around with both the inputs to the functions or to the tools and the outputs that you get back from the functions and tools there you see finally we take that response we send it back to the model and it formulates a response back to the user because this is a tool response here that we append to the messages not from the assistant it's not from the user it's from the tool and you can see now the model can actually use that to formulate the assistant response back which is great your booking is set for Friday at 8:00 p.m. please arrive on time there's no need to reconfirm enjoy your dinner this is sort of an example of using functions as tools and having more than one argument for those tools or functions and how you would process this through and you can basically put some conditional logic in this to make it be a loop that just goes through and keeps asking questions until it's got all the arguments to send for one particular tool or function so that's something that I've done before in some of the examples for the open AI ones if you go back and have a look there you can see those ones anyway overall I'd say that the Mr Large model is definitely an interesting model for checking out for these kinds of things it's cheaper than GPT 4 it seems to me to be very good at the sort of reasoning and the function calling kind of stuff and unfortunately there aren't any sort of benchmarks that I know of that we can easily just sort of test you know a thousand examples against each and see what they score that's something that you could do if you've already got function calling implemented for open AI you could try this out very simply just by changing your code a little bit for this so overall I would say it's great to see that we've got a new large language model that can do this sort of stuff now I think Gemini Gemini Pro and Gemini Ultra are in the process of adding function calling and sorting out the function calling that they've got a little bit so I'll probably make a video of that at some point as well but overall Mr Large is definitely an interesting model definitely worth you checking out and seeing okay what you could use it for in your particular use case as always if you like the video please click like And subscribe if you have any questions or comments put them in the comments below like mentioned before I always try to read the comments at least for the first 24 48 hours that the video comes out and I will talk to you in the next video bye for now