How to select the right LLM?

Key Takeaways

I get a lot of questions about how to select the right model when I'm delivering my trainings or I'm preparing my courses this is an important topic that we go through when discussing how to build with large language models and I often have these conversations with developers and Founders because it's an important question and it's important because you want to find the right model to experiment with evaluation is already very difficult and timeconsuming in my opinion in order to get the right answer to this it is always going to be about experimenting on your own there's no like secret recipe out there where you can just go and it will just tell you what model to use based on some requirement what is often the case is that you do a lot of experimentation with different models but you need a sort of framework to be able to pick the right model for your use case and so what I came across is this interesting guide well it's an official guide from opening ey how to select the right model it is their models but I think there's something here that we can learn from that can transfer to other LM providers as well or if you're using other models out there what I want to do in this video I want to go through this official guide from open ey on how to select the right model and have a bit of discussion on some key points that I mention here I'm also going to be sharing some insights and some of my own experience on how I go about selecting the right model for my use case what di menion here is choosing the right model whether GPT 40 which is their most powerful model or a smaller option like like GPD 40 mini requires balancing accuracy latency and cost so you always hear about these tree and often enough you will be making decisions about which model to use based on these aspects now again what you will see here is a framework it turns out that this works really great for openi models and I can attest to that because we heavily use the openi models and we follow a very similar framework now this framework might not apply to every model or llm provider out there but I think there's something that we can learn and something that we can transfer and apply and adopt to other llm providers and language models and even use cases so let's get right into it so at the core you'll be doing two things to properly select a model so you're going to be optimizing for accuracy first that's the first step again this is why I refer to it as a framework and then you're optimizing for cost and latency second again the first one in summmer will be to optimize for accuracy on until you hit your accuracy Target you always have an accuracy Target right then how they discuss this is you have maybe some Target on cost maybe you have an accuracy Target and so on and then eventually what you want to do is as a second step is you want to maintain that accuracy but you also want to make it cheaper you want to make it faster and the only way to get it cheaper and faster is by potentially using a smaller model which in this case as you can see here will be something like GPD for a minute but there are other ways how you can achieve this this is just one way all right so let's focus on the first part here so it says Begin by setting a clear accuracy goal for a use case this is no different from any machine learning project when you're clear on the accuracy that would be good enough for this use case you can go to production that's the goal at the end of the day and so here are the steps that you will be typically going through so you have this accuracy Target as an example they say 90% of customer service calls needs to be tri-edged correctly at the first first interaction then you develop an evaluation data set this in itself would Encompass a huge amount of effort if you have develop machine learning models and machine learning applications in the past it's no different with llms it just becomes more difficult and a lot more work but again this is assuming that you already have knowledge of that this is more showing you the high level framework so you can create a data set that allows you to measure the model performance against these goals to extend the example above capture 100 interaction examples where we have what the user ask for what the llm Triad them to what the correct Triad should be and whether this was correct or not and I really love seeing this explanation because this is exactly what we do in our course when we talk about prompt engineering let's say we were working on like a chop bot system and we have the prompt itself where we prompt the model then we have this response that we get from the model we have the desired response that we want which is the ground truth and then an extra label whether this is correct or not this is basically what they're doing here and it's more for this triaging use case of customer service calls the third Point here is using the most powerful model to optimize you want to start off with the most powerful model in open AI case it is GPD 40 a lot of people don't even know this but you should be experimenting with the most capable model at the beginning it might save you a lot of time on how to optimize your prompts it might save you a lot of time on deciding whether these models even make any sense from these llm providers so I think it's an important decision important step in this framework especially this first part where you're focusing on accuracy first so start with the most capable model available to achieve your accuracy targets log all responses so we can use them for distillation this right here is super key and this is why I tell people prompt engineering makes a lot of sense it's not going away anytime soon I'm not making a case for promp engineering in this video board but I'm mentioning it because it's part of the development process of building llm applications in fact when you're doing this prompt engineering you're doing a lot of experimentation and evaluation on those prompts and so when you're testing the model with different prompts and you're Gathering these responses you're essentially building out a data set you may be doing something like fuch prompting which they do mention here in a bit all those examples that you're putting together or exemplars in this case well those are going to serve not only for improving on the task that you have you can gain insights as well as you try to patch and solve some issues that the mall is faing but eventually you can take all these examples let's say you have hundreds of them and then find you them all and this is what they're referring to as distillation of a smaller model so it says here use your trial augmented generation to optimize your accuracy and this depends because you don't always need rag but it is often the case that rag is pretty useful not only for when you want recent events it's actually really useful as a system to improve the quality of your responses because you're giving the model more relevant context so rack can be used for many different things and then it says here use fine-tuning to optimize for consistency and behavior so there's a lot of ways of how fine tuning is being used but one of the ways that you can use fine tuning is in this framework is to optimize for refining outputs for instance maybe fine tuning a model also to structure the output in the format that you want that's really hard to do with these models out of the box so it makes sense to do some fine tuning you know those responses that you're getting might not be in the style that you want or the tone you would use fine tuning for that and I often find that fine tuning is very helpful for those use cases not only for the open am mods but also for other providers as well this is a really important advice so it says during this process collect prompt and completion periods for using evaluations F learning or fine-tuning this practice known as prompt baking helps you produce high quality examples for future use you can see that they refer to it as prompt baking prompting the model optimizing your model getting these responses and logging everything making sure to log everything that you're doing and for this case you may need some kind of experiment management tool or if you're logging things into a database make sure you have that don't just experiment and not save the results that you're getting out of those experiments those might be actually useful later down the road then set a realistic accuracy Target we're still in the first part it says calculate a realistic accuracy Target by evaluating the financial impact of M decisions for example in a fake news classification scenario so correctly classified news if the model classifies it correctly it save you the cost of a human reviewing it let's assume something like $50 incorrectly classified news if it falsely classifies a safe article or misses a fake news article it may trigger a review process and possible complaint which might cost us $300 so it says our newest classification example we need 85.8 accuracy to cover cost this is regarding this specific scenario but you may have completely different targets for your own scenario so targeting 90% or more ensures an overall return on investment and you can see here this is more objective because now you have all these measurements that you have done like this is going to cost $50 this is going to cost $300 and so we need to have at least 90 in order for us to get a return on the investment so use these calculations to set an effective accuracy Target based on your specific cost structures so it will vary depending on your resources so again what we're trying to do is trying to figure out whether the decision that a mall is making how it is going to financially impact C we have to have an idea of this before even starting to invest time and resources into these projects and again no different from classical machine learning projects it's the same here and you're just being more objective which I really like from this suggestion the second part here which is I think the more important one is optimizing cost and latency so once you have a really good Target right and it's more objective then the next part is what are the kind of optimizations and efficiencies that I can get with these models and what are the models that I'm going to use so it says here cost and latency are considered secondary because if the model can't hit your accuracy Target then these concerns are mood right they don't matter so this is why we have that first part where you're selecting the best model running those experiments have that eval data set and now you're being more objective in the work that you're doing you don't have vague calculations you don't make assumptions about whether you're getting a return on the investment whether the models that you're using make sense for your use case now you have a framework says however once youve got a model that works for your use case you can take one of the two approaches compare with a small model zero or few shot so swap out the model for a smaller cheaper one and test whether it maintains accuracy at a lower cost on latery point so initially we started out with this big model now we swap it out for a smaller model so with the open ey models it would be something like GPD 40 mini so smaller models are becoming a thing and this is the reason why and I mention here zero or few shot that's really important right that's what you want to be experimenting with it says model distillation find you a smaller model using the data gathered during accuracy optimization so in that first step we were putting together that data this is why I said you don't throw away that data in fact it makes sense to be logging that information as I mentioned with some experiment management tool or whatever you have in place so either of these approaches would work and then it says here cost and latency are typically interconnected reducing tokens and requests generally leads to faster processing that's correct the main strategies to consider here are reduce requests limit the number of necessary requests to complete tasks I see developers make a lot of mistakes around this you know these models are quite powerful and you can use them to achieve a lot of different tasks it really helps to think about the tasks that the model is performing breaking down the task potentially even running experiments separately with different prompts but in the end what you are doing is you're trying to figure out whether that model makes sense for your use case because now you have tested on different subtask and what you can do towards the end is put all of that together and now you have one prompt that achieve all task and so you're not making multiple requests to the model this is an important one so minimize tokens lower the number of input tokens and optimize for shorter model outputs as we know these models they tend to be very verose so by default they will generate a lot of tokens and that's not good and so what you will do is you will use methods to limit the length of the outputs you can do this in many different ways I noticed that structure outputs or the structured output feature from open air works really great for this because you only get the Json output you don't get all that additional context and explanation that's one way there are other methods where you explicitly in the prompt tell it to minimize the length so instead of generating paragraphs you would generate a couple of paragraphs as an example or even like sentences although the model cannot do this reliably it's still an effective way to shorten the outputs that you're getting essentially you don't need to Output a lot of tokens for the mold to be accurate in fact it's going to make more mistakes if it's generating way too many tokens so it avoids also a confusion and then it says here select a smaller mod so use moldes that balance reduce cost and latency when maintain accuracy so this is the point that they were making about using some smaller model as well to reduce the cost okay so there are some exceptions here and they mentioned that clear exceptions exists for these principles if your use case is extremely cost or latency sensitive establish thresholds for these metrics before beginning your testing then remove the models that exceed those consideration once benchmarks are set these guidelines will help you refine all accuracy within your constraints there are use cases again that will have some kind of high latency but in that case you will set your threshold what is acceptable and then filter out the models that don't meet that ly requirement but again this is being done after you have already run that first set of experiments so here's a practical example to demonstrate these principles they develop this fake news classifier with the following Target metrics and you can actually use like these classifiers as examples to test whether this frameware works for your models and your llm provider so I like the fact that they give you an example like this you can literally go and put together a very simple classifier example we have done that in our courses and this is why we use a classif for example cuz it's easy to put together example is not a lot of work but at least you get to apply the framework and it says here the accuracy achieve 90% correct classification cost spend less than $5 per 1,000 articles latency maintain processing time under 2 seconds per article then in the experiments they achieve the following goals so zero shot use gbd4 with a basic prompt for 1,000 records but missed accuracy Target so you can see here the zero shot 84.5 remember our accuracy is 90% and that was based on the financial implications and then few sh learning this is f shot prompting included five F shot examples meaning the accuracy Target but exceeding cost du to more prompt tokens you can see here in the table you have the accuracy but you also have the cost and this doesn't meet the cost Target because few shop prompts use demonstrations and use up a lot of tokens but it's not every use case some use cases like smaller Tex classification problems don't often use too many tokens but this is something to keep in mind as well and this is something you'll be measuring and then they also have latency here so it says find your model fine tune model GPT 40 mini so fine tune in this specific model we have a video on that if you're interested I'm going to link it in the description with 1,000 label examples meeting all Targets with similar latency and accuracy but significantly lower cost so you can see here this is gr accuracy so we have hit our Target we meet the cost Target as well this is awesome because it's much cheaper and in fact this is why I tell people often it is the case that a fine you model would be much cheaper because you don't include all of those examples in your prompt and so if you you have hundreds of examples in your prompt in your fuse shop prompt then you know fine tural will always be something that will probably reduce the cost significantly and then here the average latency is being cified as well so you can see here how we went from like this first part where we have these accuracy targets we have done a little bit of evaluation put together an evaluation data set we have this realistic accuracy Target as well that is based on the financial impact of those model decisions that's really important and then how do you as a second part then how do we optimize the cost and latency right we want to consider maybe model distillation or maybe we use the zero or few shot using a smaller model and so this is how you go about testing with these models and again keep in mind here that the model distillation is in most cases going to give you very good performance so that's expected and this is something that you want to experiment with you don't want to leave this out okay let's say in a scenario where in the fuse shot example or F shot prompting you know with a smaller model is giving you really good performance and you think that satisfies pretty much your goal but fine tuning a model on that will even reduce potentially the cost further and in some cases potentially even increase the performance so this is why it's important to have a Baseline and to have a way to compare and you can see here that this particular one where we use f shot is pretty much the same accuracy as this fine tune model you see here same accuracy but where we're getting significant savings is on the cost here and that might be quite impactful for your company so this is why it's important to show the experiments and the different comparisons that you're getting with the different experiments and solutions that you're trying out so in conclusion by switching from GPD 4 to GPD 4 mini with fine tuning we achieve equalent performance for less than 2% off the cost using only 1,000 label examples having a good framework will allow you to unlock not only the capabilities of the model but also give you a way to more objectively test with these models and select the right model for your use case so I really love this guide from open ey thank you to the team for putting this together I found it this morning and I just wanted to go through it in this video hopefully this was useful let me know what are your thoughts if this is something that you're already applying and what are your findings as well generally speaking this is what works best for us with the open a models so I think there's a lot of value in this