Model CI/CD Course: Setting up LLM evaluation

Key Takeaways

the model that we just trained is still being evaluated it's generating samples for our evaluation data set so this might be a good time to take a look at our evaluation code in um this case study we will use the concept of llm as a judge and that means that we will provide an advanced llm like GPT 4 um the instructions that we need to that we need to respond to and we will provide two responses one will be from our Baseline model this is what uh what we are evaluating against and we're trying to improve on top of the Baseline and we will provide the instructions from a candidate model and because we are using model registry we'll pull both the the Baseline and the candidate from model registry based on aliases so let's take a look at this code as you can see we are um starting with with a configuration object and we're uh providing the model that we will use as a judge so in this case is gp4 specifically version um o.6 uh 13 uh we're providing it with a system prompt and the system prompt says you will be presented with a choice of model A and model B response for an instruction the response should follow the instructions and use the provided context if there is some and we are using the chain of Chain of Thought prompting so we're asking the model to think step by step which respon respon is better provide the reason why and then asking it to answer with a choice if Model A or B or equivalent is better um we are uh providing it with a baseline the candidate model names that we will use to parse our our data we can limit evaluation to certain number of samples we're again providing uh our weight Andes project entity and again for your use case you may want to change that entity to your uh either personal entity or the team that you're working in um and uh we're also providing it specifically the aliases of our of the model we are evaluating so in this case it's our candidate model and the Baseline and this is something we need to make sure exists in the model registry so we have models that are assigned both Baseline Alas and the candidate alas um we will be using instructor to get structured outputs uh from our um from our from our llm from our evaluation llm and I want to highlight here um a great course that we have on getting structured outputs from llms uh that you can also access on on weight Andes courses and there you can go deeper into how instructor uses pantic to get the structured outputs in this case we're collecting two fields from our llm one is the The Chain of Thought which is actually the reason why the model selected uh Model A or model B and then the choice which is either a b or equivalent then we have the about evaluator class which uh takes in config um it patches the the open AI client with instructor so that we can get the structured outputs out of the client um and then we're providing it the the is the the instructions and the responses from both models the candidate and Baseline um when uh using LM as a judge we need to be careful about some of the biases that LMS has and one of the biases is position bias so sometimes um whether a response comes first or second that might influence the choices of an llm and for that reason we are shuffling the answers randomly so with enough samples like we will will be able to avoid that that position bias so shuffling the answers we're giving the uh the llm um the the system prompt that we we already reviewed and the and the input which is the instruction the generation from model a generation from model B and uh we're asking it to respond with with a choice and then um the there are different metrics that you might use with llm as a judge When comparing to models one classic metric is Win rate but um I find I find win rate sometimes uh difficult to interpret especially if you account for ties between models um you need to do certain estimation on how that win rate should be calculated and I like simple metrics so the simple metric that we will use is whenever um Whenever there is a tie we will return zero score whenever the Baseline is chosen the score will be minus one this is the the score for the the Baseline is minus one whenever the candidate is chosen as a better response we'll get a score one now if we average all of the scores across our evaluation data set we'll get a metric which we will call candidate preference score and if that metric is positive then that means that the candidate is better than the B Bine if that metric is negative that means that um the Baseline is better and that metric goes between minus one and one in the case of minus one all of the responses in our evaluation set were better for the Baseline in the case of one all of the responses of the candidate were better so it's a it's a pretty good metric that is very easy to interpret the higher the score uh the better the candidate so you can make certain um you can put certain certain rules for examp example if the metric is above 0.5 that means like we're always switching the candidate for the Baseline if it's around zero like it means think maybe it's borderline maybe you need to do some more some more F uning or get a better candidate and if it's negative that means strictly that the Baseline is better so that will be the the metric that we'll use and and we'll send off all of the responses to to the to the llm uh via the the API client will um gather the response and and calculate our candidate preference score and log it into weights and biases one thing I want to uh I want to do here and highlight that I I like to do personally I think it's a good practice is um as you store the the evaluation uh results in weight and run I'd like to add lineage specifically to the to the both the candidate uh and the Baseline model and update this in the config so you will see that I'm I'm using our um our artifacts API I'm calling wb. use art uh artifact with the with the evaluation baseline from from the model registry and then I'm taking the path of that model and saving it in my config as evaluation model path and I'm doing the same with the Baseline and that will make sure that whenever I look at the results from this evaluation I understand exactly which model was evaluated against which Baseline so I'm downloading the table uh from from each of this this models the evaluation the Baseline merging the table passing this to our evaluation that we reviewed uh then saving the results into into a results data frame calculating our candidate preference score and again positive score will mean that our candidate is better and then also saving the results into W and bu table that we can inspect uh logging it into a weights and bies both as a table and um and also storing a a CSV file that we can can use locally if needed so uh our our evaluation run is almost finished so in in the next video we'll take this evaluation script and we'll try to set this up for for running in an automated way based off model registry triggers see you there