How to evaluate using LLM-as-a-Judge (Tutorial)

Key Takeaways

in this demo we are going to show you how to evaluate using llm as a judge metrics using opic so we're going to explain also the metrics such as hallucination and answer relevance that we're using to assess how good the performance of the LM application is at the desired task all right so we're going to run the first set of lines of code here which is just the usual Imports and again we're performing tracing for our application and then we are using the GPT for Mini model and here is the chb application the typical way on how to call the food chb for a response given a user input and this is just our evaluation task very similar again the code stays the same as before and the only difference would be this hallucination and answer relevance keep in mind that we are still using that data set right that food chatbot eval data set that we were using previously so the metrics here are hallucination and answer relevance and what are these measuring so the chop out generates responses based on those user queries and based on the information that's provided the information that's also provided here is the menu items and those menu items contain information that the llm should be able to use to respond factually to the user queries in a chat setting so Hallucination is is the metric that allows you to check if the llm response contains any hallucinated information and so in order to check for this you will need to provide the llm input which is this input the user query and you also need to provide the llm output as well which is this part here and in addition to that you will need to provide the context so this particular metric uses those Tri bits of information this is very different from the previous metrix in the previous metrix theistic metric such as lonstein distance for instance was using and comparing the output of the model and the reference in this case we're using these three and for answer relevance the idea is that this metric allows you to evaluate how relevant and appropriate the llm response is to the given input question or prompt so in order to assess the relevance of the answer you will need to provide the llm input which is again the this input right here and the llm output which is going to be this part here and so what the answer relevance does is it's going to mainly focus on the appropriateness and pertinence of the response rather than the factual accuracy of it which is what the hallucination one is doing so that's the of answer relevance and both of these metrics are based guess what on llms this is why these approaches are referred to as l LM based evaluation because they're using an llm or prompting an llm to do the scoring for both of these metrics and the experiment name is again the one that we have been using here and then we run the evaluation and the evaluation expects the experiment name the data set the same data set the evaluation task again this one here and the scoring metrics which are those metrics and then the config which you can pass any information that you like here we're passing just the model name here all right so now we can run this again takes a little bit of time a minute to two is what it takes for these experiments so sometimes it fails to compute the answer relevance metric and sometimes that has to do with the prompt itself The Prompt that's used to do this evaluation and sometimes what you'll need to do is you need to find a work around that sometimes you need to specify custom metrics so these metrics are not perfect by any means again because they are based on an llm but anyways so it finished here and you can see that it definitely failed on one so we're going to look into that the UI and why exactly it failed but you can see the hallucination metric here is 0.42 and answer relevance metric is 0.72 and that's the average score so definitely we're getting really nice score for answer relevance that's really nice to see and for hallucination we want to see that as low as possible so if it's one that means that's the highest and if it's zero that means we have no hallucination so this is really good to see already and again number of samples in 56 on the little report here all right let's switch over to the opic UI and see what went on here so here is the new evaluation or the experiment that was conducted just now so we can click on that and you will see here is the aage score that you saw in the terminal so these are the average scores and now we can take a look at some of these outputs here so let's try to take a look at this one so this one notice that it doesn't contain the answer relevance metric right so if we look at that more closely we can see that the Ed op was this so the menu does not list any gluten-free deserts specifically are there any offerings for gluten-free desserts and so we can take a look at the output of the model here and the output was we don't have gluten-free desserts but we do have delicious options like fresh Berry parfait so for some reason it wasn't able to provide a score for this and it raised an exception but as I was mentioning these metrics are not perfect by any means and our job now is to think about oh we might might need to actually develop some custom metric for this so that we can get more reliable results but this is still good we only got one fail so that means most of the other ones were working but we can quickly assess here that yeah it might have been the way the model was scoring because this looks okay there's nothing wrong with the actual output as far as I can tell let's look at an example here I want to really look at some of these examples they're really fun to see like for instance this one answer relevance metric so this one offers an explanation as well so let's look at this one is the vegan Beyond Burger a popular choice and then the expected output here is the vegan Beyond Burger is a popular choice on our menu with a popularity rating of four out of five looks like it's a popular one and the output of the model was the v b Burger is a delicious favorite rated for out of five featuring a tasty plant-based Patty and all the fixings plus your choice of fries so very interesting tone very enthusiastic and we can look at the answer relevance metric and the reason why this was scored so high this is very close to the highest score that you can get for answer relevance so it says the answer directly addresses the query about the popularity of the vegan Beyond Burger by seting it as a delicious favorite rated four out of five that explanation is actually coming from the llm and that's why it's referred to as llm based evaluation and then it says it provides relevant information about the plant-based Patty and fixings and so on so this is really cool because you can get an idea on why this was considered to be super relevant in terms of the response that we're getting from the llm and given the input and the menu information and for hallucination there's no hallucination but you can see the explanation here it says the output accurately states that the vegan Beyond Burger is rated four out of five in popularity so that's really awesome to see and we can go on and continue to look at the different results here specifically we might want to take a look at the answer relevance how we can get this higher and and then how we can reduce this as well so you will look at different examples where the hallucination metric is high and where the answer relevance is low and so that will give you some intuition into what else you can fix how you can iterate on your prompts so the idea is to gain insights as you do this evaluation and you continue to run more and more and more experiments that's the idea behind this type of experimentation that we're doing here we are using the concept of llm as a judge is very easy to do using opic