Custom Evaluators | LangSmith Evaluations - Part 6

Key Takeaways

hi this is lash BL chain this is a sixth video in our series um focus on langth evaluations so in the first video we just kind of laid out why are important the second video we laid out the core Langs Smith Primitives we then talked about how to build data sets um first developer curated we built one for the datab bricks um The datab Brick model which is discussed in this blog post so we built a question answer data set um and we showed how to edit that we showed how to append um example to it we then also showed how to build data sets from user logs so if you like an app in production you're you know getting feedback questions from user you can basically Plum those into a data set um we then just talked through how to use an evaluator uh in our particular case uh built-in life with evaluator to judge our app which we built here so we a very simple app that answers questions about a given context and Es case be plunged in the blog post and we're using G GPD 35 to answer questions we're comparing those GPD 35 uh Generations or or answers to our data set which was manually curated so that's really it we used uh a built-in L with evaluator called coot QA which is very convenient for question answering so that's kind of where we are that's what we did now let's build on this a little bit and this kind of gives you a summary of what we just did now let's say I want to do something different I want to find my own custom evaluator so let's say I want like a simple assertion that act an answer actually was generated or not like was there some kind of like um you know failure in in in generation of the answer so that you know to turn some an empty string or or you know uh there can be other reasons why and L1 Sals to produce a generation right so in this case I'm changing it up a little bit I stole my data set a question answer pairs dered from that data post but in this case I want my graders to be it's like I you're the hardcoded decision it's just like is there an answer or not right it's reference free there isn't like a ground truth there it's just like is there an answer or not right so it's more like criteria based uh grading in this case you can this is where it's a little bit squishy because you think of this more as a unit test it's a very simple assertion for functionality um as opposed to it being an evaluation which is using like an LM as a judge or human as judge this is just like a quick test is there an answer present or not and of course this can be used you know in offline evaluation potentially like Ci right so it's like a nice test or sity check that my system's actually working so how would I do that so it's actually really simple this little function is actually all you need but let me actually explain a bit about what's going on here remember our flow diagram let's actually go back up so there's this run and there's an example so the run is my app basically my app as some output right so the Run has this do output which is basically the the dict or the object that comes out of my app and then likewise my example which I pass an evaluator also has this dot outputs which is basically everything that's available to the example in my particular case what I'm going to care about is you know in typical evaluations I often care about just the answer but the point is uh when I'm doing evaluation there's this example object and if this run object and these two things are being compared in some way so that's really it so remember I have an example and a run now if I build my custom evaluator I plum in that example and run just like we just talked about and I could just easily fish out from the run which is my llm app I want to get the answer now let's actually go all the way back up and make sure we understand what we just said so I'm going to go back to my app here's my little app answer data ver question open AI it returns a dict with this answer key that's where me get that answer thing from so that key depends on my app in this case my app returns you can see it's it's a dict with answer in it that's it so that's why in my custom evaluator when I go down here I just go to my outputs from the run again it runs my app the outputs contains everything output by the app in that case that output is a dict but has a key answer so I'm fishing it out like that and I do something really easy if there's nothing there then it's not answered otherwise it is as simple as it gets remember I have this data set dbrx I'm going to run on this so again here's my app basically we already talked about this here's the data set we already showed how to build that this is my new little evaluator is answered um I'm going to create a new prefix for it uh it's some new metadata kick that off that should run pretty quick and I can go over to my data set it's already done so okay this is kind of interesting so now you see two evaluations the first one we show previously using the built-in uh light chain evaluator now I had this new one you can see it has this new prefix and let's open it up and see so in this case those scores are all one because there is an answer every time that's it that's a custom evaluator it shows up here in Langs Smith this scoring is arbitrary we can choose different means of scoring but in this particular case I just chose one or zero shows up here that's it pretty simple um and this is extremely flexible I mean this is just a function it can be anything you want again all you need is the ability to fish out your output you could do string matching could do arbitrary different kinds of evaluations and you can also include for example llms in this to do arbitrary reasoning over your answers and produce scores so I've used this a whole lot it's extremely convenient very flexible and building customer values is extremely simple particularly once you understand that kind of that information flow we showed here all that's happening is I have an example I have a run each of those has this outputs thing which gives you access to in the case of my run that output object in my case it was just a dict with answer and this do outputs actually gives you access to everything in this example that's really it that's all I need to know um and that's really all all I want to say about building custom valuers it's very powerful and very general thanks