Evaluation Primitives | LangSmith Evaluations - Part 2

Key Takeaways

hi this is Lance lanching this is our second video on Lang Smith evaluation I'm going to walk through kind the core Primitives or the basic components of lsmith to give you a sense of how to kind of build up your understanding from scratch so in the pirate video we talked about kind of landscape evaluations and the things you a want to do in terms of these four categories building data sets implementing evaluators um and then applying evals and of course specifying a task you care about so let's let's actually kind of build this up let's say have an LM app like have a task and rag is a very popular use case there is an index you do retrieval and then you do generation based on this the retrieve documents to produce an answer so it's question in answer out now the single unit of work for each of these steps we call a run so in this particular case there's two runs here one is question to documents and then documents to answer so it's indexing or retrieval and generation those are the two core steps so to speak SLE units will work runs can be tagged they can you can have feedback to runs you can have metadata to runs which we're going to talk about later now runs are rolled up into traces so traces just collection of runs um and for example in this particular case the rag app Trace would have the two runs the Run for the retrieval step the Run for the generation step that's kind of it and then a project is just a collection of traces for given app so let's say I had this rag app it was in production somewhere a bunch of users used basically used it um that all those traces would this be logged to that particular project so that's really it so kind of again I have an app it's a rag app it has two steps each step is a run the app going from you know input to Output is a trace that Trace contains the two runs and every time I run that app all those traces get rolled into into a project that's kind of it so those are the kind key key components of like of kind of the task component or the app component of this four you know these four pieces that we've been talking about now data sets here we kind of talked about the flavors of data set you might want manually curated from logs or synthetic but really at the core data sets are pretty simple they just have an input and optionally an output um so in the case like if you're doing a rag app you might have a data set with question answer pairs in that case there is an input and an output each of the pairs typically doesn't have to be is just like a for example a dict or a key value pair like question um your question answer your answer um now that's an example and the data set is just a collection of examples and as I just mentioned a lot of times the input outputs are just like key values you know so in the case of rag question answer you know key value pairs for my input and my output um there are other types of data sets that are interesting though chat and Ln in particular these are a little bit more relevant if you kind of want to do things like fine-tuning in which the data set requires a particular format like for for example chatter Ln format for a lot of applications we'll talk about key value is is completely fine and again your data set composed of examples each example just has an input and optionally an output that's really um and now evaluator so what's happening here we we kind of talk through a few different flavors here there's different kinds of Judges you can chw so like LM as a judge heuristic judges like function the human judge and then different modes like comparison comparing two different traces for example two different Generations um reference free so like there's no ground truth you give an llm judge a bunch of General criteria um or if Human been judge General criteria and then of course you can also provide ground truth so are kind of like the things you can think about in terms of evaluation now here's a very important piece where these all come together so like what's the information flow when you run an evaluation what's the evalu really doing so again our data sets composed of examples each example is an input and an output when you run an evaluation you take that input and you Plum it into your application so let's thing about a rag app for example the input's a question we Plum that into our rag chain we get some output answer that's that that's what you see on that bottom uh kind of in red now also you have your ground truth answer as an output in your data set that also gets plunged through to the evaluator so the evaluator which is kind of very intuitive has access to a grou answer in the case of this rag toy example and the Ln answer your rag chain answer and then it's doing some judging at the two now this is where there's a lot of different interesting options here you can build a custom evaluator like we talked about you can also use a bunch of off-the-shelf lsmith evaluators to do the grading so in this particular case we have an answer from you know our LM we have a ground truth answer and we want to use a built-in evaluator we can use one of the evaluators that are label Centric so they are like tuned or or designed to do uh comparison between like an Ln output and a groud truth output and we'll talk about the various options later but I just want to give you a flavor of really what's happening here is you have your data set of examples you're passing the you know the expected in this particular case example to your judge um you have your app which is receiving input from your data set it's producing its own output so now you have an output from your app you have the ground truth output you pass them to a judge a judge make a decision and outputs effectively a some kind of score that's really it that's the big picture for how to think about the information flow when doing evaluations and that's really it you know for like the core Primitives this is the key thing to understand and once you understand this everything else is kind of is pretty easy um so we're going to be talking about a little bit more detail and and show some code examples in the next video thank you