LangSmith Highlights: Human Annotation

Key Takeaways

one of the things that we help you do in Lang Smith is ADD human feedback to annotate your runs so we just showed how you can use automatic evaluation to have llms grade your runs or to programmatically Auto evaluate each of your runs but there's really no substitute for a human adding annotation feedback on runs as well you might do this for a couple of reasons maybe there's some kind of measure that's hard to have an automatic evaluation on or maybe you've used auto evals on thousands of runs and you want to have a human just pick through a small sub subset of those runs to make sure that your llm grader is is still doing a good job so I'll show how to do that in this video so this is a test run we have uh some feedback already recorded on each of these runs things like correctness helpfulness and sensitivity uh as well as uh embedding cosine distance and what we're going to do is we're going to pick all of the RS that had a correct score uh and we're going to grab them all and send them to an annotation queue so we're going to add this second human review uh annotation cue and now all of these runs will be queed up in this way that we can easily go through and add our own own feedback and we can we can see here all of the tags this Ron already has but maybe we want to have a different kind of feedback like creativity which is harder for llms to creade and it can have a score of 1 to five and just making this up but this one is a creativity of SC two maybe you have a rubric that a human evaluator wants to follow that this one is done I can now add a score of again creativity to this run here we'll give it a score of five this one is done and you can see how I can quickly add tags at to each of these runs and add some additional feedback manually uh to to the ones that are in my queue so that if I'm uh supporting a a flow of making sure that each of the runs have a good response uh you can do that pretty seamlessly within your annotation que and we're all cut caught up meaning I have no more to review and this is really helpful if you're in a supporter role or you're helping curate data sets uh to make sure that you have the appropriate tags and feedback on each of your runs