Eval Comparisons | LangSmith Evaluations - Part 7

Key Takeaways

hi this is L Lang chain this is the seventh video in our Langs Smith evaluation series um so our first video gave kind of a context as to why EV vals are interesting and important the second video talk about Lang Smith Primitives our third video showed how to create manually curated data sets we built one based upon this datab bras blog post um the fourth one showed how to build data set from user logs so if You' have an app in production you want to kind of capture user questions and create a data set from them you can very easily do that we talked through that um we then talked about various judges for data sets so different types of valuators we should how to build use a built in Lang chain evaluator um for question answering we applied that to our data data set um and we just talked through custom evaluator so again we've kind of showed this this flow diagram and go to go to those videos if you want to kind of Deep dive into those topics now we're going to have a little bit of fun so this is where you know you kind of get into like very real world use cases and and needs you often want to do comparisons so let's ask a really practical question how does mistra 7 be running locally on my laptop compared to gp35 turbo on this little challenge we've set up again remember we've four question eval set on this data bre blog post p and open L undo versus uh gbd 35 turbo so just a little note here I'm using Alama for this um just you can download it going to uh uh ama.com you can do Alama P mstr to get the model and you can kind of follow instructions here um so here's my setup I'm going to create a new function that does the same thing that we already were doing with open AI but here I'm going using MW this is running locally on my laptop so again it just I can answer ask and answer questions about the particular blog post so I answered I just asked a question and here we go so you know the answer is streaming out very good and it's obviously slower than open AI exactly what we would expect but we really care out here is I want to know about quality how does it actually compare on this little challenge I built for myself so what are we actually doing uh here we a developer curated data set of four examples on this data bricks block post um I'm using LM as a judge again remember the built-in coot QA evaluator I'm using um and I ground truth answers for every question I'm doing an AB test between gbd 35 turbo and mraw running locally on my laptop so that's the setup um and it's pretty easy so remember we've already built or defined this data set dbrx we've already used this to evaluator coqa so that doesn't change at all all that changes is I'm just passing in this new function now now let's go back and look at that function it looks exactly like our old one a few little twists I'm using ol instead of open AI that's really get same output object answer you addict with an answer that's it simple we just saw it work here so what I can do is kick off the SE valve this will take a little bit cuz it's running locally I have an M2 Mac uh with 32 gigs by the way so that kind of gives you some sense I've heard a lot of people having good results using Mr all 7B on on Far smaller machines though so it's a really nice open model to work with if you want and you can see it's still churning it's streaming its answers out here it's actually done so it didn't take that long it ran against my four questions now here's where it can get really interesting let's go over to my data set now I can see here that there's three runs so this is our initial run uh experiment you can think of with open AI this is that second one to do with a custom evaluator we're not interested in that that was just kind of a quick more unit test that wasn't a proper kind of LM based evaluation and now here's our latest one so here's where the gets really interesting I can click on um this and this so my mraw run my open eye run and I can click this compare thing that opens up this compare mode and you already see some nice statistics here so what I can see is the average score so the first run which was up in AI indeed does quite a bit better in terms of score latency as exactly we would expect M draw slower by quite a bit um and here's a latency dis and so forth so you get some immediate statistics about the two runs now here's where I've done a lot of work in the past and you know this is kind of the Crux of AB testing that's extremely useful um that's why it's very helps to do this inside Lang is all kind have captured for you managing this all yourself is can be quite painful here's my first question here's a reference output here is the output returned by mraw here's the output returned by open AI so I can actually look at those in detail I can like kind of zoom in look at the answers and like hey they they look very similar here that's really cool and you can see my greater also assesses them both as correct and again we talked about you can actually click on the evaluator run for each case to see why but they look good now here's where it gets a little interesting it looks like my mistr running locally is is did not receive a correct grade um and I did so let's actually look at what was the question what is the context window of the drb ex Dro model okay so it's 32k token right what did mrra think uh cont is oo 2048 so that is definitely wrong and we would have to investigate as to why Mr believed that to be the case it could be you know there could be many different reasons why I failed for that one but indeed our grader is correctly grading that as as a as a wrong response for fun we can actually dig in and look at uh that particular grading trace and we can see why stud's answer is incorrect the student States at the context window is 2048 um the Contex says clearly 32k there you go so the grer is doing the same thing and we can kind of go through each one so this is like a toy example but it shows you a very important useful concept of comparative AB testing so like you might want to compare different prompts different llms and this is a very nice way to do that you can see it's extremely simple we've just supplied our data set name um so we're of course run against the same data set where you know typically I like to apply different experimental prefix to to enumerate the different experiments and running so that's easy you can also capture that in metadata by the way so that's another way to kind of differentiate your experiments um and I'm using the same grader of course and I'm just modifying the my function which in this case was just hey swapping out mistol uh swapp swapping in mistol for for opening I so again this just shows you how to use this compare mode and limith to do ab testing really nicely um in this particular case we're comparing mrra versus open AI we can look at you know kind of overall run statistics as well as granular answer wise differences we can inspect the greater as shown here we can look at the runs as shown here so this gives you a very flexible General place to do ab testing uh across different parameters in this casat I use different uh different llms um and I've used this all the time for lots of different kind of projects and it is indeed quite useful it's very nice to have this all kind of managed in one place for you um so we're going to be kind of diving into some uh deeper themes after this this is kind of the final video of our like kind of introductory Concepts so if we kind of Zoom all the way back out what did we talk through we just talked through man building your M Cur data set in this case running Elm as a judge against the ground Truth for AB testing so we kind of went through that we also had talks through um you know same setup but basic just simple unit test using custom evaluators we talked through um yep we talked through yeah different data set creation yeah we talked through here we go Len is a judge with ground truth um so for like you know uh just just you know Len is greater evaluation um but no AB test so in that case you're just looking at like a single a single model and evaluating it using LM as as a judge um we talked about the information flow for valuation we've talk about different ways to build data sets from user logs from manual curation and that's really it this gives you kind of like the groundwork you need to do a lot of different things building custom evaluators and AB testing is frankly covers a huge surface area of use cases uh and we're going to do some deep Dives following this so stay tuned for for additional videos thanks