Benchmarking RAG over LangChain Docs

Key Takeaways

hello all right today we're going to go over how to Benchmark some retrievable algorithms using the new uh Lang chain benchmarks package um as well as the public data set that we're releasing um specifically the public data set is for question answering over Lang chain documentation um and so we've provided uh about 86 examples of inputs and expected outputs um and as well as some utilities for doing some basic kind of like in question um and then we've evaluated a bunch of different methods for doing retrieval and then retrieval augmented generation so generating answers to those uh questions we can then kind of like score each retrieval or generation method we'll we'll kind of like change up both pieces we'll score that against the ground truth and see how we do and so we've done that on a bunch of architectures so far um and what we're going to do in this video is we're going to try it out on a new one uh because part of the the the value in open sourcing this is that it can serve as a way for people to Benchmark you know all the different retrieval algorithms all the different models that are coming out and see how it actually does so we're going to be working off of this uh this notebook right here I'm going to I'm going to basically be copying some parts over to my jupyter notebook running it changing some stuff up because uh I want to change stuff up I want to evaluate different things specifically the thing that I want to value8 for this video is uh some sort of like multi-query um query expansion type uh of retriever so in typical rag the question comes in and then you look up five or six documents relevant to that question what I want to do is instead expand a single question into two or three or four different questions look up documents for all of those and then answer based on all of those documents the idea behind this is that when you're looking things up it might be better to break down the the overall user question into distinct sub questions look those things up and then respond with an answer over all of those talks so I'm going to try implementing that and then benchmarking it so I've got this I'm going to copy some of the code over here and install everything that's needed I'm then going to set uh some uh environment Vari Ables um so let's copy this um I need my Lang chain API key so I'm going to go here grab my API key from here U if you don't have access to Lang Smith uh shoot me a DM on Twitter that's probably the best way and and I'll get you an access code um I'm going to do this in a little bit but I'm going to do it along with the other uh environment variables I'm I'm going to do that I'm going to pause when I do that so that you don't see uh my screen and my secrets there the other environment variables I'm going to set at the same time are these I'm not actually going to use hugging face I'm just going to use open AI for everything so I'm going to set my Lang chain API key my anthropic API key and then my open AI API key let me do that and then I'll come back okay we're back I set my API Keys now I'm going to follow this code so I'm going to use uh the Lang chain benchmarks code to check out some we have we have a variety of tasks in here the one that I'll be doing is this Lang chain doc Q&A task we'll we'll we'll go into more details on some of the other ones um but this is the this is the main ready one as of today um so if we take a look at it I can see that there's a data set ID I can click on this this is a public data set that we've published under the link chain org um so you can see that there's 86 different examples um and then there's also a bench of test runs that that we ran um along with some scores for how well those did so once I've done that um or I haven't done anything yet I'm just kind of like looked at what everything's is now I'm going to clone this public data set this is going to move it in to my link Smith account so now if I click on it this is in my personal link Smith account um and I have no tests on it because I just cloned it in here but these uh inputs and outputs are all the same from uh the data set that we collected so if we take a look at some of the questions you know we can take a look we see question will this work from Lang chain. chat models import chat open AI llm equals chat openi model equals CLA 2 and then the answer is here no Cloud 2 is a closed Source model developed by anthropic the chat open AI class interfaces with open ai's chat model so these are um so there's 86 examples some of these have you know simple answers like yes no other of these have longer uh natural language output so once we do that um we're now going to set up the retriever I'm not going to change too much about the retriever um in in this video I'm going to use the basic retriever the only slight difference that I'll do is I'm going to change this from hugging face embeddings to open Ai embeddings and then this stuff uh so so this is loading some of the docks that we're going to use and then this is creating the retriever um this abstracts away a little bit of what's actually going into creating the retriever it's a it's a kind of like handy method for uh just creating a simple retriever if you want to change up some of the logic of this um the Logics in the the uh Lang chain benchmarks code we'll probably try to make this a little bit more apparent in the future what exactly is going on but for all intents and purposes this is creating a really simple Vector store retriever so let's run this and it'll do some work while it's running we can take a look at the next cell so here's where we actually set up the chain that we use um and so let's take a look at what's going into this chain we've got this format format docs function this takes in a list of documents and returns a string so it's basically formatting the documents into some context that we can pass into the language model this could be one thing that we play around with we could play around with kind of like the the tokens used to denote um or pass in various metadata or Source or content um this is definitely something we can play around with this is the prompt um again this is something we can modify right now it's pretty simple you are an AI assistant answering questions about link chain respond Solly based on document content and then the human question we're going to be using uh the anthropic uh new model that was just released um and then uh the this is the so this is generating the final response it takes it's the prompt llm and then it just turns it into a string um it turns it into a string because the chat model returns a chat message by default and then if we look at the the the chain um we can see that we are adding in a new variable um this variable is called context we're getting context by taking the question passing it to the retriever passing the response to format docs this is getting a string so this is assigning a new variable called context with type string and then it's passing it along to the response generator the response generator takes question and context so context is coming from the retrieval questions getting passed along so runnable assign just adds in a new variable so it passes along everything that's in it up to that point we can see that this looks like it is done so let's um so let's copy this code but we want to do some changes specifically what we want to do all of this looks fine um all of this looks pretty fine what we want to do is we want to change this from taking the question passing it to the Retriever and then and then just formatting the documents we want to take this question generate a bunch of sub questions from there we want to uh uh look up all those questions with the retriever combine all the documents put them into a single string and then that's what we're going to use as context so um what we're going to do to generate those sub questions is we're going to go here and this is we we call this there's there's two different ways that we kind of refer to this we call this multiquery or query expansion so I'm just looking in the prompt tub to see if there's any prompts that I can use to kind of like help with this query expansion doesn't return anything if we do multi-query we can see that there are a few different prompts um including one that that I uh that I uh created when I was uh checking this out preparing for this um let me um let me try to delete this um or I'll just work with it basically if you want to work with your so you can see that this is forked directly from Jacob multiquery retriever what I what I don't like about this prompt is that it's asking it to generate different versions of a given user question and I really want to generate more like sub questions so if I try this out in a playground if I pass in I also don't like the fact that I have to specify query count here I want to like dynamically choose how many it should generate but let's just say there's three and then let's say say the question is like how do I use open AI with chroma DB um let's pick a model um let's pick yeah 3.5 turbo seems fine um so if I look at all the questions that are generated they're um they're not they're all basically like variants of the same question which are exactly what the template asks for what the prompt asks for but I really want more specific things like I want something like how do I use open how do I use chroma DB it can look up things for both of those and then it can kind of maybe figure out how to use them together along with the original question or or some rephrase of how to use them together in case there is a document just around that so basically I want to modify this so what I've done um actually yeah what I did was I clicked the fork button and then saved it as my own um so now what I'm going to do is I want to modify this a little bit um so also I think there's some weird entation in this prompt so I'm going to say like I want to generate one to five different sub questions or alternate versions of a given user question to retrieve relevant document from the vector database by generating multiple versions of the same us question by generating sub questions you can break down questions or concepts that you can break down questions that refer to multiple Concepts into distinct questions this will help you get documents for constructing Final Answer um yeah I'm fine with XML tags so now let's try this out how do I useb together um let's use chat openi for this let's start seeing so it's still kind of focused on how to use them together um if uh if multiple concepts are present in the question you should break into some question one question for each concept um let's try this out that's amazing okay so that's what I want I might have like overfit slightly on this example that's fine there's a lot of prompt engineering that goes into things like this we can evaluate this and see how it does that's the whole point of this video um so let's commit this um to there um if we go back here cool it is updated um let's copy this because we're going to use this in our code code um and then from model chat that's fine oh yeah don't want to do that quite yet chain let's call this Q gen equals um question Cool okay so it's generating each of those um so so I also want to probably keep around the original question as well so I basically want to take the original question question generate some documents generate some code questions get some documents for those look them all up comine all the results fine um so now I need to add in something that basically splits this apart um oh I also want to add in a stop sequence um because I want it to stop when it uh this is just going to force it to stop when it generates that to now let's see this um okay cool um I don't like how it's doing one two three four um so let's go back here let's open this in the playground um let's uh let's wrap these in quotes and see if that fixes some of it to switch this how I okay so that seems good so so wrapping in quotes um i' still uh quotes are going to be like a little bit annoying to parse out so let's just do bullets and okay cool um oh yeah you didn't see there okay so that's so I don't have the stop token here so it kind of like runs on and creates two blocks which my stop token it'll save that so let's commit this now if I pull down this new thing cool I get back some bullets um so what I'm going to do now is I want to basically write some output parser or some parsing logic that's going to split this into a list of questions um so let's do string um I can first I want to uh let's get rid of this so let's do text text strip um questions um let's do questions equals uh text. split new line Dash space return question let's add this up here let's now type this in to split let's run this okay so it's getting one so we can just do um it's getting one like weird thing there so we can just do return Q for Q questions if Q I think this actually works cool so now I've got a list of questions um let me just strip that andri that I will get rid of the the new line that I saw there okay so this get a list of questions now what I want to do is I basically want to uh uh create this okay so I have this Q gen this returns a list of questions one thing that I could do is basically just Q gen um Q gen retriever map so what's going on here and and actually let me let me combine this into retriever chain equals Q gen this is going to take in uh uh question return a list I'm then going to do retriever do map this is going to take each element of the list pass it to the retriever so it's going to take a question pass it toet take second question pass it to the retriever then I need something to flatten this list of documents or format docs so I have this here um what I'm going to do is I'm now going to create version that is format list Docs um this is going to be a sequence of sequence is because it's getting here it's each element that retriever returns is a list so it's going to get back a list of lists um and then uh I'll do doc lists for docs in Doc lists that that um so there's going to be some weird things because of the index I um so what I'm just going to do is I equals [Music] zero for docs for Doc and docs I plus equals one cool um and then let's do format plus Docs this is a retriever chain let's try this let's see what happens so this should get back a really long string because it's going to have all the docs concatenated so yeah we get back this really long string cool we're using CLA 2 because it has a really long context window so this is still a really long all right so we're getting a lot of context back maybe this is good maybe this is bad this is why we have evaluations um so let now we can do retriever chain um and I don't actually so so this plucks out the question but I don't even need to pluck out the check question I can just do like this um and let's wrap this and so this is just some nice config to make it show up nicely in in traces um but I can just do this this is going to take everything pass it to retriever chain retriever chain expects uh a dictionary with a key question so that's perfect um cool okay so this is my new modified chain let's try it out on this question if we're curious about what's going on in the hood we should be able to see things in our uh we probably haven't set tracing equals to True okay so let's actually do that before we go any further because that'll be very helpful Imports um if I want to have things logged here I can go to setup I need to set this equals to true I wonder if this will work if I have to restart the notebook if I have to restart the notebook then I will not do that um and I think I may actually have to um yeah okay so it's not working the the the tracing isn't working because I haven't uh set the environment variable before that's fine um would have been good to do so I could see what's going on but it looks like the response that it's giving is is it's reasonable um actually I think it would be really cool to see what's going on so let me pause the notebook or let me pause the video restart the notebook with that environment variable set and then we can see what's going on under the hood because I think it's going to be really cool okay so it was actually uh running before it just wasn't logging to the project I expected it to be logging to because I had some other environment variables set from before so I changed it up I'm now logging to uh dedicated project for this multiquery eiles if we click into what was just logged from this run um we can see a few things that are going on um so we can see that there's this step which is basically going to be the retrieval step um that has this format docs thing and then there's this chain that has generate response and this is where it's generating the response so this is the retrieval this is the response the retrieval is using this multiquery thing so if we look in here this is the prompt that we had um it's taking in the question and generating uh uh two sub questions to ask we're then parsing that and then there's two calls to a retriever here so for in each of these we can see basically the documents that are returned I'm actually curious whether these are the same or not so how-to cookbook linkchain expression language introduction how tobook linkchain exression language introduction so these are actually returning the same things um one thing we could and should do is D duplicate these in format list of docs um I'm debating whether it's worth doing that live or not uh probably will'll leave that as an exercise left to the reader um but then that's all pred into this anthropic model um and so you can see the very long system prompt here because it has lots and lots of context um and then it returns uh this actually I I probably will um spend some time D duplicating it so we can see here um that uh okay so I think what I'm going to do is I'm going to add this um for do hashes um I'm not could be a set to be faster but not too worried um I'm just going to Hash the content for now I could hash everything actually I'm I have the doc string actually wait yeah this is going to be uh way easier than I thought um if doc string not in formatted docs then I'm going to pend increase ey okay so now if I rerun this hopefully I shouldn't change the answer too much it actually should just be completely D duplicating um things and I can peek out what's going on under the hood this again runs in real time so I can kind of see what's going on um still pending a bunch all right there we go I was just debating whether to pause the video or not so good thing that it went through um so now if we look at what's going on we can see that the retriever oh okay so it's changing it up it's actually generating uh three sub questions um so it's generating uh yeah three sub questions now I don't have temperature set to zero so we are seeing some Randomness this is a cookbook and then in cookbook introduction how two L expression language cookbook introduction how two L expression language okay so it's getting the same docs if we now look at format list of docs um this is should be shorter than uh let's make this wider what we can actually do to see if it's shorter or not is we can look at some stats so in this final thing um if we click into here um we can look at the uh input um and we can see this input here we can look at some of the things on the side so this is right about 10,000 total tokens if we go back to the previous one and we look at everything this is at about 6,000 total tokens y this should be duplicating why is it not duplicating um it is not duplicating because we have the i in here okay so what I'm going to do real fast is just look at okay so I do need this then I guess because I have the ey in there actually um yeah let's change this up let's change this up a little bit let's remove this for now and this for now and then going to do something like this thing for I and enumerate form a docs remove this remove this um so now I don't even need ey so basically the I was causing things to be different that was the only difference so now I can just format them you duplicate them along the way and then add in the I later on so if we do this we do this we do this we do this now this should result in fewer tokens so let's take a look it's still pending it'll take a little bit awkward pause in the video there we go and if we take a look at this trade we can see here it only uses 3,500 tokens yay we've successfully duplicated uh the documents so that's going to save us some good tokens um as we as we start using this all right now we're going to run it over the whole data set and get some evaluations so let's copy this code we'll put this here let's copy this code we'll put this here and this is running on the data set and we're getting some progress of what's going on we can take a look at it in Lang Smith and we can see that we're running things this might actually cause some errors because I have some rate limits on anthropic that are not super high regardless this is going to take a while to run so I'm going to pause the video If I make any changes I will restart the video and say what I did but if not I'll restart it when this is finished running because this is going to take a little bit and I don't want to sit here for a while all right so we're back if we look at our notebook and finished running we get some nice um metrics from our uh uh evaluators that we're using so we calculate the embedding cosine distance the the accuracy score that's uses in llm um and then a faithfulness score as well um and so if we go into here now we can see that we have one test run we can click into it and we have a uh we have a bunch of data points we can look at it what class type is returned by initialized agent the right answer is the agent executor class this gives a more lengthy explanation but it still says the agent executor class we can see that nice it gives a string score of one so this is using the llm and it's recognizing that this is the same as this technically you know it's not exactly the same that's why we have to use an LM the eming coine distance is um I think that's seems to be yeah okay so that's higher than usual so that's not great so so the main metric that we look at here is this score string accuracy one um we can do some filters so we can look for so let's add uh less than5 so these are ones that are actually let's even make this worse less than3 let's try to find ones that are really bad what does on tool start mean so this is a method that is defined on classes um uh that works with callbacks um nothing in the docs that that we could find if we want to figure out like so this gets a bad answer because it doesn't give the correct answer um at least it's not making up an answer um so we should uh uh you know we we can add other metrics to calculate whether it makes up something or whether it just I think that actually might be what faithfulness is um one thing we can also do is if we click on this run um we can open the run and we can see what's going on so we can see everything that's going under here we can see that we have this multiquery thing that generates like what is the meaning of all to start can you explain the definition of all tool how would you define the term on tool start we can then see the retrievers that are used um these are returning slightly different things Okay cool so we're getting um some differences we can then see the model that's being used um and uh the output there if we go back overall we can see some overall metrics so we've got a faithfulness of77 and an accuracy of 63 if we compare that to the original public version of the test set we can see that accuracy um uh I don't think we have we don't we didn't run faithful enough on this one so if we look across accuracy this is actually crushing all other accuracy metrics um point so 63 that's higher than the best one which was the 61 so 63 is you know and and I I barely did anything here I think this actually speaks a lot to the power of the multiquery or this query expansion thing it's really really powerful um there's a let me try to find uh open AI retrieval Dev day so when they talk I'm looking for one image in particular this one yeah okay so if you look on this um if you look at this image query expansion they have this jumping up the Gap let me close my messages there um if you look at this it jumps up it it's it's one of the things that worked and so I think query expansion is really really powerful um so yeah hopefully that uh that helps um show how you can evaluate new retrieval methods um um we are going to add more data sets to evaluate on and we're also going to make the public benchmarks actually show more comprehensive leaderboard that you can easily explore because right now it's a little tough to know what's going on if you look at this we have a corresponding blog post I'll link to that in the YouTube um description I'll also link to uh I'll link to that I'll link to the The Notebook here so that you can run it yourself I'll link to this public data set um yeah that's that's pretty much it um really looking forward to continuing to Benchmark things as well as uh to to see what you guys come up with um you know there's lots of cool ideas out there ideas are only as cool as you can Benchmark them so let's do that