Advanced RAG 05 - HyDE - Hypothetical Document Embeddings

Key Takeaways

okay in this video we're going to look at the concept of hide or hypothetical document embeddings and this comes from the paper called precise zero shot dense retrieval with without relevance labels so remember dense retrieval is just looking up things with a similarity search or a semantic search usually with a vector store nowadays involved here so this paper came out uh last year at the end of last year it's one that I kind of feel like not enough people have paid attention to it's a technique that while it's very simple is very powerful in the way that you can improve your rag system overall so if we look at this diagram from the paper this shows you the fundamentals of how this thing works and this is that we're going to input a query into our system and then rather than straight away just look up the embedding for that and compare that embedding to all the possible chunks to find the best answer we're actually going to use a large language model in the middle and what that large language model in the middle is going to do is write a hypothetical answer so why would you use this let me explain this sometimes if someone asks a question that's very Bland or often that perhaps doesn't have any nouns in it or any things that we can easily look up to find an answer that can be quite challenging so if we look at the example of McDonald's right we all know the concept of what does McDonald's sell it sells food but if someone's asked the question okay what are McDonald's best items now if I ask you that question there's a high probability that you will know that I'm talking about food and then perhaps I'm talking about burgers or shakes or fries or the type of fast food that is the challenge is none of that is mentioned in the query so if I'm taking an embedding of the query and I'm then looking up an embedding of something else and let's say I didn't mention the word McDonald's maybe we've been talking about McDonald's and I just say oh okay what are their bestselling items now bestselling items plus you know looking for big Maxs in our semantic Vector store the chances of finding it is probably going to be reasonably small so in this case the one of the best ways to do this is to have the language model write an answer first now we're writing the answer not to let the end user ever see the answer the we're writing the answer just to get something which we can then take an embedding from so for example if the large language model writes the answer and it says oh you know their bestselling item is Big Mac or their bestselling item is the Tuesday burger or something like that even if it's the wrong answer you'll notice that it's talked about a burger it's talked about McDonald's is talking about stuff now which when we take an embedding of that will be a good representation to look up and find the right answer so this allows you to do something where you're not doing query to answer embedding similarity we're doing answer to answer embedding similarity and it turns out that this often works really well now it doesn't always work well if you're talking about a topic that the language model has no knowledge of at all remotely you probably want to stay away from this because it will tend to hallucinate things that are totally different and I'll show you an example of that in the notebook but on the whole if you're dealing with something where you need to look up something that's going to be quite specific it can often be quite good to have the language model do this now one of the other things it can do is rather than just generate one answer you can generate multiple answers and then you can feed those into you know you can average those out in an embedding model so you can take an embedding of each of those and average them out which will allow you then to basically look up things as well so there are a number of different ways you can sort of use this but it is a very powerful technique for this let's jump into the code and have a look at how this actually works in the code so remember that the hide is hypothetical document embeddings and the idea is that we're going to create a hypothetical answer with a large language model and then we're going to embed that and use that for the search against our documents in our rag here so okay we can see that the the code part to do this is we need some base embeddings or base embedding model and we need an llm chain and you'll see that we'll change the prompts we've got some basic prompts in there but we'll also then go for some custom prompts as we go through this all right so you can see what I'm bringing in here I'm using open AI just for Simplicity for the large language model for embedding I'm going to be using the BGE embeddings and you'll see that as we go through that this is going to be generating out the embeddings you could swap this out for any embedding system that you wanted to use which would be fine even if you were using like a local system you could use the quantitized embeddings for this in here all right so this is where the magic sort of starts so you can see that we're going to get the embeddings out of this but actually what we're doing is making a chain in here so this is from large language model chain we're passing in the actual embedder that we've set up here so we've set up our B embeddings and what this is going to do is this going to take in our query and you can see that when we look at the prompt for this we can see that okay what's it going to do please write a passage to answer the question so this is a very simple sort of example you'll see later on that it's often good for us to write this and customize it to our particular use case in this but okay we can see what we've got there let me just sort of go through and explain again what I I talked about earlier on was that something like what items does McDonald's make now we know that McDonald's makes food and it makes burgers and it makes fries and things like that but you'll notice that none of those things are mentioned in the query so this is where if we're taking an embedding of the query versus the answers back it might not be a great thing because you know the answers we would want back would be french fries Big Mac that kind of thing but none of those are mentioned in the actual query so here the large language model and I've just turned on Lang chain you know debugging here so that we can see the output in every step we can see that it goes into the large language model it basically says write an answer to this it answered that open AI writes is McDonald's is a fast food chain which is known for making a variety of things these include signature Burgers such as Big Mac Quarter Pounder as well as Chi chicken sandwiches wraps desserts we've got a lot more stuff now that's content that the embedding would fit this so something like Big Mac is mentioned in there now when it's going through looking for pieces of text it's probably going to match things that are Big Mac related in there so that's the first way and the simplest way to do it with just their built-in prompt that kind of thing the next way we can do this is we can also have multiple Generations so rather than just generate one answer we can generate multiple answers so here you can see you know when we ask it okay what is McDonald's bests selling item again we're not mentioning actually we're not even mentioning food here right so if we were just doing the query sort of lookup it's possible that it could get confused of not knowing that okay we're actually talking about food whereas the large language model kind of gets it even if it's not giving us the exact right answer it's giving us oh McDonald's one of the most popular fast food restaurants in the world its menu items include a variety of items but one stands out is the best seller the Big Mac right and we can see that we've got different answers in here bestelling item is undoubtedly the Big Mac and we got you know a number of different we got Big Mac mention again here so when we take the embeddings of these and we combine them we're definitely going to get this representation of Big Mac coming through for when we do our search with that embedding there so this is one of the key things that makes hide so powerful right is that you could take a number of these things combine these embeddings another thing that you could also do is use your own prompt so if I don't want a long answer like that back and I just wanted to mention the food I can say okay please answer the user's question as a single food item so now I kind of know I would use this in a case where I kind of know what the question's going to be about or the topic and I would guide the llm to that topic so if if we were doing this for the McDonald's website you know we could put in there you are a helpful assistant that works for McDonald's you answer questions about different things on the menu please answer this by recommending a single food item kind of thing in there so now when somebody just says what is McDonald's best selling item or even worse in this case is where people would just say oh what's your bestselling item right now what's your best doesn't even have McDonald's in it for the embedding that wouldn't go down well whereas we can see that if we've set up this user prompt quite well we can get something back which then in this case we just wanted a single item what did it return Big Mac we can take our embedding on Big Mac there's our embedding for Big Mac and then we could do a search on it for this kind of thing so just quickly to show you how to put it all together so in the original example they were just using a single file I've put in some text loaders in here and what I'm going to do is ask it now I got two examples here one that's not here one that I am showing here so let me show the one that works here first and this is you know please answer the users questions related to large language models again we're guiding it with a custom prompt here when I ask okay what are chat loaders it it kind of makes something up that's wrong right chat loaders are software tools used to load large language models into chatbot applications and that's not quite right but it doesn't matter because it's mentioned all the key things so that the representation is able to then find these chat loaders so that we can you know we can come in here and we can look at this this article chat loaders fine tune chat model in your own voice in here you for doing this so this is an example of how it would you know it would work you'd have to be a little bit careful so originally I was going to do the example of Langs Smith like I've done in some of the other notebooks the challenge there was that the large language model knows nothing about what Lang Smith was so unless I was to overly guide it in my question prompt it's it will just it came back saying that oh well it's language learning software so you want to be careful for if it's something that's totally new that the the language models perhaps never heard of you know then you want to be a little bit careful about using hide but for other situations using hide is often going to get you better results than just embedding the query and doing the query against the answer sort of similarity match there so this is a very powerful technique I encourage you to check it out and and try it out on your own projects and see how you get along as always if you got questions please put them in the comments below found the video useful please click like And subscribe and I will talk to you in the next video bye for now