Discover LlamaIndex: Custom Retrievers + Hybrid Search

Key Takeaways

hello everyone welcome to another tutorial with llama index today we will explore a unique type of search system in lava index known as custom retrieval or hybrid search engine in our previous tutorials we primarily use vectors such based on embedding similarity however relies solely on its method may not always be correct so let's say you are looking for some information from an essay by Ball Ground regarding these activities afterwards time at Yale our different search methods would approach this time differently let's say if we employ a keyword search the system will search for exact matching expecting results that is however if programs say it does not directly reference real a keyword search might not return any relevant results living with an empty set and contracts um let's say a vector searching understands the context or meaning of your file it can provide your advantages even without a direct mention of deal it achieves this by utilizing embeddings to manager exact context of websites it's similar context in available data this way it may permission meaningful response regardless of whether the keyword ale is present in the text so how does our hybrid search handle this situation if we set it and it will search for results that match both the keyboard and vectors are expected them since there is no direct mention of real in the essay this convolution would not result any results so through this example system [Music] depending on circumstances you can select to use either both keyword and Vector sets or just one of them in this tutorial we'll guide you on how to implement this flexibility so you can get the most out of yourself system so let's get started with it let's start with the installing gamma index and then we even import login and set the logging things and then we got all the things related to our customer and I understand then simple keyword table next to build their indexes response insurance and service context and storage context for setting our different service or storage index as well and you can check my previous videos to understand all these different things uh positivities explain different indexes and storage context service context and or other key components [Music] [Music] so yeah we need this and we also need the data set for this program index products um it's a data folder of scheme that in data folder and Dot txt [Music] so yeah so that is a delivery and then we make the documents and set the default sample size you can read the boxer make notes accordingly so and create a vectors for index as well as a simple keyword tablet X let's create them um and then one is all this index separated and so uh to create the customer retriever or the hybrid search that involves both both the usage of the trendex and period index basically we will retrieve uh the nodes using the animating similarity as well as uh specific keywords from the Keyword Index answer so we need to create uh two reverse Financial territory one and keyword retrieval and then pass it to the class uh and then the mode is and or are so and um and means you return the nodes uh both from the internal Retriever and keyword retrieval uh using those two levels and then if it is and you just combine only common nodes uh if it is odd you combine notes on both of them based on the user that you would like to have earlier [Music] [Music] and initialize the default uh response synthesizer and create a retriever query engine so the query engine will be with custom Retriever and then you see the response inside that we have to Define and we even have a temporary interval you can also to compare further quality and you have a keyword code in there as well YC and so we gave an answer I would order done now let's do what the author did what this time until uh sincere has never mentioned uh about checking the essay so the response would be even and the retrieve nodes will also be zero for the same but if you do uh something uh with the letters research um if you have an answer but but again if you see uh the number of nodes are present in it it will still retrieve some nodes because of the embedding similarity but here uh the custom retriever we don't even retrieve all the names because uh combining both websites and keyboard sets so in that way we will be able to make it more accurate and also the cost because the amounts are zero the costs are taken for generating a RSM will also be low so this is how you can have a customer table and then work on work towards it with the interesting resources and you can even check the same in the top 2007 about the customer table and that's all for this video um I will read more interesting content in the next video thank you