Techniques for Optimizing Search in RAG Systems

Key Takeaways

let's explore Advanced research techniques to improve how you find information in your rag application we'll cover everything from Old School keyword searches to more recent methods such as graph Rag by the end you'll know exactly how to make your data work for you let's start simple with keyword search keyword search finds exact or close matches to your Search terms it's perfect when you know exactly what you want if it's there you'll find it for example if you're looking for company polies about AI ethics keyword search will find document with those exact words two popular algorithms for keyword search are bm25 and bm42 bm25 or best matching 25 is a ranking function used in information retrieval it's like a sophisticated scoring system for keyword search first it looks at how often a word shows up in a document which is called term frequency or TF then it gives extra weight to words that are rarer across all documents which is known as inverse document frequency or IDF it also makes sure that longer documents don't automatically get higher scores just because they have more words finally it recognizes that just because a word appears a lot it doesn't make it drastically more important thanks to a Smart saturation function this saturation function makes sure that after a certain point like having aund times the same word these additional occurrence of the word add less and less weight to its relevant score this helps in balancing the influence of frequently occurring words ensuring that they don't unfairly dominate the scoring process since a word appearing a 100 times isn't necessarily 10 times more relevant than if it appeared 10 times bm25 balances all these factors to give you relevant results without being fooled by keyword stuffing or document length there's also bm42 an evolution of bm25 specifically designed to handle longer documents and more complex queries it introduces several key improvements first it uses a different saturation function that's better suited for very lengthy documents it also has a more effective method for dealing with query terms that don't appear in a document instead of Simply penalizing the document bm42 considers other factors like the overall context or related terms to better assess the document's relevance lastly it introduces a length normalization component that's more adaptable across various types of content so bm42 basically aims to maintain bm25 strength while addressing its weaknesses particularly for modern web scale document collections where document length can vary drastically both these methods help make keyword search more nuanced and effective going Beyond Simple word counting to deliver more relevant results in general keyword search works better when you need quick exact results it's great for well organized data and specific terms but it has limits if the words don't match exactly it might miss related IDs if we come back to our AI ethics example a text referring to AI ethics as something like ethical considerations and machine learning a keyword search won't find it that's where we need something smarter ideally we'd like a search that could understand what we mean not just the exact words we say that's what embedding search does embedding search or semantic search or vector search all the same thing transforms both the query like our a ethics pass query and our documents into high-dimensional vectors which we call edings these edings do more than just representing the words we have thanks to the models training they have learned to capture the semantic meaning of the text they can simply be seen as huge list of characteristics with a value for each of them which tells it for example if it's an object what is its color its shape Etc it can then use these new representations of words to compare them mathematically the search process involves Computing the similarity between the query embedding which is our user question or the term we want to search and our document embeddings which would be our data set that we transformed into this embedding format it typically calculates this similarity using a metric called coine similarity it just means that it Compares each value of our query embedding Vector with the other of an embedding Vector from the database it then repeats this for all embeddings that we have in the data to find the most relevant matches this approach allows the system to find relevant documents based on conceptual similarity even when exact keywords don't match it excels at understanding context synonyms and related Concepts but requires more computational resources and sophisticated models we use embedding search when we need to find similar IDs even if they use different words it's great for research tools when people might describe the same thing in different ways for example it could help find papers about AI ethics even if a user says ethic the artificial in French because it understands that both mean somewhat the same thing even if it's not the same words or even the same language but we don't need to choose between keywords and embeddings we can simply use both that's what hybrid search does it combines the exact Ness of keywords with the captured meaning of embeddings it can leverage both exactly what we say and what we mean hybrid search is perfect when you need both exact matches and related content it's great for mixed data and when you want to balance finding everything relevant which would be a lot of information with finding the most relevant things like the top three for example in online shopping hybrid search can help customers find products even if they don't know the exact name at the same time it can still find exact matches for product code likewise for our AI ethics example if the user types something like ethical considerations of autonomous vehicles you will find exactly the information about autonomous vehicles but also will be able to find some broader concerns related to Ai and vehicles that might be relevant ethically balance right it's the best of both world now let's level up with graph rag a new method popularized by Microsoft this isn't just about finding information it's about understanding how it's all connected graph rag uses relations between objects or people which we call Knowledge graphs instead of just finding relevant documents it discerns their relationships and extracts entities and relationships creating a structured representation that captures semantic connections from the data we use graph rag when the data has complex connections it's great for answering tricky questions that involve multiple pieces of information think of legal search where you need to understand how different cases and laws are connected but be careful graph frag is powerful but quite slow currently it can take up to 10 times longer than simpler methods as measured by a recent Microsoft blog post make sure you really need it before you use it sometimes the key to finding what you need isn't in the main text but in the extra information about it that's metadata filtering metadata filtering lets you to narrow down your search using things like dates authors or file types it's basically about using labels to organize your files use metadata filtering when you need to search within specific limits it's great when your data has clear labels or when you want to give users more control over their search for example if we have a large data set about AI in general and we have numerous tags for each text in there like rag llms autonomous vehicles or ethics we then be able bble to directly get all the text related to only ethics without any heavy search just apply one filter and that's it we can then proceed to search in a much smaller part of our data with embedding search for example to find the most relevant information from this filtered setep but then how do you know which one to use can you just use one or use all of them that's where a router comes in a router chooses the best search method based on what you're asking and what kind of data you have it's usually a powerful language model that understands your query and the search options you have and figures out the techniques that would best fit your needs it would just be like you after watching this video having a good idea of why one should use keyword search or embed search or graphs routers makes your searches faster and more accurate they can use different techniques for different questions all in one system for instance a router in a customer helps system might use keyword search for product codes embedding search for general questions and graph rag for complex problem solving so which search technique should you use well it depends on your need for example organized data take with keyword search or with an existing SQL database if you need to understand meaning and context go for embedding search want a bit of both try hybrid search dealing with complex connected information consider graph rag if you have a large data set and need to filter results try metadata filtering and if you want a system that can do it all try out a router remember the best search strategy fits your data your users and what your system can handle don't be afraid to mix and match and create the best search for your use case as you build your next data project think about how people will look for information the right search technique can turn a frustrating experience into a truly smart system if this video was useful and you are watching it from YouTube consider taking a look at our course where we dive into rag from Zero to Hero with all these filtering and search techniques and more advanced approaches for improving llms in real world applications with code and everything the link is in the description below thank you for watching and I hope you enjoyed the video