Advanced RAG techniques for developers

Key Takeaways

welcome back to real terms for AI where we break down modern AI concepts for professional developers we did one episode on Rag and it's a powerful architecture for improving the quality of responses from an llm but sometimes basic rag just isn't enough to reach the quality you want luckily when that happens there are many Advanced Techniques based on rag that you can try one of the most important things to remember is that the context you provide to the llm matters a lot context in this case is the material you retrieve and augment your prompt with if the context isn't highly relevant to the user's prompter question the LM May generate a response that isn't relevant to the user's prompter question and if there isn't enough context the llm May generate an incorrect response it might hallucinate or it may not generate a response at all because context is so critical to response quality many Advanced rag techniques focus on improving the context included with the user's prompt all right so to make it easier to follow along today we'll cover these techniques based on the stage of the rag information flow they are used at and we're going to start with pre-processing in basic rag we divided our knowledge base into chunks and stored the chunks in a vector database so what are some of our options that may improve accuracy well our first option is that we could store some metadata with each chunk like the main topic the category the chunk fits in or a specific product that the chunk is relevant to or whatever makes sense in your specific use case you can manually input this data at ingestion time based on what you know about the data sources you are ingesting perhaps all the chunks from a specific products user manual should have that product ID in the metadata or labeling the information with the country tags of the countries that the specific choke is actually relevant to you can also ask an llm to expand its own understanding of existing chunks by generating new metadata for example providing an llm a list of potential labels and asking which ones to apply to a specific chunk via classification when you go to retrieve your chunks you can then use the metadata to filter your vector database before finding similar chunks that was a lot can you give me an example okay let's say that you know a user is asking about a specific product if your chunks have the metadata for the product or maybe like a product ID you can filter to just data about that specific specific product before you do the similarity search to ensure you only return the information relevant to the specific product for the prompt another technique takes advantage of the fact that questions or prompts and answers or responses often use different words in different order in this technique when you pre-process your data you ask the llm to generate a hypothetical question or prompt that could be answered by a specific chunk of data that you are processing then you store this question along with the data when you need defined information relevant to a user's prompt you search for similarity in the hypothetical questions instead of or in addition to looking for similarity In The Raw chunks that's pretty cool a final option we'll talk about here is that you can also Implement at the pre-processing phase a way to change how you store the data and rag doesn't have to use a vector database you can use traditional relational databases or even a graph database as well if it fits your data if the data you're using for context is structure something like a relational database may be better suited for that than something like a vector database and you can also store the same data in different ways for example you can store data twice using two different chunk sizes and pull from both when generating the information to accompany the user's prompts to the LM and if you have data in multiple data stores you can combine them at the retrieval stage of rag I think we need another example fair enough if a customer's prompts was about I don't know delayed shipment you could combine information about shipping methods from documents in your vector database with information about a customer's recent orders from a postgress database and general information about the weather impacts on shipping from maybe the shipping company's API and then R lm's response would potentially contain information from all of those sources if it was needed to actually answer the question exactly we also should keep in mind that while most rag tutorials use Vector databases Vector databases are not a musthave or even the best retrieval method and storage method for every use case based on requirements you may consider using other retrieval methods like relational databases keyword search Hybrid search graph databases and any search API you already have in your systems once you've retrieved relevant data you can help the LM use that data more efficiently through a process called reranking and reranking can get complicated and it probably deserves its own video but we'll try to give you the specifics and a couple ways that can be useful here so when you add reranking to your rag application you're adding a step between retrieving the chunks and sending those chunks to the llm with the prompt in the reranking step you use an algorithm of some kind to score the chunks by which ones are most relevant or useful to the user's prompt you can then use those scores to reorder the chunks and choose only the best ones to send to the llm so let's say your ranking algorithm May return aore for each of the chunks and then you can program your rag system to only send those chunks with a score of at least. n as example to the L and your reranking algorithm can take many things into account as it determines which pieces of data are most useful for example maybe you want more recent information to be considered more relevant or maybe you track user feedback and you want that data taken into account when deciding what data is relevant or maybe you know you found through experimenting the the particular sources Like official documentation often produce higher quality answers and you want those sources to be considered more relevant when deciding which context to include with your prompt and of course you can use Ai and data science here there are a variety of algorithms that can score how relevant a given chunk of data is for a given prompt and potentially you can combine several reranking techniques to get the best reranking for your use case I think the most important thing about reranking to think about is that it gives you another chance to ensure the context you're sending to the LM is the most relevant and the most helpful it can be in context of answering the question that's a lot of ideas for how to tune your rag app we've still got a few more ideas that people can try most rag systems make one call to the llm per user prompt but you may be able to improve the accuracy by making multiple calls to the LM as we mentioned in a previous video it can help to have the LM optimize the user's prompt this can remove spelling mistakes and unnecessary words and maybe replace words with more common synonyms you can also ask the LM to summarize all the data chunks you retrieve from your data store which may also improve the quality or accuracy of the responses and speaking of accuracy you can also ask the LM to evaluate the accuracy and relevance of its own results once you've generated an answer I know it seems like the llm should always respond that the text it just generated was correct but that's not how it works in practice all right are we done now we're done there's a lot of different things you can change about the basic rag architecture to improve the quality of the results and which one works for you depends on your use case your data and the other standard constraints like budget and latency that we've discussed before all right and if you'd like to try some of these techniques we've included links to Cod Labs on semantic search and using methods like graph rag in the description below this is a and Jason signing off and happy prompting