How to Make RAG Chatbots FAST

Key Takeaways

in this video we're going to take a look at how we can do retrieval augmented generation as an example of the sort of tooling or the sort of capabilities that we can build out within Nemo guard rails so typically what we would do when building out a right pipeline for llms is we'd actually take two possible approaches both approaches are going to kind of use a similar components so let me just draw those out very quickly we're going to have a vector database okay we're going to be using pine cone we're going to have an embedding model we're going to be using a text we're going to be using r002 so that's your embedding model and well basically we would have taken some documents from somewhere and we would have fed them into our embedding model and saw those within pine cone already okay now the two approaches to rag that we can do with a lens or the two traditional approaches is we can take a naive approach so naive approach is that we have we have an llm okay up here um let me just do it here we have an LM and let's say we have a query okay we take our query and what we do is we actually take that straight to the embedding model here okay that creates our query Vector xq which goes into Pine current and returns a set of documents or contacts which are like relevant information okay for that particular query now what we do is we bring that over here and we're going to merge it with our query okay so now we have the query plus the context and we feed them into the LM it doesn't matter what our query was in this instance our query could have just been Hi how are you right and we actually went to Pine Cone or we went and embedded that went to Pine caring retrieved some context that's why I call this the naive approach um the pros of this is that it's very quick right to to embed and search through Pinecone is it's like incredibly fast so you're not waiting on particularly when we compare that to the other approach so the other approach is slower more complex but you know potentially more powerful right so that is where you have like an agent which is essentially like a big rapper around your LM that allows it to have multiple thoughts over time like inter like an internal dialogue so when you send your your query I'm gonna just put a query over here this time it goes over to your agent and then it's getting processed within that agent for a while right another thing that the agent can do is it has access to external tools right one of those tools may be like a retrieval tool so the agent is going to say okay your query if it's hi how you I don't need to do anything I'm just going to respond directly right it's just going to respond directly to you you know I'm doing okay actually I'm not doing okay because I don't have feelings because I am AI all right it's going to say something like that um but if we do ask it something that requires some external knowledge what it can do is it's going to refer to its like external knowledge tool over here it's an external knowledge tool is going to point that query or a modified version of that query to our embedding model that is going to create xq again which then gets sent to pine cone and it gets some context okay they get passed back into like our tool Pipeline and process internally by our agent again which will output and answer okay based on those context in our original query now I mean you can see straight away that this process is it's heavier right because before we even get to this retrieval tool the LM needs to have generated the fact that it needs to use that tool and LM Generations are basically always going to be the slow part of the process at least for now so before we even get to that tool we have one LM generation right and then we go through um and we come down to here we feed those contexts back into our agent and then we have at least another LM generation and actually if you use like out of the box approaches from Lang chain I think at minimum you're going to be doing uh three LM Generations because it also has like a summarization step in there as well so basically you're going to be waiting a while but you'll probably get a good result now what guardrails does is kind of cool because it allows us to not do either of those approaches but instead do something that's kind of in the middle so this is looking kind of messy already but let me let me try Okay so we have a query that query is actually going to go directly to guardrails okay so um call it g over here right so that's going to go directly to guardrails guardrails is going to actually use an embedding Model A different embedding model but still in embedding model so let me just put e for embedding all right send it take your query and create create a not a query what is kind of like query Vector but it's not using it as a query Vector all right let's just we'll call it V for Vector at the moment what that's going to do is it's going to look at the guard rails that have been set right so we have those definitions of user asks about politics or Define user asks about large language models and it's going to look at whether that query has a high similarity to any of those things what we might want to do is if the user is asking about language models we want to actually trigger the retrieval tool okay like our own ritual tool so we're going to say okay is that semantically similar and we based on that semantic similarity we decide on a tool or we decide to just generate a response so we've now kind of done what the agent was doing but when not doing that first LM generation which is makes things a lot faster right so now okay we've decided yes we do want to send our vector or our query over to picome right so actually what we're going to have to do is we're going to have take that query and we're going to bring it over to our embedding model here because they're different embedding models so we have our embedding model and then we have our query vector that goes into paint gun from that we get our context and here is where those contexts would actually go into our LM now how do I do this I've made a bit of a mess basically we want to put those two together our query and the context and we're going to feed them over into our llm okay here and then that's going to return our answer to us okay so it's gonna come over here so we have one llm call there and that is that's all we really need depending on the tool we may actually need to decide to actually use an LM column beforehand but yeah it kind of depends and it means that for those queries where we didn't need an LM call like we're saying Hi how are you we won't generate two we'll just generate one so that's where guard rails comes into this whole sort of retrieve augmented generation thing and the sort of unique approach that it takes um to this which is significantly faster than the agent approach while still allowing us to use tools which is pretty cool naturally just as with a normal agent we're not restricted to just using one tool we can also use many tools so that's I think pretty cool now let's take a look at how we would actually implement this okay so there's this notebook again as usual there'll be a link to this at the top of the video we're going to just install a few libraries so Nemo guardrails Pinecone for the vector index we have date this is hook and face datasets which where we're just going to download some data from that we're going to be querying against an open AI to create the embeddings and also for the LM course so uh yeah let's come down to here now there's this whole like indexing process with Vex databases I'm going to be very quick going through this because I've spoken about it like a million times so I don't want to repeat myself every time we're just going to start with this data set it's from hugging face it's a data set I created it's basically just a load of papers that are either the Llama 2 paper or related to the Lima 2 paper that I scraped from archive okay and it contains all this information we don't need all of that okay what I want to do is I'm just going to create some unique IDs and after I've created those unique ideas I don't want any of those other irrelevant Fields because there's quite a few in there so we just want to keep a unique ID the chunk the title and the source okay now what we want to do is embed that data there's not too much in there by the way just under 5 000 records so what we need to do is embed that for that we need an open AI API key you know 5 000 embeddings with all the zero zero two it doesn't cost much money by the way it's pretty cheap but you just need to enter your API key in here okay I will run that and now we can go ahead and create some embeddings so it takes embedding R to zero zero two uh this is how we're going to create those embeddings that response will give us this object data model usage we want to go into data and we've seen data we have two records each one of those records is one of our embeddings which every embedding from the r to 002 model is 1536 Dimensions all right now what we need to do is initialize our Vector index we need an API key and a environment variable for that from Pinecone this is all free so we head on over to app.piken.io and you should see something kind of like this so you'll be like your name default projects um you go to API keys and you sort of copy your API key and also just remember your environment here your API key you just need to put into here and the environment for me was us West one GC yours will probably be different and once you've entered both those you just run that and this is just checking that we have connected successfully I'm going to create a new index simply called Nemo guardrails rag with actions you can call it whatever you want it's okay and what we're going to do here is we're going to create the index if it doesn't already exist now obviously this is your first time working through this the index shouldn't already exist so it will create a new one we use a cosine similarity metric that is just recommended for order 002 and we need to specify the dimensionality of the vectors we'll be soaring within our index which is a 1536 that we saw earlier now we're just going to wait for that notes to be fully initialized before we connect to the index let's run that this will take usually like around a minute to initialize an index so I'll just skip forward a little bit okay and we see that our index is currently empty as expected because we just created it okay and then we add everything into our index we're embedding things and just kind of putting everything up there in terms of like 100. yeah that again shouldn't take about a minute to run okay once that has finished we can move on to actually creating our rag pipelines with guardrails so with guardrails what we we're going to be doing here is using gutter actions which are basically executable functions from within the guardrails like colline file if you saw the previous video you will know about these so we need to initialize one of those functions which is going to be the retrieve function okay and we need to make sure that's an async function because when we are using functions with async generate within guardrails they need to be asynchronous functions otherwise we're going to get an error so we're just going to embed our query to create our query vector and I'm going to retrieve relevant items from Pinecone and we're just going to return those okay then we follow that with another function okay I'm just going to print red code so we know when this is actually being called later on and this is going to take our query and the context that we retrieved from our retrieve function and it's going to put them into this kind of like prompt template which is saying okay you're a helpful assistant loads query from user in some relevant context answer the question giving those contents okay that's what we're doing here and then we're passing that back to open AI to generate a response we're going to call all of this from within guardrails given a particular criteria okay so we set up the initial the typical config for guard rails we're not really going to be using text Adventure here at least not for the red component so actually here I will remove that I'm a no let's say I'm a simple assistant I don't like talk of politics um so this is going to be our layout against talking about politics right we don't we don't want to do that okay let's see that here we've seen this before then what I'm going to do is Define okay the user is asking about Llama Or I think LMS in general right I'm we can change that to llm right user ask LM and Define flow okay now basically what this is doing here is it's creating a set of semantically embedded vectors and what God Rose is going to do is take our user's query and compare it against these and if it sees that they are very similar it's going to say okay the user is asking about LMS and that will trigger this flow here right and then in this flow we perform retrieve augmented generation so we get our context given the user's last message and then we create a retrieval augmented answer based on those and then we just tell the bot to return that directly because it's this answer has been generated by our LM so it doesn't need to generate a new answer based on that answer okay okay let's uh let's run that now yeah one other thing we need to do here is we need to register actions okay so we have this execute retrieve execute rag that's great but uh guard rails doesn't know which python functions we're talking about here so we need to register them so here we just initialize our rails and then here we register those so you can see we're passing in that function and we're specifying the name that that function has within the curling file so this could be different this could be like um get like instead of retrieve and that means that in our code line rather than calling execute retrieve we would be calling execute get okay but I'm just going to keep it as retrieved because I think that's easier to easier for it to read so register those actions and now what we can do is try out our rag agent built with guardrails a rag pipeline via guard rails okay so we saw um simple prompts when asking it to we're not asking anything about elements here so it shouldn't use the the rag Pipeline and it doesn't now let's ask you about llama two we should see that it will call okay here we can see right code that means that you use the pipeline and we can see that it gives us this answer line material is a collection of pre-trained and fine-tuned large language models so on and so on that's pretty cool it's a good answer it tells us everything but I think maybe something that I would like to know here is how does that compare we're using rad here what if we just don't use rag work just use guardrails directly without our road pipeline let's try all right so we're going to do this one uh this is a no rag kolang file okay so it just defines the politics flow doesn't mention anything about the Llama stuff let's run that okay so that is how no rag rails and let's ask the same question tell me about llama2 okay so it just says sorry I don't know anything about llama2 can you find a bit more information so I can help you better that's actually a better answer than what I got last time which was telling me I think it was telling me about the actual animals the llamas let's try another one so this is again without rag there's this thing called red teaming that llama 2 did basically it's like stress testing the model um so let's ask about that and it's like okay I don't know the answer to that maybe we could just try searching the internet for more information on the topic interesting I'm getting different responses now but still kind of sure is the point now let's try this with rag okay let's run that and here we go so what is red teaming red team is used to identify risks and to measure the robustness of the model with respect to a red teaming exercise I screwed by a set of experts it's also used to provide quality of insights to recognize and Target specific patterns in a more comprehensive way that is what red team was used for within the sort of training process now let's try our rug rails and I'm just going to ask you okay what color is the sky so it probably doesn't need this is a question but it shouldn't need to use a rag here and it doesn't it's just usually blue and so on and so on right so we can see that it is deciding when it needs to use Rag and it's not using rag when it shouldn't which is exactly what we wanted it to do that I think is a very good use case for where we can use guard rails it gives us the ability to basically create almost like an agent like tool that can use retrieval tools or other tools like a agent would but without that slow initial LM call and that means that when we are using tools that just need to be kind of triggered rather than parameters this approach is actually faster so that's it for this video I hope this is kind of like an interesting approach people technique that we can use guardrails for and as I said we can use this with a lot of other tools as well which is radical so yeah that's it for this video I hope all has been useful and interesting thank you very much for watching and I will see you again in the next one bye foreign