A gentle introduction to RAG (using open-source models)

Key Takeaways

it's crazy how many people message me because they want to start building rack applications so this is what I decided to do I created the simplest rack system that you could ever imagine only a few components only the very fundamental components for a system to work and my goal is to break down on this video break down every one of those components into why they're here uh how they work why do you need them so anyone who knows a little bit of python can watch this video and leave with a full understanding of the fundamentals of rack applications that's the only prerequisite for you to watch this video you must know python other than that you're going to be able to follow along and understand a bunch of stuff that people don't tell you normally because everyone jumps into the code too quickly in my opinion uh there are a bunch of bonus here so number one I'm going to be using Lama 3.1 which is an open source model so you don't have to worry about open AI or paying them money or getting an API key none of that nonsense we're going to be using Lama 3.1 which is going to be running locally which is a big big plus everyone knows open source there is however one downside of using llama 3.1 if you only have a CPU on your computer this is not going to be fun because it's going to be slow okay if you have a GPU you're golden if you don't CPU a motor Runner locally ah not a good mix so here is what I'm doing for those of you who don't have a GPU I'm going to be using lining studio now lining is sponsoring my video here but I want just to take a second to show you you what studio will do for you because it's super super cool so first of all here is their web page I want you to go there I take a look at all of the features that they speak about here but what I want you to understand is that lining is like having Visual Studio code in a browser that's my quick summary and I know they're going to kill me because it's way more than that but my quick summary is Visual Studio code in the browser but all of the sudden we have access to gpus a lot of gpus so we can go as crazy as we want just by clicking a button and switching environments and that's another big big big thing of lining I can be doing all of my work on a CPU and I click upon and now I have a GPU and my entire environment is going to be ready for me I don't have to reconfigure everything which is huge so just take a few seconds go to their page you're going to see all of the featur features and anyway you're going to see me using Studio right here let me switch to that tab this is The Notebook that I built and as you can see a bunch of diagrams we're going to go through all of them step by step you're going to find the code number one if you open the studio on your browser you're going to get all of this code obviously and number two there's also going to be a link with a GitHub repo that you can clone on your computer and just go out it there so this is the simple application that I built okay I decided to just build a system that can answer questions from a PDF now I want to be very very clear most of what I just built here you can just do by just uploading the PDF document to chat GPT and I'm asking questions about the document the document is not even that long I still decided to do it this way just to keep things simple okay you can extrapolate what I did here to a whole folder of PDF documents many many more documents that would not fit on chat gbt's context window okay so I went to Paul Graham's website and I grabbed do things that don't scale this essay here and I exported it as a PDF and that is this file here paul.pdf that is the file that I'm going to use to break down save it in a vector store database we're going to talk about that in just a second and use the model to answer questions from that asay so it's going to show you how to expand the the model's knowledge with a PDF document that the model doesn't necessarily know about okay so here is a variable I'm just starting out here with a variable that points to the document it's pa. PDF as you can see here very simple and the model that I'm going to be using is 3.1 llama 3.1 now how do we use a model let's start by configuring that how do we use Lama 3.1 on our local computer so I'm going to open a terminal window here and in my terminal window I'm going to be running if you go to the readme.md file you're going to see the commands that you should run in order to configure your environment so I'm going to start by just uh upgrading uh pip which is the package manager that you know help us here with python and then I'm going to be installing the requirements and here in the requirements.txt file if you open that file you'll see all of the libraries that I'm using actually I'm not using streamlet so I can get rid of that uh you don't need that here you can see all of the libraries that you're going to need to install in your environment in order to run my notebook okay so very very simple I'm going to close that now next we need to install olama so let me open here a browser with olama ol Lama is what's going to allow us to run a model locally so this is an application you're going to have to install now here I have I'm running inside studio so that is a Linux environment so if I click download here you're going to see that you're going to get information on how to install it on Macos Linux or Windows I'm going to go to Linux because that's what I'm using I'm going to copy that command and I'm going to run it here in my oops not there I'm going to run it in my terminal window you can see it's just installing a Lama and that is ready to go notice something very important it says Nvidia GPU installed and that is because I'm running right now my studio with it one T4 GPU okay so I can switch to CPU or I can select another GPU by going here through this configuration I'm just using a T4 GPU because that's plenty but that's the smallest one that's plenty for what we need to do here all right so now we have Ama now we just need to make sure we download 3.1 I'm going to do a Lama run llama 3.1 when I do that it's gonna download the Llama 3.1 model and it's gonna just present me with an interface uh just to try that model because I'm using the Run command if I were using like the pull command would just download the model so as soon as this model is down we can just try it out and that is going to be running locally within that studio right that studio instance if you're running this from your computer theama instance is going to be running llama 3.1 on your computer on your GPU or your CPU if you don't have a GPU which is going to be really slow so let's do tell me a joke that's usually my go-to phrase and here is one what do you call a fake noodle and imp pasta oh Jesus this is just a bad bad model all right so let's do buy okay so I have all of my libraries I have a llama and I have llama 3.1 running right here locally that's all of the setup that I need I can just go directly to the notebook and go at it now this is a jupyter notebook the way you run a jupyter notebook let me just really close my terminal is you go sell by cell and you can just execute every cell so again PDF file is going to be poll. PDF and the model is going to be Lama 3.1 later on we're going to use that variable so how is my application going to work well I want the following to happen I want to ask a question to the model about that essay that PDF document but obviously the model doesn't know anything about the PDF document so I have to provide the content of the PDF document to the model I have to tell the model hey answer a question using this content as your knowledge base okay don't try to hallucinate something just use this content as your model base so think about you're building a system for a comp company and the company has a knowledge base basically that knowledge base explains how the company carries certain processes and you want your employees to chat with the knowledge base to ask hey how do I finaly an invoice what is the process to submit a request for vacation and you want the model not to hallucinate that you want the model to just use the knowledge base as the reference point right so you have to combine the question that the that the your employee is asking this system with the knowledge base those two components go together to the model for the model to answer the question but here is the problem what happens if that knowledge base is too large what happens if you have pages and pages and pages of PDF documents well these models have a limited context window okay it's just a limited space of how much content you can pass them in one question so if you have megabytes of PDF documents or gigabytes of content you cannot just provide it all for these models just too much instead you need to pass only fractions of documents now what is the fraction what is the page in this case we have a PDF document and we want to ask a question and only give the model uh the pages that might be relevant to answer that question how do we know that well that's exactly what we're going to figure out right now so before we do anything I'm going to break down that PDF document into multiple Pages because you and I agree at this point that no you cannot send the entire knowledge base okay so I'm going to break down the PDF document into multiple Pages that's what you get here in this diagram that I put together here this is my wife's uh doing by the way we get the PDF document and we're going to use a PDF loader which is class from one of the libraries that we installed and that PDF loader is going to give us back every single page of that document as a separate object that's great that's a good start so let's take a look at the code here let me execute it first by the way if you have not used a notebook before I execute every cell using shift enter uh you can also just click this play button here and that will execute the code of that cell so this is what I'm doing I'm importing the pi PDF loader which is the library that's going to help me load the document I'm just creating an instance of that loader passing the name of my PDF file remember that's poll. PDF and then I'm just loading that PDF file now this load function already returns a number of pages like every page as a list so I'm going to get a list of pages notice that here I'm printing out how many pages the length of that list and you can see we got nine pages and then I'm printing I'm saying length of page and I'm actually printing the length how much content how many characters we have in the second page page one is zero second page is one I just it's just to have an idea how many characters fit in one of these pages and that is 3,217 characters and then finally I'm just displaying the contents of this page and that is content of the page is 10% a week and while 110 y y y y you get the idea here okay so that was the first step now I have a bunch of pages but I'm going to go one step further Pages still contain too much content not because the model does not support that you know Bigg space but because the more content like if just think about you if somebody ask ask you can you answer this question using this book there's a bunch of pages here yes you can read the entire book but it's going to take you a long time to find the answer of that question in this book instead if I tell you can you answer the same question just using this particular page well now you're going to find that answer much much quicker why because the space that you have to search for that answer is smaller I'm going to try to do the same thing here instead of just accumulating Pages where an answer or a model might find an answer I'm going to chunk out those pages I'm going to create smaller chunks of characters so I'm more precise at the time of asking a question to the model I can be more precise and instead of telling the model find the answer in these three pages I'm going to tell the model find the answer in these three smaller chunks of information I want to be more precise so how do we split this how do we do this well that's exactly what this diagram shows notice how we load the PDF document we get all of the pages we have nine pages and then we're going to take those pages within a Splitter Splitter is going to take a page and it's going to break it down following a configuration we're going to see the configuration right now now into multiple chuncks this is the other cool thing that the splitter is going to do for us not only we're going to be chopping off a page into multiple uh chunks but we're also going to do some overlapping between those junks imagine this okay imagine you ask a question and that the answer to that question starts at the end of a page and finishes at the beginning of the next page that is pretty normal to happen actually this section here in this book I don't know if you can see it but this section here does not finish there it goes all the way to the next page so if we're only sending the model a complete page to answer a question we might be sending incomplete information to the model because we also need a little bit more for the next page so how do we solve that well one way that we can solve that is by adding a little bit of overlapping between the chunks so instead of chopping off a page let's say in 10 different chunks we're going to do the same thing but we're going to make every chunk use a little bit of the content of the previous chunk so we're going to go let's say from character Zero to character 1,000 and the second chunk is not going to start in character 101 the second chunk is going to start in character 800 00 and go all the way to 1,800 and then the third chunk is going to go all the way from 1,600 to 2600 that overlap is going to help us chain all of these chunks together in a much nicer way so let's see how we do that it's very easy actually and I don't know if you've noticed but I've been using L chain as the library it's like the glue that's going to let me build this system in a much nicer easier way than trying to do all of these by hand so I'm using all of these pre-created classes like the PDF like the P PDF loader that just built out for me I just need to use them okay line chain is open source by the way it's just free it's open source so you can use it so in this case I'm using the recursive character text splitter that's a mouthful but basically what it's going to do is take my pages split them into chunks by a number of characters and add me some overlap which is what I just explained so I'm going to be configuring here I'm saying Hey I want every chunk to have, 1500 characters so not the 3,200 characters that are in a page I'm going to be chopping Pages like in two like you know in half approximately and I want a chunk overlap of a 100 characters so there going to be a 100 characters of overlap between the these chunks after I do that after I configure my splitter I just need to call it and split so you can see here how I getting I'm I'm getting this splitter and I'm calling the split documents function and I'm passing the list of pages so nine pages come in the splitter and 32 chunks come out so about three to four uh three between three and four times the number of chunks compared to pages that is what I just created by the way here you can see that I can also print a chunk and you can see just the content of the document now these chunks are way smaller awesome this is table Stakes this is the first thing that you do when you go to a company and you want to organize your knowledge base you need to do something like this you need to prepare the content that you're going to be sending to the model to get it with a question now there are a bunch of questions that we have not answered yet like now I have 32 chunks when I ask a question which chunks should I send to the model we have not answered that yet we're about to but we're going to get there in just a second my point is that when you're working at a company and you want to build a rack application the creation of this knowledge base that's ready to be sent to a model is first a necessary step it's one of the first necessary steps that you have to go through okay so now all of these documents all of these chunks in this casee are in this list the list chunks I have to store them somewhere okay so I can use them over and over again the next time I come here they're stored somewhere so A New Concept concept quote unquote that came out during this AI Revolution is the concept of a vector store so let me tell you in English what that means that is a database that's really really fast at storing vectors of numbers and searching for those vectors of numbers that's pretty much in English what that means that's what a vector store looks like again it's just a database so I can get all of those chunks the 32 chunks that I have and I can store them in a vector store database but I'm not going to store them just like if the vector store was just a regular database no before storing those chunks I'm going to turn them into embeddings I'm basically going to get like this fraction of a document and I'm going to be generating a vector of numbers just an array of numbers that represents that text now what what does that even mean well I have a bunch of videos talking about actually a bunch of videos like two or three videos talking about embeddings so I'm not going to go too deep into that but I want you to think of embeddings as like a compression mechanism that's very smart so it can read a page imagine you reading one page of this book and telling me in a few words let's say 10 words what that page is about okay so if I read all of this page and summarize this page in 10 words only I might say uh well this page is about model architecture and how to transform sentences I don't know if those are 10 words but imagine that I do that and I keep doing the same thing with every page at the end what I'm going to get out of this book is a bunch of 10w sentences summarized in every page by comparing those 10 words sentences I can easily find similar pages that talk about the same topic or maybe similar topics right so if page number one is model architecture maybe page number 12 also talks about model architecture and you can easily see just by comparing 10 word sentences you can easily see which pages are Rel related to each other that's sort of like the same idea behind embeddings right instead of just using 10 words sentences embeddings are using numbers but it gives you the same benefit with that list of numbers I can easily determine what other chunks are similar to Any Given chunk now the other way I like to think about embeddings is like coordinates right in the world like latitude and longitude now latitude and longitude those are only two values but they can represent any point in the globe right if I say latitude 24 longitude Nega 33 I don't even know if that makes sense but that will give me exactly one point in the world and I can compare that point to any other latitude and longitude to determine whether I'm close to another point or whether I'm too far away is the same thing but instead of thinking of two values latitude and longitude now think here about many many many numbers many dimensions but they just represent a coordinate in this multi-dimensional space two chunks that speak about the same content that you know focused on the same ideas are going to have coordinates in this multi-dimensional world that are that are very very close to each other and that is a fundamental idea here for what we need to do you're going to see that in a second so let's try to get this and store it in a vector store let's convert it to those vectors let's get every chunk generate the coordinates for that chunk and store them in a vector store so this is the way that will look like here is sort of like the diagram representing that process I'm going to get the chunks I'm going to be using an embedding model this embedding model is just a model that's provided by the people who built Lama 3.1 in this case we're going to be using that embedding model and that embedding model is going to give us back the coordinates or the vectors we also call them embeddings right and these are the multicolor things notice that every embedding might be a little bit different I mean the numbers are going to be different but we can later on compare these embeddings to determine which chunks are similar to each other and that is a very very cool thing and then all of these embeddings we're going to store in that Vector store database the reason I'm using here phas or F uh that is uh a vector store that's going to stir in memory F was created by meta um again it's just just free and that is going to keep my Vector store in memory if you were building like a real application maybe you will want to use a different Vector store database there are multiple Vector stores database out there Pine conone is one of them I mean there are multiple H is has the ability to store vectors right now so you can use whatever Vector store you want in this case I'm just using pies here so let's look at the code two lines going to do all of this so first I'm going to be instantiating my embeddings model and notice how I'm using the AMA embeddings that's the class but I'm passing the name of the model that I'm using which is Lama 3.1 every every model here usually comes with their own embedding model that's the case here so I'm going to be using that specific embedding model by the way I do not have to use the same embedding model as the model that I'm using I can use any embedding model but in this case I'm just using the Lama 3.1 embedding model so after I have that embedded model I can use pice that's the class that I'm importing right here from the list of vector stores supported by L chain and I'm saying hey just create this Vector store from the list of chunks that we have and use these embeddings class to generate the vectors what's happening here behind the scenes it's grabbing this class is grabbing all of the chunks is generating the vectors and is storing them in PES so after this line This Vector store is going to contain all of my chunks by the way I've forgotten to run all of these I think uh let me run this one here uh there we go and let me now run this one here which is going to create all of those embeddings and store them in Vector store notice that this is going to take a little bit more time I mean it's not that slow but this is actually going through all the chunks generating the embeddings and storing them in fight so now at this point my Vector store contains my entire knowledge base ready to use that is table steak again that is very very important okay so how do I get data from my Vector store how do I access those chunks that knowledge base that store there for that we're going to have to set up a retriever that's the concept that uh L chain uses to explain how you can get data out of the vector store so this is the way that retriever is going to work and pay attention because this is really really cool so the retriever which I'm representing here by this dashed line it's going to get a question so the question let's say the employee asks hey how do I what is the process to submit an invoice that question we're going to give it to the retriever inside the retriever behind the scenes that retriever is going to use the same embedding model that was used to store the information inside the vector store database and it's going to turn that question into the coordinates into the vector why does it do that because now with that Vector the retriever can find any similar chunks the idea here is that we want to take the question find any documents any chunks stored in the database that are similar to that question because that's likely where the answer is going to be stored so if we do this and we retrieve any similar chunks from the vector store the retriever can then out output every similar chunk that we have in the database see what I'm going with this so now you ask any question whatsoever and immediately we can give you back the chunks stored in the database that are most likely to contain the answer to that question very very simple I think you can see how easy it's going to be from here but anyway how do I create this retriever well it's just a single line that's the power of line chain that it sort of like hides all of the complexity and we don't have to deal with any of that and that line is this one here I'm going to create a retriever by asking my Vector store just give me a retriever please and that Vector store is just going to give me this variable and now notice how I can say well invoke that retriever I'm going to call the method invoke and I can pass one question what can you get away with when you only have a small number of users and this is is a question that you know should be answered uh from that PDF and I'm getting back four documents why four documents well by default the retriever is going to send back four chunks that are the most similar as computed by the retriever from your knowledge base so I pass a question and you're going to get back for chunks now I can pass whatever question and the documents might be different like in this case you can see I'm getting a chunk from page seven a chunk from page zero another chunk from page seven and another chunk from page three that's what I'm getting back now let me ask something different I don't know what did Paul say let me just find here fragile Airbnb now seems I don't know I'm going just going to pass this paragraph here okay whatever that is I'm going to pass that it doesn't have to be a question obviously just a piece of content I'm going to pass that and if I execute this now I'm getting pay 707 and four not three like before I can ask something over here I'm just going to pass this I'm just doing random stuff you get the idea here I just want that retriever to show me how I can get different things now it's 7704 so it's just the pages are are in different order you get the idea I'm going to just go back to the question that we had awesome 7073 so that's the idea with the retriever we can use that retriever pass a question get all of the relevant documents so it's time to bring the model here it's time to build like the main trunk of our solution so far we've only focused on putting all of the information in the cloud or sorry not in the cloud in the vector store which in this case is running in memory not in the cloud it could be running in the cloud but in this case it's just running in my computer or in this studio lightning Studio instance all of the information is there now after we do do that now we can focus on the actual system using the model to answer questions so this is the basic s system let's start with the basics we're going to configure a model so we're going to give a question to the model and that model is going to give us a response that's it this is what we want to build right now how do we do that well I'm going to be using the chat o Lama class from Lan chain and I'm going to be specifying Lama 3.1 as the model temperature zero remember the temperature is basically telling the model how creative it needs to be when it's close to zero the model will usually uh respond back the same way not really but that's the idea the larger the value of temperature is the more creativity the model is going to show so in this case temperature zero is fine the model is going to be Lama 3.1 and then we can do invoke who is the president of the United States notice that here I'm just asking a question that is general knowledge so the model should be able to answer that it doesn't need a PDF document to answer that the model should use its general knowledge to answer that and the answer is AI message content as of my last update of April 2023 Joe Biden is the president of the United States y y y y awesome but I don't like that weird AI message format so what's happening here is that this model invoke is returning an object an AI message object I want to get rid of that I just want the answer please don't give me an instance so L chain gives us the ability to use parsers and a parser is nothing other than a class that you can combine with other classes to modify the answer from the previous class in that chain so I can create a chain which is what the name Lan chain comes from with a model and a parser and the answer from the model will be the input to the parser and the parser will transform that answer so just to keep this simple here this is the first chain here beautiful chain very very simple chain the question is going to go to the model but the the response from the model is not what we're going to be consuming we're basically going to send that response to the parer to get rid of that AI message and just give us the string I'm I'm going to be using a string parser okay and that's going to be the answer the final answer so let's see that parser I'm creating a string output parser there are a bunch of parsers U you can parse out like Json XML a bunch of different things string is what we need here that's my parser and notice how I'm creating this chain I'm using the pipe operator so you can see it here chain it's going to be the model pipe parser that's the way L chain allows us to create these pipelines workflows or chains or one component is followed by the next component so when you see this chain and you want to interpret what's going to happen it's very simple the output of model will be the input of parser and the output of parser that's the return okay so now if we print out if we call chain. invo who is the president of the United States notice that the answer is the same but now there is no AI message content equals we just get this string why because this parser is taking that is extracting the text and it's printing out the text beautiful let's make it a little bit more complex so now is time to start thinking about the actual system that we're building remember we're going to be providing a question but we are also going to be passing context that context is the knowledge our knowledge base the content in this case from the PDF it doesn't matter that it's from a PDF right now we just know that context is stored in a vector store that we're going to be using here so if this is for the system for the employee to ask hey what is the process to submit an invoice we want to pass the question and we want to pass the Rel relevant chunks as the context now to do this the best way of of organizing this is just creating a prom template the prom template is nothing else than a just very easy format that's very easy to modify to understand and to share with other people in this case this is our template notice that it says you are an assistant that provides answers to questions based on a given context that's clear answer the question based on the context if you cannot answer the question reply I don't know that's kind of important I don't want my model to start making stuff up so if he cannot answer the question using the context the knowledge base that I'm going to be giving it he should say hey I just don't know how to answer this question be as concise as possible and go straight to the point and then I have context and a variable variable this is what this looks like it's just I'm using corly braces to have context there and the question and I'm using curly braces for the question this string that I just created here I can now use with a promt template class from L chain see I'm saying hey just create a prompt template from this string template here and that is our prompt now that prompt I can format it with with the two variables that I just created I can pass a context and a question that's exactly what I'm doing here just for testing purposes notice that I'm saying prompt. format and I'm going to pass a value for context here is some context and a value for question here is a question why context and question again because these are the names that I use here if I type here potato well here I'm going to have to type sorry potato here that's just the way the variables work all right so I'm going to go back I'm going to execute this and as you can see this is the output which is just the prompt already ready to use by the model it says you're an assistant y y y be as concise as possible and go straight to the point context and here is some context question here is question so think about this for a second when this is ready instead of context column not instead but after context column we're going to be passing the chunks the relevant chunks to answer whatever question we pass at the end that is what we want to do okay so let's put this prompt now as part of our chain let's connect the prompt to our chain so we have the full picture of what this system will look like so we're going to get the previous chain let's go back remember the previous chain had a model and a parser but now instead of passing the question to the model we're not going to be doing that we're going to be adding The Prompt in front of the model model because the output of the prompt that full formatted prompt that is exactly what we want to give the model so we're going to have now the prompt in front of the model the parser after the model and then the prompt receives two variables receives the context and the question that's it okay this system is starting to look the way it should look all right so let's look at the code again prompt now is in front of model which is in front of parser and now can invoke that chain passing context Anna's sister is Susan and question who is Susan's sister so if I execute this uh there is a little bit of a problem oh apparently what happened here apparently I don't have a model maybe I forgot to create a model somewhere let me just see this is what usually happens no this is I don't know what happened but let me just reexecute everything and see why it didn't work this is what usually happens when you're running this live things stop working all right so this is loading all of the data into my Vector store my retriever okay let's create my model I probably forgot to execute this that's exactly what happened I'm going to run this now and now I'm going to be setting up my template and finally I'm just going to be executing my final chain and again the context here was Anna's sister is Susan and the question is who Susan's sister so the model is using this information to answer and that that's correct the problem that I had here was because I forgot to execute one of the sales the sale that defines the model I did not execute that one there the way you usually fix it if if you know which cell you forgot to execute just go to the cell and execute that sale I usually prefer to execute everything from a scratch just so I don't execute any sales in the wrong order because that can screw things up so anyway final component here because now the context I need to find or I need to get the context from the vector store that is what I need to inject this system so how do I do this well we have a retriever we can connect that retriever as part of our chain okay now let's look at this diagram before we talked about the code let's look at the diagram this is the final diagram okay so I have the promt the model and the parser as we saw before now I'm connecting the retriever here but not is this the retriever expects the question remember the retriever needs to search the vector store using the question it needs to search the vector store to find the similar documents to that question so the retriever will be expecting that question okay and the retriever will be outputting the context the context is just the list of the four chunks that are the most similar that's it okay so the context will go into the prompt but there is also something else that prompt also requires the question so that is what you see the question going into the Retriever and the question also going into the pront so we going to have to do some magic there in Python for that to happen happen it's not not hard but that is we need to pass the question to both the Retriever and the prompt and because this is sequential is a chain we're going to have to do that little bit of magic now also notice obviously that the retriever is using the vector store remember we created a retriever off of the vector store so that's what's going to be using to get the information back so here is our new chain and again it's not extremely more complex so besides prompt and model and pars now I added this component here and this is the magic that I was telling you about of passing the the question to multiple places so not is this so I'm going to be invoking this chain with a question okay not the context because the context I had to invoke the change with context here but now the context is coming from the retriever so I don't need to invoke the chain with the with the context anymore I'm only going to be invoking looking it with a question but I need to pass that question to multiple components so look at this this is a chain that is going to be getting the context to pass to the prompt from the retriever notice that the retriever is here but I need to pass to that retriever I need to pass the question the way I do that in Python is by using the item getter function which is basically going to be getting that question from the input to the chain and it's going to be passing it to the retriever just using the pipe operator like we've seen and the output of that I'm going to be storing in context here and I also get the same question from the input to the chain and I store it here in this attribute question and this is the dictionary that I'm going to pass to prompt and if we go up there you're going to see that that is exactly what my previous chain was expecting expecting a dictionary with context and with question so by just figuring it like this and using that item getter thing uh I can actually accomplish this the question going into the Retriever and into the prompt so this is my new chain and what's really really cool about this chain is that now I can ask questions and the context will be augmented from the content in the vector store so here is what I created and I suggest you make modifications to this and augment it so I created an array of questions I have three questions here just to try this out what can you get away with when you only have a small number of users what's the most common on scalable thing etc etc and I have I have a for Loop that goes through all of these questions grabs the question prints the question and calls that chain invokes the chain passing that particular question now internally now you know what's going to happen when I pass a question to this chain number one the question is going to be passed to the retriever that retriever is going to go to the vector store and it's going to return by going to the vector store is going to get back four different chunks from the knowledge base in this case a PDF that are the most similar to the question that I asked and it's going to get all of those four chunks and it's going to pass those chunks to the prompt and the question is also going to get passed to that prompt so the prompt is just going to form my at the document that is going to pass or the the full prompt that is going to pass to the model the model is going to answer that question using the knowledge that is provided right there as a context the answer the response from the model is going to go into a parser it's going to get rid of anything that it doesn't need to be there and that final text that final answer is the one that we're going to get printed out so let's see working first let me execute this line that works and now I can just run this and it's going to go question after question answering that question so here you can see what can you get away with when you only have a small number of users just try to Delight your users what's the most common unscalable thing delighting users individually what's the one biggest thing they want to seem big too early blah blah blah now there are a bunch of modifications that you can do here my suggestions are number one try with different documents documents that you care about grab a site save it as a PDF try to ask questions about that PDF ideas will start flowing for you number two play with the size of the chunks okay right now I'm using 1,500 characters for every chunk I would suggest you increase that number see how the results change as you change the configuration of the splitter remember we use the splitter to generate those chunks here is the splitter somewhere right here so as you change this configuration and you increase the chunk size and maybe increase the overlap you will see how the model reacts number two try to use a different model here I'm using an open source model I'm using Lama 3.1 changing this code to use chat GPT or gp4 or GPT 4 o or gp4 uh o uh mini it's very simple because most of the code stays exactly the same you only need to change because you're using land chain you only need to change for example here the actual definition of the model so if I change this line and nothing else to start using chat GPT here this is still going to work which is very very cool you can also change the embedding use different embedding classes coher embeddings or or chbt embeddings and see how your results change okay try to parse and store multiple PDF documents instead of one document try different Vector store databases I have a video that shows how you can use pine cone for example as a database try that that will be my recommendation for you to build upon what I just showed you here and the more complexity you add this the more useful these systems become now as a reminder I run all of this using a T4 GPU you can run this on CPU it's just going to be very very slow B of llama 3.1 which I'm running using all llama if you were using uh let's say GPT 4 with an API key you will not need a GPU because gp4 will be running on on open ai's Hardware not on your Hardware you can come here by the way uh in linning studio and here is where you can even select a bigger GPU if you wanted a bigger GPU for some reason or you can just go to a CPU if you wanted to do that hopefully this makes sense if you have any questions whatsoever try to leave those questions below in the comments and I'm going to try to get answers to as many of those questions as possible um let me know what's your IDE idea for the next video bye-bye