Build an Agentic RAG System with LangGraph

Key Takeaways

Hey, how it's going guys? In this video, we are going to build an agentic rag using langraph. So, if you look at here on my screen, I have this GitHub repo of course which is already available on my GitHub. It's called Langraph agentic rag. Uh rag is an acronym for retrieval augmented generation that automatically basic know corrects your retrieval errors using langraph and it uses fast for vector storage. Right? So the idea behind here is that most of the time when you're working with knives rack okay where you don't have a very intelligent way of finding out the retrieval errors and then uh try to uh correct it and again re rer it then it becomes a problem right you don't get the uh right set of information uh in your retrieval process that's a fundamental problem with rag right and rag uh fundamentally has been of is uh you know named after uh Facebook AI lab where they created this paper in 2020 uh or 2021 but after that uh we have been working with nive rag or hybrid rag and so on and so forth but this self-correcting retrieval is a very important step if you want to build a very good rag system and langraph is one of the best choices when it comes to uh orchestrating this entire agentic thingy because langraph is an orchestration framework work for building agentic applications right or agentic systems. Uh in this application that you're going to see of course we're going to look at the CLI version of this. We are using fast for vector storage. So rag basically have two processes or two steps rather you know retrieval and a generation. Retrieval happens through an embedding model where you use to create embeddings and store that into a vector database and then you use some kind of uh approximate nearest neighbors or a algorithms to find out similarities between different vectors uh and once these are retrieved right some uh vectors in forms of chunks or page content whatever you call it and then it's given to uh LLM for generating or synthesizing the responses. That's how it works. So this agentic rack system has these features. It has a self-correcting loop. Uh automatically detects and fixes poor retrieval quality. So the grading also comes up. If you look at your document grading over here uh LLM based relevant scoring before generation module decoup decoupling. So the way we I have written this is a very modular way. written the classes where you can just replace radius with pine cone, pine cone with cudrant, open AAI with entropic and so on and so forth. So from a algorithmic algorithm standpoint we have selfcorrecting retrieval and that happens through document grading and it rewrites the query. That's what it is right. So I'll show you that in a bit. So if you look at on my screen now we are on Windows Surf. So I'm using this IDE named Windsurf. Feel free to use VS code PyCharm cursor whatever doesn't matter. Right now if you look at from left to right this is a very high level architecture of this entire agentic rag that I'm showing you. So we have a start and then we have an agent. Agent is going to retrieve some sort of information and then it grades the documents. Documents are nothing but each chunk that it's each it's basically not the chunk but each page content uh is basically called as a document. So basically grade the documents over here and then it sees if it's relevant then gives gives this to LLM and LLM basically generates the answer and the loop is end okay but let's say if the documents that have been retrieved and after grading if this is not relevant then the agent will rewrite the query right because uh there are ways you can do multi-query decomposition just for the subatomicity right you basically decompose the queries or you rewrite the queries uh uh in different ways and then can go and retry uh to retrieve the thing and grade the document and find out the relevant answer. This is a highle understanding of this entire process. Right? This is an overview uh which is which is a bit more detailed. So if you look at this one right we start then we have the agent node. So we are using lang graph. Flang graph is basically based on graph. So uh when you work with graph you have uh ages and you have nodes right that's what you have. So ages are basically all the uh connectors you can say like the connects or the integrations. Now node is where you know node is the mother load. So if you look at this agent node we have this agent node which basically is the decision-making node for us. Now when you work with agentic rag the vector part the database part is used as a tool everything that agent uses it it's a tool it's through a tool so agent are basically three things that it basically uses one is llm one is there can be multiple tool multiple models and it can also have memory and other things uh in place so if you look at the tool call we're using fast vector store for retrieving the vectors and that's our retrieve node. So so far we have agent node and retrieve node. Now we have a great document for relevancy that basically checks the relevancy and if it relevant then it generate node will be used for producing the answer or generating or synthesizing or augmenting whatever you call it. It's your choice and if it's not relevant then it will rewrite the query. So we have a rewrite node query transformation and then again it goes back and try the same way. Now so the graph.py file where the basically it's your state machine the graph.py where we have our nodes and ages grading logic and then all the three nodes agent rewrite and generate right and then we have some configs where we going to keep our envings and so on and so forth. Right now the retrieval part is very simple we have a document injection and fast rep. So let's have a look at that now. So left hand side is my complete folder here that you see. Now we have created a virtual environment in the in this venv we install all our dependencies. So I'm using uv to set up this project. So I'm using UV for this. So I'm using lang chain langai because we are using openai GPT4 miniodel. Feel free to use any model does not matter a lot. Langchain community because we're going to use some open source modules and dependencies. We have langraph as an orchestration framework. Pythonv to manage the credentials and secrets. Tick token just for all the token related thingy. Fast CPU as a vector store. Fast is created by Facebook AI lab fair lab. And then we have beautiful sou 4. Maybe we don't need that. Uh maybe we need it. I'll I'll tell you why why we need it. Because of the data source that we are using. Let me show you the folder source folder. Source folder is the main folder. Now within source folder we have two subfolders agents and config. So the config is pretty simple. So in config we have our open AAI.py PI file where we are saying that okay use open AI embedding model and open AI chat LLM okay GPT4 mini chat open AAI is basically uh you know it's a class it's a module where you can use that the LLM into a chat template kind of a format so it's it's basically further fine-tuned for chat thing so we have a model temperature is set for randomness creativity and whatever and we are getting our open AI API Same goes with embedding. We are using the embedding model over here. So that's what our openi.py. Now settings is pretty simple. We're using loadenv uh and then we just getting open ai api key which is in my env file. This is our config. Now after config we have agents and within agents we have three files ages, nodes and graph. So if you look at this node thingy. Now in the node thingy we are using pre-built. If you look at this lang graph pre-built and then we have langent core messages, human message and system message. First thing is we are creating an agent node. Now we are giving tools access to this agent node and it can uses llm that coming from your config get lm thing and then we bind that we have to bind the uh tools. You can look at here lm with tools and we use llm.tbind tools. That's what we do. And then we are mentioning a state because guys lang graph is a stateful library. That's the beauty of Lang graph you know it comes up with lot of uh inbuilt features for state management sessions management uh scratch scratch padding uh in inbuilt memory storage and so on and so forth. So that's the beauty of lang graph. Lang graph is something that you guys should learn. Okay. Now we have our create rewrite node here that we are using an llm. We have a rewrite prompt. Look at this rewrite prompt. We are saying you are a query rewriter. Your task is to transform the user's questions into a more effective search query. And then we are relying on agent itself because it's say if I'm asking a question, I might not ask the question in the right way, right? And if agent can basically make that question into a bit more machine readable format. So it I mean the format is still remains English but into more of a machine readable words or terminology. So the effects of attention goes higher right. So that's what happens over here and you can see that and then create generate node and basically just using the generation prompt for generating the answer. So this is our nodes. Now in agents we have multiple things. One is your grade document that we discuss. Uh you can look at this binary thingy. Documents are relevant to the question yes or no. And there is a reasoning behind it. Please explain that why documents are not relevant because let's say if you completely rely on this agentic thingy for uh relevancy into some kind of regulatory industry like healthcare, pharmaceuticals and finance and entire BFSI uh explanability of rejecting the query might become a bottleneck. Right? So your leaders might ask the governance might ask that okay why the hell is that happening? So in that case you need some kind of explanation and that explanation again comes from agent itself. So it might not be really really uh quantitative I'll say rather I mean it's not quantifiable it's difficult to quantify that now the create documents over here as you see we again have great prompt uh uh we have again you can read this document grader your job is to determine if the retrieve documents contain specific and direct information because we're building a rag so in rag is like it should have the nearest uh similar vectors uh with all the information so that's the priority uh in this so that's what we have now graph is interesting so as I said right lang graph is based on graphs so we have to build the graph now when we are building the graph you can see we have a agent node rewrite node generate node and we have a great document functions and this is how you so basically it basically it's a workflow you can also generate this diagram so the workflow you can see the workflow we are adding the nodes nodes nodes nodes nodes and there is a entry point which is for the agent in at the top that you see and then there are some conditional ages. So conditional edges where the you know the grading document functions will come up because it's a conditional thing. So this is how it works. This is there. Now we have a retriever pi. So I'll show you the retriever because I want to show you the source document as well. So our our retriever looks like this very modular right as I said. So we have an inest document function where we are using webbased loader guys. So in langchain community document loader we have web based loader. Uh here we get the docs. Docs are basically in a python list when you use lang graph or lang chain and we are using recursive character text splitter. It starts it's a top to bottom approach. You start with line breaks breaks. So of course paragraph and sentences and words and whatnot. Uh and then we have a chunk size of thousand. Now chunk overlaps would be anywhere between 10 to 25% of chunk size. That's what the industry says. So we have that as well. Now we get the embedding over here using the embedding model which is like let's say embedding three small or large and we're using fast. You can also persist that if you persist it will create two files pickle and dot fast files and then we return the vector store and then we're using the get retrieval tool to get that in as a tool. And we have a k equal to four. So uh that when we use the algorithms we are saying k equal to 4. Now uh main.py is where we have the thingy. So our these are the things we are using guys the data source. The question is okay what are we building rag on like what are the data source we are using. So we are using this blog here. Uh you can look at the uh Lilian wing blog lm power automation agent written couple of years back a very good blog that I have used. So I'm using this as a web source. You can use any other data sources if you want. It's your wish to do that. Uh I'm doing that. You can look at here some ingesting documents into fast. We are using the retrieval tool and then we have some couple of questions. This is our sample questions but you can change the questions and you can look at it. So let's run this and see what happens. I'll just make this big big. Now you can look at over here right I asked this question. Uh so I I ran this file. So it says initializing agentic rack system and this is completely CLI based. I do have UI based similar solutions that I previously created. I'll give you that video link in description as well. I wanted to keep this completely in CLI. Okay, but very modular. So you can use it into fast API or wherever you want to use it. Now it says ingesting documents into fast vector store documents ingested successfully. Building langraph state machine graph build successfully. So the graph is built the aentic system is ready. Question is what are the key components of an AI agent? So if you look at this node agent output grading is no. So it does not go into the grading. The retrieve document discuss it gives you the reasoning of it because the reasoning I told you it's comprehensive list blah blah blah and then node retrieve planning. Now the agent breaks down large tasks into smaller manageable sub goals. Now if you look at here that right this interesting. So it's basically have the rewriting thing which gives you an output. What are the essential elements and functionalities of an AI agent and then it gives you the outputs. Such a nice such a nice way to uh you know bring the atomicity over here right when you rewrite the query and get a better answer. If you look at the final answer it says the essential elements and functionalities of an AI agent include planning memories tool natural language interface blah blah blah. Right? So that's for the first question. Then we have the second question as well. How does prompt engineering improves LLM performance? Same goes for this. Okay, if you look at here the the query gets rewritten. Prompt engineering techniques on the performance of LLMs. And it also generates this answer over here like prompt engineering techniques significantly impact the performance of LLM and it gives you some headers like zero sort, few sort, empirical science, benchmarkings, complex tasks handling and so on and so forth. So these are the question that I tried on but you can change the question of course you can change the data source as well if you don't want to use this particular blocks that I have been using it. Now this entire uh aentic rack is built in lang graph. So it's already uh on a level that you can start building ps using this. Maybe not in the production directly but p is for sure right now. It's available on my github repo. I'll give this link in uh uh description. Have a look at this. Uh it's pretty good. Read me is pretty nice and clean. Uh feel free to contribute if you want to contribute more. Here's the workflow. Agent receives questions. Retrieval happens, grading happens, decision point if generate not relevant and then generation. This is a state machine flow because lang graph is based on that. And here are some questions that you see right now. This is what I wanted to show you in this video. I wanted to show you that how we can build an agentic rag with self-correcting retrievalss, document grading and query rewriting in langraph. Okay. uh I do have other ripples which are with UI as well uh that you can also have a look at that it's called maybe this as well AI agent with MCP tools or there's something called selfhealing rag that is something that you can also look at self-healing rag is also there so self-healing rag uh is also well appreciated already there are people using this repo uh you can look at this right so we have same thing And so we have Heidi here hypothetical document embeddings uh query decompositions like I telling you about the atomicity of the question C rag which is the corrective rag thingy. So combine both the technologies guys to make a very powerful rag system because rag is something that you should now master it completely. That's all for this video. If you uh like the video please hit the like icon. If you have not subscribed the channel yet please do subscribe the channel guys. That motivates me to create more such videos in near future. Please share the video and this channel with your friends and to peer. Thank you so much for watching. See you in the next one.