Agentic HyperGraphRAG w RL: Graph-R1

Key Takeaways

Hello community. So great that you are back. Let's talk about the latest in AI research hyperraph rag combination with reinforcement learning. Now you know the classical graph rag. My goodness I have so many videos on this year from a graph neural network scaling graph rag beyond graph rag and then we had light rag and prag and older variations. But today today we need some new tool. We need a hyper graph rag system. And this was done here by Beijing University at the end of May 2025. You have your GitHub uh repo. You have here the complete code. You have everything available for you if you want to try it out. And here you have now the standard rag system, the classical graph rag system. And now our new brand new hypergraph rag system. And you immediately see we operate here with a hyperraph structure that you know from mathematics and theoretical physics. So let's have a look. If you want to compare here the knowledge construction and the knowledge retrieval process for the standard rack, the graph rack, the light rack, the prag, the hyper rag and the brand new hyper graph rag system. Here you have it here one view you see exactly what is the difference. And if you want to have a definition of a hyperraph, yes of course easy. Each hyper edge connects two or more entities. This is it. Now this was based on a publication also here by Beijing University here and they had here hyper relational knowledge graph. This was by May 2023 and they simply here developed here this methodology further and we have dual attention layers and self attention layers for local level attention. I hope you're familiar with this simple example. You have a person this is a function the position held and you have so much more information to this single person. You have so many others like start time replaces the end time replaced by. So all this information you want to have now in a higher dimensional complexity instead of your simple graph rack system. So what we do we have of course domain knowledge. Normally we have it in PDF in books whatever on the internet. So we have tons and tons of pages. So we extract everything into a hyper edge structure. And you see here beautiful give you here this particular example and we have now if you want a hypograph the knowledge hypograph is constructed automatically just have a look at the GitHub it is done completely autonomous and we'll go through the process how this is happening in detail in about 2 minutes time the beauty is suddenly over our hyper edges we have now if you want a clustered multi to dimensional added content. So therefore, our rack extraction will become much more efficient. So let's talk about this. This is now my third video in a minieries that I have about how can we really go to the leading and bleeding edge of AI research. In my first video, we talked about search or one training the LLM to reason and leverage a search engine as a tool with reinforcement learning MCP. The second was unifying a rack system, a classical rack system and the reasoning the deep reasoning through a reinforcement learning. And today we talk here about the next step in the complexity. We talk about graph R1 from the end of July 2025. A gentic graph rack framework via an end to end reinforcement learning. Human completely out of the loop. Here you have an AI system that tries to self-learn, self-improve, self reason on multiple levels of complexity. Absolutely fascinating. Now the idea here from from graph or one is simple. No, you know our one here we have here the GRPO optimization methodology and graph you have understood it right now. We go now with a hyperraph a knowledge hyperraph. So therefore you see this is now the third video in this series and now it is so easy to understand what is happening. So you have here your agent ask me anything else reason should a knowledge hypograph to find the answer. You have a multi-turn interaction not just a single turn interaction here given that you have so much information on a knowledge hypograph. And then let's say the user has a simple task. Hey, who is the spouse of the director of the film in memory of Sergio somebody? So you see you need more than one simple run here to a rack system because you have here multiple links and the agent now thinks hey to answer this I need to find out who directed the film and then find out who is the director's spouse. So you see simple query knowledge hyperraph director of the film in memory of Sergio and now it is so much easier because you have a higher dimensional complexity on your graph structure with hyperraphs and so every piece of information adds on adds on and finally you have the perfect answer. Let's have a look at this also in GitHub. Everything is already available for you. They have here different data set that they had already transformed to AP format and then for a knowledge hyperraph. I will explain in a second how you build your knowledge hyperraph. It's rather easy. It's completely automated. You have here the Python code already. You can set up a port and run it immediately. But let's just start to understand what is the difference here to the classical graph rack pipeline. And remember in my videos about graph rack, I showed you the knowledge graph construction was stage one. Then we had the graph retrieval. This was formulated as a two-step process. We retrieve here the candidate reasoning paths. And of course we had to prune here the irrelevant out. And we had here a beautiful optimization plus the answer generation was then simply given the human query and all the selected path. The answer generation produces here a natural language answer. Why? Grounded in the graphbased evidence. And I explained the formula to you already. Now if we go now to a hyperraph retrieval, you guess what it is almost identical. You just have now a dual path interaction process for the hyperraph retrieval. You also have the entity base retrieval and the direct hyper edge retrieval. Remember this is here where we have an added complexity. But you see nothing else directly retrieved the hyper edges hyper edges based on the query hyper edge similarity. more about this in a minute and collect their associated relational facts. You just have a lot of more information if you just compare this to a single note representation and a single multi-dimensional edge representation and then you fuse everything back together via a rates protocol rank aggregation. Great. So here you have it. This is here if you want one two three steps here of the classical graph rack pipeline and here you have your one two three steps of your hyperraph pipeline and I just wanted to show you it is so easy it is so similar you just have to integrate here the higher complexity that provides more information now you know in my video where we talked here this is here the thumbnail and here at 8 minutes 52 seconds I showed you here the GRPO O objective the the object that we are doing here a mathematical optimization process on and I explained to here all the different terms to you if this was here the GRPO objective and the search R1 template now I show you the same for the graph R1 so the graph R1 template you have here and you will immediately see oh it's so similar almost identical yes of course because we just exchange now that we have somewhere search on the internet with a search here on a structured hypographs and you're not going to believe it here our objective here also going with our 1G RPO methodology is simply this term is there something special yes we have here row parameter for adjustments for the distribution shift we're going to expect and our advantage a that you know that you're familiar with we normalize here normalizes here the reward using a particular scaling function f never mind it's a technical detail but you see you're immediately familiar we have a group clabber divergence our beta parameter our clipping that is happening this is everything that we know that we understand how it works now what is different and now careful now the reward function and you remember in this video I told you the autoome search1 told us hey we use the simplest rulebased reward system that we can find with a final outcome reward function and they said yeah in the future we will go to more advanced rework function adaptive retrieval strategy multi-tool integration and so but now here in this new one they really went this extra mile and have a look at this so they decided now different to search R1 in graph R1 to go now here also with an outcome directed reward function but we have here a two ports at first and this is what you're already familiar with a format reward. So it simply encourages here our AI agent to follow here the intended reasoning structure. So if the output includes here a well-formed block that you have defined and then the second one we have the answer reward. So here we measure really the semantic correctness of the answer generated if you compare this now to the golden ground truth answer. So again a process that we know but now a little bit more complicated and you say hey wait I know the I operator here. Yes of course this is exactly that I showed you here when we have the loss masking in search R1. You see it is so easy if you have a threepart video series now you just see this and you immediately understand what's happening if you have seen the other videos. But you say, "Hey, wait, there was the second video know where we talked here about rag 3 in reinforcement learning and this your two as I called it. If we looked at their reward ideas at 17 minutes and 34 seconds, you also remember no, we had a two-stage idea for the reward function. The same we had a retrieval activation where the reward function only cared about the correct tool use and the answer the correctness was ignored in the first step and then only in stage two we went here for further optimization for the answer quality. So we evaluated the correctness of the final answer here in UR 2. And you see that this idea from UR 2 from the authors of UR 2 you find now also here in this publication here in the third video where we also have here the format the format reward function and the answer reward function where here in the second time we go for the semantic correctness of the generated answer. So you see so many parallel things here. This was by Jingua University. Great. So what are the insights now in our third video of this minieries? And I thought I will frame it now differently. And up till now I only thought hey we have to improve the performance of the agent of the AI agent. We have to improve here all the memory optimization key value cache optimization and so on. MCP optimization everything about the AI agent the LLM in itself. And now here I see brutally I see it is also about the structure of the environment because the agent interacts now this is the definition of an agent here with the environment and how this environment is structured the complexity the agent encounters now in the environment is a defining factor for the performance of the agent and I think one of the reasons for the graph one success and I'll give you the performance data in a second It shows us that this structuring the knowledge here of the environment is not just a data engineering task that we just clean it up here. No, this is a fundamental part of building here a high performance reasoning agent. So short no longer about just building smarter agents because the agents interact with something and we have to have to look at this as a complete system. You know, it's a kind of a symbiotic system where you have the agent and its knowledge environment. If you go with a hyper knowledge graph, those systems have to co-evolve in their abilities when they improve their performance. Okay. So, one of my next videos will be something like this. From an unstructured chaos here in the external world to a semantic order, the environment as a catalyst for advanced reasoning of multi- aent system because we have to rebuild for an optimized representation of the environment given our current AI system. And talking about current AI system, just yesterday I found this video here on the internet. years old that you can find it yourself. What was the main topic? They crash tested here the performance of a lot of AI enabled cars. This is here from China. But absolutely fascinating to see in the night on a highway high speed and you have suddenly here a car standing but not with the front or really the side kind of a diagonal interpretation. No. And it is if you see the title you understand exactly what happened there. And interestingly one of the commentators noticed that at higher speed at highway speed maybe the OI system if you have so many sensors and lighters and whatever and laser and on. Maybe it is too complicated the fast changing environment with a highway speed for our current DI systems that are built into our cars not just to detect and analyze the complete environment. Understand the positioning of the objects in the environment. understand everything about the dynamics of the objects and then try to calculate here an escape path an escape route that you are not crashing into this obstacle and absolutely interesting if you see this in real time so thank you here to the Chinese I don't know TV or broadcasting station Adas test great this was absolutely fascinating to see those AI system in real time and how they fail, massively fail, and the more systems they have on board. There was a correlation to a failure rate. And I had to think about this. There was one car, a really cheap one, not a lot of sensor, you know, the the cheap version, and this car managed to solve a lot of problems. So you see the more sensor arrays you pack into one system you have I don't know tons of data streams coming in at highway speed to have here the speed currently with our current implementation. It clearly shows us here the limitations of our EI systems. Summary the real world is filled with so much complex facts especially at highway speed that connect to more than two entities. So if we go with just a classical graph structure it will not be enough. And I have here this sentence here this is here from the graph paper here the R1 paper. Look at this sentence. You have a relation between a patient with a particular condition hypertensitive with a partition a specific level of an object with a specific quantification and a specific diagnosed great. Now if you want to map this here to a classical graph structure you will miss out something. You will miss out on the interconnected complexity on the interreation of those facts. Yeah. Standard rack you just have chunks. Standard graph of course you have here breaks down into a series of lossy incomplete pairs like those. And it really takes here kind of a hyper graph rack this hyper edge idea that you really have here if you want a grouping a clustering here that you have all the nodes that are specific to this specific patients and we have immediately access to this information. So I think the idea here of a hyper edge could really be the jump forward that we need because this hyper edge here links all four entity notes here of this single sentence together. So now we have here how we do it and now we just have to understand what is now the exact flow of this operation before we let reinforcement learning just automatically learn it. The DI system learns this automatically. So therefore we have to have now the non RL version where we really define in absolute precision the flow what RL will learn in the next step. So the mechanics of how to use not a hyperraph in detail. Four points at first we have to have the knowledge hyperraph construction. You have the code for this. Imagine you take Wikipedia here the text of Wikipedia and you tell I don't know you're Google or you're Microsoft and you say now okay now please build me a hyperraph from all of Wikipedia with the global Wikipedia really interesting if you would have this data for free for developers second we have a dual path retrieval strategy as I already showed you much more than a simple keyword search we have a twopronged attack we have the entity retrieval and the Hyper edge retrieval plus then we have the hyperraph knowledge fusion. Remember this is the critical step takes the retrieved entities and expand its search to find all connected hyper edges. But it also takes the retrieved hyper edges and expand expands those hyper edges to find all connected entities all nodes that are within the hyper edge. No. So you have a birectional expansion like the big bang of the universe. You ensure that it has the full data content or full information content here for everything that is even close by and then you have the generation. Then you have the hyperraph guided generation your classical rack system here. So the final retrieved knowledge here is you can combine it with whatever other system you like like a traditional text chunks and you fed this into an LLM with a specific prompt and then you have your classical rack system. Again did you notice something? I was very fast here because at point two the hyper edge retrieval in parallel it finds hyper edges that are semantically similar to the user query and I saw this is not possible. So going back to this point two the authors really decided that they have to build a vector space in addition a vector space for an embedding. So for the process of find similar expressions like to the human query they still go with a vector space. We have so many other opportunities but okay they go with a vector space and they say listen we have a separate sentence embedding model. They use anything right out of the shelf never mention and they create now a vector space representation and embedding for every node and for every hyper edge. So exactly as you would do for the classical WAX system and why because of course also for the graph R1 agent that we just built we have to find here a semantic similarity search optimization. So we have to go through this cosine similarity function and now the question is what are the elements in this cosine? What is the complexity of the mathematical space? And in the simplest case, it is a vector space. But of course, you would like to operate with a hbert space. I knew this. But for the moment, we have to go with a vector space. Okay. So, just that you know, we also have to build a vector space and embeddings and then we have a semantic similarity search classic. So now we have everything. Now everything is understood. We understood the process. We understand the steps. We understand the flow the complete flow that is happening and now we just tell the AI hey buddy you know how to do reinforcement learning yeah this is your data set this is your methodology this is your flow learn you see great that's it so therefore the hyperraph construction method in the hyperraph rack system is exactly the one we used in the pre-processing here of graph R1 nothing print the multi-stage retrieval and diffusion pipeline in hyperraph rack is exactly the process that we need for the graph R1 agent and we just use here the classical reinforcement learning with GPO I show you already here the objective function to learn the system and let it perform autonomously its optimization process that's it so and the beauty is instead of a fixed human coded hardcoded the number of loops, the number of the level and whatever pre-programmed material strategies. Now we have a more complex mathematical optimization process where the graph or one agent learns on its own its optimal multi-turn policy to find a solution to the human query. I want to show you the prompts that they used on the hyperraph rack system because I was really interested myself and I said how complex will this be? Guess what it is again? Another LLM, another EI system. The human is not anymore in the loop. So the relation extraction prompt T is missing. Yes, this this is a screenshot from the document. So this prompt is designed for extracting here a structured not a binary graph relational facts but an n area relational facts a hyper edge from raw text documents. This is done here with this and some llm is performing the job for this. Then of course we need an entity extraction prompt. Now I have the tea. Congratulation. So this prompt this is the prompt. Of course, we have here examples filled in on real data and the query that we really have. This prompt is used to extract you the key entities from a user query. But we understand I've given you the sentence here. I know the meta command and this is a person with a precondition and there's a quantifier and then there's also something. So we extract here our entities and then here the retrieve augmented generation prompt. everything that you're familiar with. I just showed you the template before. Now you have here the complete prompt structure. This is it. So nothing special, nothing fancy. Just we go to more or less a hyperraph rack system and we combine it in an intelligent way with a new optimized reinforcement learning methodology. What are the results? Is it any good? In the annex of the graph one paper you find table five and I highly recommend this because here yeah let's start here you have the query you have here very particular medical question and then you have the golden cor answer the human answer ultrasound whatever and then we have here a lot of benchmark that benchmark test here different perspectives here of the machine so let's have a look Yeah, you see we have here standard rag, graph rag, light rag, p rag, hippo rag and the hyper rag but that I can read it I found here this. So let's look at the first three naive great standard rag you have here the scoring for each of the benchmark and then the graph rack here with the evaluation scoring and you get a first impression where we are with the benchmarks. Then we have light rag, prag, hyperre and then here finally the relation score and you see first one is 100% 70% 92% 90% 95% 100% 100%. So on this selected paper by the authors of graph R1 for the complexity chosen by the authors of graph R1. This is the result that the authors of graph R1 want to show us as the best perfect case for the new methodology. Absolutely fascinating. But you may say yeah but how dependent is this here on the specific LLM that you use? And they check this out. So they use the GBD4 omni they use the Q1 2.5 a 1.5B a 3B and a 7B instruct and then you have for all the different data sets the wiki data set the hotspot question and answer music and error and you have here all the parameters now and you see here again for standard rag graph rack light rack path rack hipper rack on hypograph rack all the performance data just look here at the very last block if you want a 7B is already a model I would kind of recommend here. Unfortunately, still the 2.5 Cuban and not the three version 3 QAN never mind compared in the methodology you see that graph of one bold is the best is really outperforming all other methodologies. Real nice result but this is of course something that we kind of expected. No. So even if you go with the classical data sets here for evaluation, this really looks absolutely interesting for the future. And there you have it was simple. Now this is the end of video three. What we achieved today, we had a rack system, not the classical rack system, not the classical graph rack system, but we went with a higher complexity hypograph system, which we need, which is vital for complex reasoning. We defined here all the elements, all the interaction, all the dependencies. We defined the flow of the process and then we just used here the normal reinforcement learning process with a GRPO objective and some control parameter. And we had here an autonomous fully automated scaling selfarning optimization problem that DI could solve. and I showed you the performance of the new system compared to the classical rack systems. I hope you enjoyed it. Would be great to see you in my next video.