ADD LLM TO Knowledge-Graph: NEW GIVE Method (Berkeley)

Key Takeaways

hello Community a very fast video on a beautiful new mechanism to combine the knowledge of llms with the knowledge the structured knowledge in Knowledge Graph and this new methodology is called GIF now you know llms like open1 are nothing short and transformative but today we're not just going to do a simple knowledge retrieval from the knowledge gra no we will have new innovative ways how we combine here structured knowledge and the intelligent reasoning together now both methodology I'm going to show you maybe you know sync on graph tog we have it now for months and a brand new gift methodology by UC Burkley remarkable advancement here in integration of LMS and knowledge craft so let's start and let's explain here the simple one the old one that we already know think on graph responsible reasoning of LM on Knowledge Graph reference work we have here uh idea research University of sou col fornia C University Hong Kong University Microsoft beautiful now what they do instead of just fetching here some simple facts and data from a Knowledge Graph the llm in this particular case could actually sync its way through the knowledge graph exploring here different roots to take which is beautiful and it works like a detective ging the cloes this is exactly what syn on graph does here how does it work here well it couples the llm with a Knowledge Graph using a simple beam search algorithm so this helps you the L&M to explore multiple possible reasoning P at once and then choosing the best one if you want a detail explanation of beam search I did to see in my video where I showed you Standford and open Mii code and intelligence Shield because they they used a similar methodology so this means the llm doesn't just R refracts from the knowledge graph it makes a dynamic decision doing so I will show you in an ex in a minute a beautiful example and the sync on graph supports also multihop reasoning which is beautiful because this is exactly what we need for more complex task so simple example we have a question what is the majority political party now in the country where canbera is located now if we have only an llm and even if you say hey chain of sword and whatever let's sing St step by step you know the model will fail if we then have here an llm plus a Knowledge Graph and we have here a sparkle query like we do normally for our rdfs you see here this is here the response that we get so we retrieve here cber is located in Australia majority party in Australia is however not found here in Sparkle So we do not have the link that Australia has this political party called whatever so the prompt now response here sorry based on my qu result from the knowledge base I cannot answer your question because there was no direct link in the knowledge C available and now this beautiful new thing is you're looking for triplets related to cber so on the graph you go here Cur is the capital off and whatever and you buil a subgraph if you want and then you kind of think so you see the most relevant one is cber capital of Australia looking for Tribbles related now to Australia and on the graph I look here for a subgraph with a high probability and I see here in Australia then for example the prime Minister then I do another singing here the most relevant one is Australia prime minister and this is Anthony congratulations so you see I go step by step thinking and building here whatever I need and then here the system is able to think hey I know that Anthony is from the labor party this is enough information collected to answer here this particular question and finally with Sy of C think on graph we succeed here isn't this beautiful if you would like to see this in a slightly different way here we have the same question of course and then we have different search depths so we have here again camera continent territory capital of part of whatever is our subgraph we have here then the second if object if you want Australia all the subgraph and then we have Anthony again with what whatever anthonyy is connected with in the knowledge graph and then having here this Mighty powerful subg graph we start to prune it we look at the most important information particular in order to answer you the user query so you see you prun the graph now you look only at the most important probability links for example if you have not enough information no problem you look now from cber to Australia you see here the network of Australia you prune here give a probability and here the pafic query not enough information you go on with Anthony and exactly like I just showed you this is just another visualization in the official publication if you would like to have this in a little bit mathematically introduced way this here is a simple introduction beam search for multihop reasoning PFF pruning and traceability and the code of course since it is now month in the making here we have a beautiful GitHub repo this is here the link for you and you see just the python main free base or if you want you can go with main Wiki base whatever you have you define here your query and your llms and whatever you have and it is simple the code is now after month quite stable to use for your example have fun with this but now knowing this rather month old model let's look at a brand new model GIF and GIF stands here for a graph inspired veracity extrapolation so the truth let's have a look while the syn on graph excels in scenario where well structured knowledge grph exist give now tackles here a different but equally important challenge how do we reason here in this dual system way when the knowledge grow is sparse or incomplete I indicated this here that we have here in yellow here a partial knowledge graph incomplete sparse information how can we use for example here the parametric knowledge of the llm to build here knowledge craft that will enable us to answer the human user query and the answer is give we have here new publication October 11 2024 University of Pennsylvania University of California University of California Berkeley you see Berkeley beautiful let's have now a more detailed look here exactly at this one so gift Starts Here by breaking a query into key Concepts so if the question is is melatonine effective for insomnia the model identified of course melatonine and insomnia as key entities and then in the internal reasoning GIF now builds group of related Concepts and I will show you an example in a minute it so not just using the knowledge graph but also leveraging here the parametric internal knowledge of the LMS so for melatonine it might put in all the related ideas like sleep hormone supplements and whatever there is in your llm but GI doesn't just look for for known facts GI can extrapolate potential relationship what we more call here as a hypothetical link that could exist based on the patterns in the data but careful this is not a crazy hypothetical link this is a link that exists here but with a lower probability density in our distribution so hypothetically means more or less not as dominant imprinted here in a Knowledge Graph but maybe it exists or maybe there's an indirect link and I will show you an example in a minute and the beauty is and we know that we can improve at reasoning capacity of our our systems if we generate counterfactual reasoning examples so the system doesn't know only 100% yes yes yes yes answers but it also knows here no this is the wrong answer this is a false argumentation this is the wrong PA to argue so if we have the positive argumentations and the negative argumentation together we know that our reasoning experience improv significantly so this approach now of a guided extrapolation and I will show you this in a minute allows give now to tackle the Extreme Complex multihop reasoning tasks even when the underlying Knowledge Graph is incomplete and you might remember just some days ago I shown you here that Harvard presents a new knowledge grph agent for medical a systems check out this video you will find a lot of similar ideas a lot of implementations that are quite following some similar ideas so you see it all comes down it all converges now to new solution how to combine nlms with knowledge graphs yeah here you see Barkley States here the problem here in the publication so we have now a new question traumatic Artic injury my goodness does the anatomy of the Artic Arch influence and aortic trauma severity I have no idea but just go here with the official example so Chain of Thought and we get here chain of s lacks of internal knowledge prompt let see step by step yeah beautiful but gives us the wrong answer if we use the rag a text based rag with semantic similarity you know the coine similarity here in the vector store in the vector in the mathematical Vector space say here they are beautifully semantically similar the traumatic aortic injury and the atic orch and the atic trauma beautiful semantic correlated but absolute irrelevant information so therefore we fail with rag so if we have now our syn onav methodology what do we do fail to retrieve he on a spar Knowledge Graph it also gives us here the not correct answer because our knowledge graph is not complete and it's not complete to an extent that Sy on graph simply fails because they are missing links the system is not able to build in a multihop causal reasoning chain but wait we have now the new idea we have now the new methodology and you're not going to believe it they give us the exact L this answer what a coincidence no so let me make this one here so we have here terms in our questions we have terms like atic injury so we take this aortic injury and then we have what goes with this here in our knowledge injury poisoning whatever or we have here orotic trauma okay we go here we have here a new group orotic trauma injury poisoning clinical attributes beautiful or we go here with anatomy anatomy abnormalities structure tissue we built here this semantic group or the last one what I missed here I missed here the atic arch yes this one here so you see you take here all more or less all the elements that you have from the user query and the system tries to understand here the context of everything so we have here old and all graph Concepts and we have here one two three four different concepts and now what they do here with give it first builds here an entity group for each query concept and then induce inner group connection using its internal knowledge and then we will use cross group connection contained in the knowledge craft so you see what we do we build here our complete understanding and this is what I meant here with hypothetical links because those links are real but they are not in the knowledge grow for this particular domain argumentation chain so you see anotomy is part of here and influences this and this affects this one and here we have a location of and the cell function goes here is an interrelated functions and you see this system tries to build here new Step stones to be able to have a multihop reasoning and if one of the step Stones like the cell function is missing the system tries to find in the vast knowledge array exactly the step stone that would connect here in a logical causal reasoning way and built this stepstone for multihop reasoning yeah that's the if you want to have a look without my things great what made it click is here this single sentence by the aors and they say hey we introduce an additional intermediate note Group by picking the multi-step reasoning plans of the llm that are most helpful for the ultimate questions so whatever they are missing in their spars graph intermediate note groups that are related to the user query they select this from the llms multi-step reasoning plans and those plans have a multitude of possible pathway forward through the knowledge array so you see it is not a hypothetical link it is just a not yet chosen link here to build these new intermediate note groups highly sensitive to your domain knowledge highly sensitive to the parametric knowledge of the llm and of course to the structural information to the structural data representation in the knowledge graph now for my green grassers short explanation now this graph inspired veracity extrapolation now you understand it a little bit better is a knowledge extrapolation framework for structured reasoning of llms on sparse Knowledge Graph and give NE fusy on explicit information retrieval like our rack nor relies he on improving the internal reasoning ability of llms by app pending triggering statement to the query no not at all I just showed you what we do knowledge extrapolation we build new stepping stones for our multihop reasoning now it is interesting that those two Frameworks sync on graph and give they connect somehow no so first both integrate llms with knowledge graph a new innovative phace to enhance your complex reasoning both tackled the problem of multihop reasoning and while syn on gra achieves this by dynamically searching with the beam through the knowledge graph give does so by extrapolating here this new relationship and filling in the gaps of incomplete knowledge in our spous knowledge graft second they both aim to reduce the hallucination this is a common flaw still in our llms tog achieves this by ensuring traceability and correctness in the reasoning path of our knowledge graph while give ads here as I show you an extra layer of counterfactual reasoning which increases the overall reasoning capacity and both Frameworks are designed with the idea here that we have smaller llms like jet GPT 3.5 as the auto show us and they with this new methodology of Sy on grth and give can outperform larger mods like gbd4 in specialized task so this is nice that here our smaller mods maybe our open-source models are able to outperform with this new methodology how to combine in an Innovative way the llm with the structured knowledge of a Knowledge Graph this is really nice so key inside because it show that by integrating structure knowledge and reasoning here we have the knowledge graph here we have of the llm we can dramatically improve the model performance without needing to scale up the model size this is a nice result but of course you can use still here the Sy on graph and give here together or in particular user cases tog is ideal when you have a rich well populated knowledge CFT you don't need anything else if your knowledge graph is more or less complete for your particular query on your particular domain and fits in your complexity level you are good to go however on the other hand if you have a sparse knowledge growth if you say wow the complexity of my queries by the human or by the students is so high I better go with here a more powerful methodology especially when your structure data are incomplete and gives gives us here some real beautiful results so both Mets create flexible powerful toolkits for enhancing the llm reasoning independent if the domain is well defined then you go with train on graphs or the domain is still evolving and sparse then you go with give great this is the official end but I want to show you something else I found out something that amazes me