Harvard Presents NEW Knowledge-Graph AGENT (MedAI)

Key Takeaways

hello Community you know do you know that Harvard has a problem I mean a beautiful problem but since they just got here the Nobel Prize here in medicine Gary ruen professor of genetics at Harvard Medical School there's a lot on the line for Harvard and now they noticed here that llms have suffer from incorrect retrieval missing key information and misalignment with in scientific and medical knowledge and Howard writes you know additionally llm struggle to provide contextual relevant answer and it goes on and Howard writes specifically llms have difficulty combining scientific factual this means structured and codified knowledge with tcid this means non-codified knowledge and those llm powered question and answers model that we have available the best model on Planet those model lack such multi-source and grounded knowledge that is necessary for medical reasoning because this requires an understanding of nuanced and specialized nature of the medical Concepts and you say this this is a real problem for Harvard you and if you look here at the test and Howard published here some tests we have here a basic medical test and an intermediate difficulty and a real expert level test and you see if we look here at llama 3B we are here with the basic below 50% inmediate we are below 40% and the expert is below 30% with Al llama 38b gbt 4 Turbo is a little bit better but we are about at 50% so for a medical system this is not acceptable a 50% performance is horrible so what to do now Howard stated here that llms face challenges in complex medical reasoning because such reasoning would require here a simultaneous consideration of dependencies across multiple medical Concepts and I will show you those Concepts in a minute plus a precise local IND domain knowledge of semantically similar Concepts that can carry different medical meanings and we will have an example this so I know what you go on to say you say okay so prevailing strategy to address these challenges here is of course like in the community today we have some rag some retrieval augmented generation yeah you know Howard discovers that also those R methods can provide multisource knowledge yeah they are really vulnerable to potential errors and Howard goes on and says you know all those data repositories and the knowledge basis those rag malls draw from they contain incomplete and incorrect information leading now here in har Medical School inaccurate retrieval for their specific cases and how it goes on you know further many rack based methods simply lack a post- retrieval verification mechanism to validate The Retreat information is factually correct and does not miss some key information especially here in medicine so you see now you understand when I say Harvard has a problem and Harvard says you know to advance now these L empowered EI models here for knowledge intensive medical question or answer it is now essential to develop models that can consider complex associations between several medical concept at the same time and systematically integrate multi-source knowledge and effectively verify and ground the retrieved information to really make sure that the contract relevance and the accuracy is there because we are working here in medicine we cannot allow to have one single mistakes in those llms so Howard develops a solution and this here is the beautiful study we have here Harvard University University of Illinois Imperial College LA and fiser hello fiser Howard writes here knowledge graph-based agent for complex knowledge intensive question answering in medicine and they developed a new methodology and they write we introduce a knowledge graph-based agent designed to address the complexity of knowledge intensive medical queries so what we are going to look now is we have a medical large language Mall all its shortcomings particular in the area of rag and we will now have a detailed analysis how we can integrate the knowledge of a knowledge grth and bring them together to have a much better performance in medical reasoning you notice medical llms they have kind of a problem because they have a lot of non-codified knowledge whereas at a knowledge graph we looking at structured codified knowledge of medical Concepts and of course we do not have maybe one Knowledge Graph but in some medical disciplines we have multiple Knowledge Graph so we have to come up with a solution sorry I mean Harvard has to come up with a solution to integrate multiple non-coherent Knowledge Graph and somehow make them work with a medical llm so as I've shown you on this test here the expert test so the question again is there an interaction between a heat protein 70 family that acts here as a molecular jaeron and the gene or protein implicated here in a particular case is something that related to a mutation and there's only one answer but in the test you have normally multiple choice here for the medical students so great now they look here also at multiple choice test but what I will focus is of course the open ended question where we do not have multiple choice because in medical research you don't already have and please choose from four answers because one answer is guaranteed to be the correct one no we go open-ended so goal is here to identify the correct answer using here an llm and a Knowledge Graph to come up with a better causal reasoning in medical questions and they say here is now our new idea so we have here a question let's say we have four answers or maybe no answer at all and then we have four faces we generate something re review something we rewise something and then we can answer everything so it's a relative simple concept but you know there's some inherent Beauty so let me Focus here on the methodology because this is great so at first we are at a generate level this is designed here to prompt the llm to follow different procedures and I will show you these procedures for generating relevant triplets and you know triplets from graph Theory so we have here as I told you two kind of questions the choice aware questions and the non-choice aare questions and in both we generate triplets goal is here extract medical Concepts in the question stem of a particular question plus here also have a look at the content of each answer candidate now the concept extraction the llm extract now a medical concept from the input question this this one here identifies now all key entities and relationships that are relevant to the question and this is done by the llm so from a query like hey what is the relationship between hypertension and heart disease the llm identifies now the medical concepts of hypertension pares plus you know a triplet here in a graph we have a relation included in the middle and this relation here is causes or is associated with so the generation phase is simply to create a triplet So based on the identified medical concept and I will show you how we do this the llm generates triplets in the form of h r t where H is the head entity R is the relationship between those entities and T is the tail entity this is what we know from graph Theory so for instance a generat triplet might be here hypertention causes heart disease HR T so we do have a structured format now for those different kind of question if you read this publication it's a beautiful publication there also focusing here on multiple choice questions I will leave out here the multiple choice because I will focus here an open-handed question where we do not have a solution given but there's nothing so we have to come up I mean the AI has to come up with a medical solution involves generating triplets directly from the question without we have answers a b c and d okay so now that we have generated a triplet and show you how we do this let's have now a look at the second phase this is now the review phase or the Rev Q action phase here it aims to assess the correctness of the generator triplets by and now we have the connection to the knowledge graph by grounding here the correctness of the triplets in a medical Knowledge Graph we do this to ensure that the relations and the entity produced by the llms are factually correct and consistent with medical knowledge so you see we have now two different system an alab and a knowledge graph and now it gets interesting and there is now a specific way to map entities between those systems and Harvard decided to go here with umls code unified medical language system code so each entity in the generated triplets is mapped to the corresponding Concept in the umls and this step ensures that the medical concept identified by the llm are linked to the standardized medical terms via the umls codes which can then be matched with entries in the knowledge graph if you have never seen an umls code for example some real simple diabetes MOS you have a concept unique identifier a simple codebase then you have to official name then you have synonyms and then you have different sources then you have a lot of treatment option and whatever so here we have codified knowledge standardized medical terms and what we do now is we just jump here from a medical llm over with this complexity in the semantic knowledge to a Knowledge Graph no we have if you want a filter and we say hey everything that is consistent and has a unified medical language system code from the official directory this filters out that only the correct medical terms come over to the Knowledge Graph and if the medical llm hallucinates some I don't know technical terms that is not with the code that does not exist well then it is filtered out and not in the next stab okay now we are retrieving pre-trained Knowledge Graph embeddings and you say what yeah it's beautiful so for each matched entity and relation the pre-trained embeddings from the knowledge graph on out retrieved so these embeddings capture now the structural relationships between the medical entities in the knowledge growth and you say hey wait a minute I have a Knowledge Graph and you are talking about embeddings yeah if you're not familiar with this never mind this is from an old methodology this is from 2013 this is the first time it happened we have a lot of um embeddings functionalities I would recommend you this particular paper this is the original trans e paper for translating EMB addings for modeling multi- relational data from cnrs and Google 2013 about 2013 and they face you the problem of embeding the entities in relationship of a multi- relational data into low dimensional Vector spaces so they have a graph and they want to find if you want here Vector embeddings in low dimensional mathematical spaces here I've given you here if you want to read the paper in detail it is so that unfortunately now it is only behind P walls of the speak Corporation but with this link you can read it free of choice and in this transm adding they give you the translation based model and they say give him a training set of triplets HLT composed of two entities head and tail the say this is the set of the entities and a relationship now they call it l not R whatever this model learns Vector embeddings of the entities and of the Relationships by doing here simple optimization problem so you see we have a method to find the embeddings in a mathematical Vector space of a knowledge gra so isn't this beautiful so what we have we have now officially from the large language mod from chat GPT or whatever you use here we have the llms internal embeddings if you use gbd4 you have to pay for those embeddings and we have now from the knowledge graph also structured embeddings as given by trans embedding so we have now a free text information and a graph structure information and we have brought them at the same level of an object of embeddings and now we have to task of fine tuning the llm on a Knowledge Graph completion task as Howard writes so what is this this is a key aspect of the methodology is now to F tune our llm on the basic knowledge of the knowledge graph that is now added to the system which involves then if we have a find on M to predict missing relations or entities now back in the knowledge graph I will show you this in a minute but just on a theodical level we have now llm embeddings and we have Knowledge Graph embeddings because the task is to somehow bring those together that one system can learn from the knowledge of the other system so how we do this there's something like an alignment phase and you know we always had this problem in the classical phase so the alignment between the llms token embeddings and the knowledge graph structural embeddings this is NOW essential to combine here exactly also the unstructured n of the llm with the structured domain specific medical knowledge found in a medical Knowledge Graph so how we do this well it is easy we simply add a projection layer for example now we do this on the token embeddings which will adjust their dimensionality and their structure to match here the format of the knowledge graph embeddings so this means we have a mapping transformation or it learns a mapping that brings the llm token embeddings and the knowledge graph embeddings into the same geometric space this is great we can normalize it and then we can work with those embeddings because now those embeddings are really compatible coherent structure and in this way the M can now combine the semantic meaning of the llm embeddings with the grounded proven structural relationships that are encoded in the knowledge craft so now the lm's token embedding and structural embeddings they are now harmonized and once the token embeddings are transform we have the classical attention mechanism that you know that you love whatever Transformer use that has attention self tension multi-ad tension it helps not the m to focus on important parts of the input and in this case it helps align the lm's token in badings with the knowledge graph relational information and it simply means it is asss more weight in the tenso structure to the important parts of the triplet now learning thereby how the lm's understanding of a hypertension should map onto the knowledge graph representation of the hypertension and the similarities for heart disease and related causes so now we have if you want a new embedding created that is beautiful we call this the aligned Bings of both systems so after the transformation layer and additional at the attention layers and the execution the final result is now a new set of embeddings for the triplet components which we call now aligned embeddings and these embeddings capture now both kind of knowledge we have the semantic meaning of the pure text is now understood by the llm and the relational structure information as encoded by the knowledge graph in a graph form from the graph Based training is also now available in those embeddings of those aligned components isn't that great so therefore we have now an improved reasoning because we do not have any more one system llm and one decoupled system Knowledge Graph but now we have combined the knowledge of both systems so the aligned embeddings are crucial for task like Knowledge Graph completion where the llm has to predict missing relations between the entities and the knowledge graph so by aligning the two types of embedding the llm can now reason about the knowledge graph with an improved accuracy and an improved understanding that it has all now the structural information of the medical Knowledge Graph to for example ground the knowledge let me give you a practical example so if we say here this knowledge graph completion task in which the model attempts to fill in the missing links in a portion complete compl knowledge gr and this is a problem that we have almost every time especially in medicine you have never a complete knowledge medical knowledge Gro there are always sometime some parts of information missing so we always have more or less partially complete Knowledge Graph for a particular topic for a particular task and now if we can now fill in the missing links and make this knowledge gr if you want more complete this would be great so let's do this the llm is now fine-tuned in order to predict here the missing relation between the entities that already exist in the knowledge grath or to predict missing entities in a relationship if only one entity and the relation are known so simplest examples hypertension associated with and then we have not here the end note for this in the relation or we have question mark causes heart disease and now we can predict the missing entities of those two elements so you see we complete the knowledge graph another step another level great so here we have it we have now since we have two system we can do now if we have an improved knowledge we can have now two options we can either update the knowledge graph so if the purpose of the system system is to continuously enhance the knowledge graph then the new predicted relation and entities that we found with the combined knowledge can now be added back to the knowledge graph we make the knowledge graph more complete let's call it Knowledge Graph plus this results see my last video where I talked about Dynamic evolving knowledge CFT that are now more complete more up toate can have a higher complexity level to have a better argumentation even in higher complexity structures plus all this Insight gained from the llm reasoning are now Incorporated back into the graph for the future use so if you want we have here absorbed all the knowledge of the medical llm and we put it now in a new knowledge graph that is much nicer much more powerful or we have the option b like here in the publication for Howard the inside staying with the llm only they say for example you say hey I just need a better llm for argumentation for causal reasoning so my goal is to get a medical llm plus so the knowledge graph remains static and unchanged in this particular option serving only as a source of structural knowledge that the llm can draw from and you bring in I don't know 20 different Knowledge Graph from 20 different medical subdisciplines you get the idea your medical llm plus will be a llm Plus+ Plus plus great so in this case the llms learns from the knowledge CFT using the fine-tuning process to better predict relationships and answer the question but the new insights are not not fed back into the knowledge graph so whatever you do great now there is of course an option C not described here in the publication that you could theoretically do both no you update your knowledge graph and you get better medical llm plus but then we would have complexities that I would like to show you in a later video so now we are after we have done the review and we have done all this now there is also the revise phase or action phase and this happens if the triplets that were generated are found to be incomplete those triplets are found to be incorrect so what we do theoretically just throw them away but Harvard tells us hey maybe there is some hidden information in there so what we do we keep those but we get them a special treatment and now since we have this super llm the llm plus attempts now to adjust incorrect triplets by proposing here new relations or new entities in our triplet structure based on related medical Concepts so whatever was learned is new also integrated in this new knowledge base and then the llm will call this triplet iteratively where the llm continues to propose and refine the triplets until the triplet are now validated Again by the knowledge CFT so you see now if you have a standard knowledge CFT that is not dynamically evolving and increasing its complexity based on the reasoning structure of the llm then sometimes the validation by the knowledge graph you will reach a plateau given by the complexity level that a Knowledge Graph can handle but might be necessary to validate here a very particular special triplet configuration that an llm generates so you have to make sure in short that the complexity level of both of your system the llm and the knowledge growth are about the same or you must know hey my llm has a complexity level of 6 out of 10 but my knowledge graph has eight out of 10 you have to know what procedures to follow great and then of course the last phase here is the answer phase beautiful so everything is great we have now a lot of different answers and now the model simply and let's go here as in the publication with the medical llm plus now this model selects now the best answer based on the verified triplets from the review process and now from all this revision if we found some better review stages so the final answer is determined by considering the relationships and entities validated now by the knowledge graph and this ensures that the answer is not only plausible for the llm but also factually grounded in the medical knowledge as presented in the knowledge graph in the medical Knowledge Graph so you see the beauty of this I hope so so for my green grasshoppers I try to have here five important points for you this is the summary for a beginner what we do we have an integration of llms with domain specific knowledge graphs and you have a certain knowledge encoded in the language model and you have a certain maybe different knowledge encoded here in the knowledge graphs and now you want to bring them together a better system Five Points combine the free form reasoning capability of llms with the structured domain specific Knowledge from our medical knowledge graphs resulting in a system that can reason more effectively about complex medical Concepts I told you that the use of these umls codes as an intermediary between the llm generated Concepts and the knowledge graph allows for consistent and accurate mapping of medical terms the fine-tuning of the llm on and Knowledge Graph completion task enables the m to bridge the gap between unstructured text and structured knowledge also leading to improved reasoning and as I told you the alignment phase you have the token embeddings from the llm and the structural embeddings from the knowledge graph allow the model to leverage both the semantic and the structural knowledge improving also the accuracy of the reasoning and especially beautiful I think is the error correction via the revision mechanism now this is a try on error maybe there's nothing there and llm does not find is not able to solve or decompose here this complex triplet but maybe it is and therefore we do have an error correction so this iterative revision action ensures that the M can correct its Mistakes by leveraging additional Knowledge from the knowledge graph leading to a more robust and accurate answers so nice ideas beautiful ideas into here and I immediately have five other ideas how we can build on this and further improve this but the goal of this video is just to make you familiar with this study by Howard and fiser Howard calls this methodology the kga Ravon so Knowledge Graph a ravion don't ask me for what it stands and enhances here the medical questioning answering by leveraging the alms grounded now with the the knowledge of the knowledge graph to the noble mechanism we just went through and they State hey unlike the standard rag approaches our new methodology doesn't directly retrieve information from the knowledge graph and this is so important to understand this this is not rag instead Howard writes it uses the llm to propose a potential relationship between medical Concepts extracted from the question represented as triplets this is not as easy as you might think and I would like to show you here an example but you see this is it in a nutshell here on this new methodology what is so beautiful that the llm is not constrained now to existing relationship in the knowledge graph it can propose your novel connection based on its language understanding and as I told you if you go here for a medical llm plus Plus+ or whatever this this here really just uses here the knowledge of the knowledge graph and then you can exchange the knowledge graph with a different subgraph or a different medical graph or a different or higher complexity Knowledge Graph so you see depending on what system you want to nurture on you have unbelievable options in front of you and the code is rather simple let me show you this now before I show you the code I want to show you the result ladies and gentlemen the results are now now H choses here also the multichoice so if you have a b c d and you have to cross a is the right answer reasoning my goodness so the number of medical Concepts the complexity of the concept is here on the xaxis up to six medical Concepts and there simply the accuracy of the system is given on the y- axis now here you have in the light blue a llama 38b model and if you take the same llama 38b and you apply this new method ology you see that you go from I don't know 50% to about 70% plus nice however if you look at the Llama 3.1 and you do the same stuff you see that the general curve shifts upward here suddenly instead of 05 we are now at 07 but the increase Now by this new methodology is not as significant as it was here on the llas 3B and un Unfortunately they do not have the data or publish the data of llama 3.2 interesting is however that if you see the more medical Concepts we bring in into the question the higher is the complexity the more multi-dimensional medical Concepts we have you see here this is nice unfortunately it's just one data point but you see the more medical Concepts we have the better this new methodology is and the less potent the classical llama system is you see the higher the number of concept it goes down down down it goes down down down okay but if you go now to open reasoning let's have a look at this again llama s8b here the light blue and here you see this new methodology is not as good and look it even if we have here multiple number of medical Concepts that we integrate in the answering of our medical question there's only this the same performance with and without this new methodology and you might ask is this based on this particular system or is this because the methodology is not working and then it's interesting they give you here the second one here and they have now the Llama 3.1 the 8B here and you see the more comical Concepts we have the further it goes down but here it separates and it has here kind of an equilibrium but it also breaks down here if hey wait a minute if we reach here a certain threshold so this tells me there is a high sensitivity to the underlying llm if you go with a llama 38b Or llama 3.1 ATB and hopefully with a llama 3.28 B we would see significant difference if we applying this methodology but unfortunately the latest models are not available for us and Harvard writes here they used here a cluster 4 Nvidia I think it was an h100 GPU configuration so it was not that complex to do all of this calculations here in this demo and I wanted now I want to close this now here with the prompts and a lot of questions are received here how complex are those prompts so have a look at the prompts this is here the first prompt the very first step here to generate here our if you want medical triplet but before we generate this you remember I told you they want here if you want to fill here the search space with enough information so what they do they say give them here a for example multi-choice question extract all relevant medical entities contained within the question stem so this means identify and extract all medic entities such as disease proteins genes drug phenotypes anatomical region treatment or whatever relevant medical entities there are and put all of this in a list with the key medical terminologies so you see you have now not only one particular term that you look for that is in the question but you look for the complete embodiment for the complete surrounding the topological cluster around your particular medical term so this contextual embedding if you would like to call it this is done here in this particular prompt second time prompt is really we generate now here our triplets so the prompt for generating now the action phase so we say given the following question stem and the medical terminologies that we just created so this is here the list of all the surrounding if you want gener a set of related undirected triplets each triplet should consist of a head entity a relation and a tail entity and the relation should describe meaningful interactions or association between the entities in the biomedical context and now you can even say that you want to specify those relationship and Har tells us yeah hey the relationship should be one of the following protein protein interaction carrier enum Target transporter contraindication iation off label use energetic whatever so you define your valid relationship if you want this is kind of another filter that you want to apply for your particular task you know here the complete set of possible interactions and then the task is hey just generate one to three triplets for each option focusing on the ones most relevant to answering the query only return to generat Triplet in a structured Jason format with the key triplet and a list oh you have here head entity relationship and tail entity beautiful so what we have we have the question the user question then the list of the medical terminologies that we just created in the step before now we get a response by the llm so the llm generates here the kind of complexity triplet structure and here also the better your llm the more the llm is able to think and reason and detect and create complex more complex triplets the more intelligent the system could be yeah if you want to see The Prompt for the revise action I was interested in this and just simply say hey given the falling triplet consisting of a head entity of a relation of a tail entity please review and revise the triplet to ensure it is correct or ensure the rise triplet is factually accurate and contextually appropriate great so now the llm is trying here to find a new triplet configuration new terms new relations that might be voted correct here by the knowledge from the knowledge graph and this is the end so therefore yeah I hope I have given you here an insight into a complete new methodology that horard Howard Medical School is utilizing here in this research publication it's brand new for me it is just 3 days old and I think it's beautiful and I thought about replacing here all my rag systems also with this new methodology because it is so much more powerful and if I don't need rack maybe that's maybe not a bad idea if I can substitute it with a more powerful methodology but you see more or less simple idea de formulated in a coherent way four phases and they show you hey we have a much better performance of our systems for medical question and answering of course they are not there yet that it will be 100% correct but this is already The Next Step I hope you enjoy this video I hope it was a little bit informative and it would be great to see you in my next video hey Siri speed is this then