Radiology AI Project: Part 2 (Segmentation & Entity Extraction)

Key Takeaways

Hey, how it's going guys? So, you know, in this new video of the radiology AI project playlist, I'm going to include couple of new features that I've been working on. Uh, one is the segmentation part. Uh, segmentation is one of the most important aspect of you know when we want to build uh AI solutions for medical images, right? uh segmentation basically helps segments uh you know some of the key objects or you know uh key parts or crucial parts uh in a given image. So we're going to do segmentation using midsam. SAM is a meta model which is segment anything right that's the model they created and after that we have seen medical variant or medical version of uh midsam. So I'm going to do segmentation and also going to extract some entities from the AI outputs you know the uh the analysis result just to uh and of course I'll I'll tell you how we're going to use these entities or keywords in the in the in the new uh new features that we're going to implement. So if you look at here on my screen right I am on radiology AI agents. Right now we are not uh building this as an agentic solutions. We'll build this in the upcoming videos. Right now we are creating you know uh individual functions or modules and later on we're going to use probably you know parentic agno lang graph whatever you know to use these as tools. So you have to wait for that. But if once these are ready we can just use this as you know part of tools. That's it. Now uh if you see here right we have uploaded an image of pathology it's a pathology image and of course I got this from uh of this uh meds folder only. So let me show you the metam. Okay. So if you look at here okay I took this image called pathology PNG. So I have uploaded this image here and of this some question. You can of use any other user questions user prompt you want. System prompt is already there. I will show you now. This is the image preview. Right in the center we have analysis results. It shows an AI output. It says the image shows a microscopic view which is correct of colon. So it's a colon. Uh there are no obvious gross abnormalities because I have asked about abnormalities in the question uh visible on the provided image. However, I can point out some potential findings that might be relevant depending on the clinical context. We haven't reached on the clinical context part yet where we're going to have you know some more information coming from kind of a rag kind of a system. Right now you can see inflammation uh uh crypt absesses and granulomaas or something right granulomass uh if present there would suggest a granomatus condition like Crohn's disease uh the presence of inflammatory cells neutrfiles or fails within the lamina proparia could indicate inflammation such as colitis or infectious process crypt these are collections of neutrfils with the crypts I hope I'm pronunciating it right of of liber and can be seen in inflammatory bowel diseases IBD. It is important to note that this is only a microscopic view and further clinical information is needed. So we have put certain disclaimer because you cannot trust 100% right these AI models LLMs multimodel models let people talk who are lobbying this right I mean see you cannot use AI for decision making in healthcare it's extremely dangerous right at least at this point of time they can perform really well on benchmarks and evaluation criteras but when it comes to real world use cases in real world patient data and use cases you cannot make decisions 100% what it can do guys better is it can empower the domain experts or medical practitioners in this case pathologist, radiologist and you know so on and so forth who are working in this domain it can empower the technology can empower them the AI agents can empower them make them more productive more efficient so you know they will be doing their job like you know as a uh like they will have a companion I'll say a co-pilot kind of a setup for them so that's what it can do the end decisions will always be you know from a certified expert you know in this in this field. So keep that in mind you know if you are somebody who was building products it doesn't scale guys to be honest. So you know if you look at here we have AI output we have extracted keywords. Now we're also extracting some keywords and these keywords I will use it later to kind of find out some uh similar case studies from different data repositories like PubMed, PubMet Central, NCBI, radopedia so on and so forth right we'll see how we can bring some information you know over here and you can find it out over here microscopic view colon tissue blah blah blah now we also getting our AI segmentation here you can see the input image and bounding box and then midsam segmentation so we have this midsam segmentation you can see the color levels yellow over here right seems like there are a few stuffs going on now I'm not an expert but if you're looking at this image looks like some inflammatory cells uh you can see it also says within the lamina proparia could indicate inflam such as colitis seems like some colitis that it has labeled it you can find it out over here right the segmentation part you can also download this of course the download feature is there so we have moved from you know previously we only had the AI output we didn't have the extracted keywords and we didn't have the segmentation part. So in this video we have added segmentations using metam and keywords. So I'm going to cover these two features programmatically. I'll show you how we built it using uh key. Key is a of course a library that uses B models to kind of do semantic aare entity extractions. You can also do it with spacy as well with NLTK and stuff but you know I wanted to have something semantic aware. So the bird model have a bit of understanding on the embeddings level how the semantic similarities are there within the engrams. So I'm going to talk about that. I'm not going that technical right now. And segmentation through meds of course let me talk about metam uh metsam over here. Medsam fantastic. Uh let me open it now. You can see it's already has around 3.6k stars. Official repository for medical segment anything in medical images. You know they have now support for 3D and video segmentation. You can give video, it can segment it. Imagine right. So we are right now dealing with the frames uh like image frames uh you know videos are also collection of frames. So we can directly give videos. Let's say you are doing ultrasonography, you are doing chronoscopy, you are doing laparoscopy, you are doing all of these things, right? You can directly pass these videos. It can segment those objects or some whatever focus part you want to do right in the segmentation uh scene. So the documentation you can see there are some have some documentation. That's what I was talking about right. talking about these things seems like an endoscopy or something over here going on. Now if I come here to show you the architecture this is how the architecture looks like. If you look at an input image so we give it an input image. Now once you give the input image it has an image encoder of course creates the features basically extract the features right through an embedding thing. You can see it over here and then it has a mask decoder that basically create a mask around the segmented objects. You can see the segmentation over here. Now these are important because earlier we used to do manual segmentations right when we used to you know work with tools like label img and you know lot of other VG a lot of other images uh such tools right to do segmentation kind of a stuff we used to work with unit architectures and stuff to kind of work with segmentation features uh in some kind of medical imaging based AI products now this is by Bobang lab so credit goes to them for metsam. I have used metsam. Let me walk you through the uh you know I'll just close this here. Now let me show you. Of course I was like taking some help. Now let me show you the nodes txt. On the nodes we have entity extraction using keyword and midsam for segmentations. There are other things that I want to build but I'll we'll go you know slowly gradually over there once we reach it. Now first I'll show you uh our entity extraction file. So if you look at here entity.py Pi we create it as a module and use that. I'll just get rid of this file. I don't need it anymore. Let me just delete this file. Smoke to trash. Okay. Now entity.py we are using key. Keyboard uses of course transformer models like B models to do semantic aware uh entity extraction. So keyword extraction as I said I have a very simple function over here. If you look at this function it's called extract under underscore keywords. it passes the text which is a text passage or the paragraph or text sentences whatever number of keywords like I want to extract let's say 10 as default and a similarity threshold which that's what I was talking about I was talking about context aware similarities right now which has understanding of uh contextual information uh you know and how that is works guys based on of course engrams right if you don't know engrams read about it don't directly dom and all which is which will not help you in your career if you want to really play a longer game in this industry right you have to understand the fundamentals now if you look at your engrams and stop words and English and I have this function that gets some things I'm using all mini language model L6 V2 one of my favorite not favorite I will say now we have BAI Beijing AI academy embeddings model we have Nomic uh we have lot of embeddings model uh Gina embeddings model and whatnot with the larger dimensions but when we used to work four or five Few years back when there was no such hype on generative AI right we used to work with these embedding model one of one of the early embeddings model we have seen with the good dimensions uh from sentence transformer we are using it we then encode the keywords right and convert it to some tensions of course getting the features here and then also looking at the duplicates that's a logic that I integrated here because sometimes it gives you the same words you know uh duplicates we that's why we are doing this uh duplicates looking at the similarity threshold And if there's any similar words we just uh we just remove the duplicates and only return that. And you can see a sample example here prediction and then it extracts the keywords. This is entity.py. Now we have used this entity.pfi in our app.py here. You can see we are using from entity import extract keywords. If you look in line number 12. So let me just go below and show you that uh there are some the system prompt. In system prompt you can see we have the same thing you know you can see bounding box. This is for we also downloading the result from segmentation. So this is some kind of data validation using pyic you know a bit of structure on how we should get the output. I'm not covering this upload diccom and anal uh upload diccom thing because we already have you know explained that in the previous video. Now here if you come down right you can find out this particular thingy which is keywords. It starts from line number 169. You know if you look at here it says extract keywords using the entity extraction feature keywords extract keywords give some numbers finding out analysis text keywords. Even if you give let's say number of keywords five it might be a thing that the confidence score is not greater than 75%. So that's why you might not get five keywords you get might one or two or three keyword but that's that's what I want because I want to take this keyword and then perform some kind of web search or web information discovery. I'll show you that now. num keywords a response and you can find it out here over there that's it right and then we have our segmentation thing so let me go to segmentation now I have a similar file called medsam.py Pi you know I'm not having CUDA in my device but if you have CUDA you can use it. We are using SAM from transformers module. So we are using SAM model and SAM processes processor through transformers library. I recommend that to use it and you can see we are using this model called metam vit base. We are using the base model of v metsam vit a vision transformer you know it's displays a mask on a metro lip axis because we're using metro lip for mask and boxes. and we just download that image and show it on a react UI. That's what we are doing, right? So, uh very simple. These are all mattly things that you should not probably care about like mostly because these are like in available. Now, this is our midsam. The same goes in app.py where we are using midsam module. You can see if you go up, we are using from midsam import generate segmentation and we are using the same thing, right? We are using the same thing the way we use keywords uh doing the same stuff uh for the segmentation as well. You can keep going you know you can see let me just keep going here extract keywords and you can see the segment uh line number 271 we have an API that we created for segmenting the uh the thing you can see generate mid some segmentation for our previously processed image. It takes the file ID these optional bounding boxes blah blah blah. You can also do your own uh annotation for segmenting it. So I don't have that feature yet but we can probably add that you know if required. Yeah but it generate the it just you uses the generate segmentation function you know to generate the segmentation and pretty much straightforward guys you can also do direct segment if you don't want to directly use it like let's say you want to use it as a separate API endpoint you can also use that. Now if you want to find out where are those API running it's running so on 8,000 because we have a fast API based API right you can see we have diccom processing we have image analysis we have image access we have direct analysis we have image segmentation so we have direct segment as well we have segment through file ID depends on what you want to do right if you don't want to including include in such application that is also fine if you want to use it as a separate endpoint but that's what it is now if I go back here in the front end we have couple of separate components. If I go to components, you can see we have segmentation view. You know the segmentation view takes care of segmentation thing uh that how do we present it on the screen. We have keyword tax in the keyword. I have followed a bit of air table uh kind of a styling. If you look at the tax in the air table, right? It looks really good and I like those kind of bit of coloring very minimal. If you look at this, right, I ask wind surf you know very agentic ID, right? Wipe coding kind of a thing. I asked it to give me a framework and of course I made a lot of changes because it was giving me error and I asked it to create something like superbase or verso if you look at these designs right very minimalist I don't like a lot of colors and because it looks very cluttered on the screen very simple you know you have a question we have three columns layout you can see image preview analysis result and AI segmentation we're going to keep on adding new features over here and guys this is not by any means a product is beyond all of these This is more of a you know a UI wrapper based on our functions that we have been creating in the back end. You just have to show the uh on a UI right there is no UX here if you look at in that way. There a bit of UX I will say but which is probably very a knife for a product. But if you are somebody who's working on vertical SAS or if you're building some stuffs right or vertical agents not even SAS let's say uh if you're working on on in that maybe you can extend this. I'll make the code open source of course when this is completed. But let's go back. So you can see we have some tag colors over here. We have given some colors and then it finds out if the if the keyword length is not zero or not null. It basically just implement that right. It just prints that if in that lemon terms. So very simple guys uh and I'm using UV. These are the requirements that we have. I think better than requirement we saw the pi project 2 ml. Here we can see we have transformers torch everything installed right and it's running. So come back here. So so far what we did let me just write it over. So we have done uh llm output right for uh finding anomalies. Uh anomalies here let me write it. Finding anomalies we did uh yeah I think these three only we did right. Let me have a look. And we have an diccom right diccom to uh JPEG. uh converter. So we have an image preview. So let me just write it image. So we have these things already done. Entity extraction you know you can find out the entities over here. And then we have our midsam for segmentation. So that we did here midsam for segmentation. We have llm output. So you can see the llm output over here. And then we have diccom to jpeg. Even if you upload a diccom image, it will first convert it to JPG or JPEG and then you can do all the analysis uh in the task and image review. Next, what I want to do is uh upcoming, let me just write it here, right? I want to use uh external data sources uh for finding similar cases or research uh research based cases for uh given set of keywords. uh and present a summary to radiologist or to end user rather. This is something that we're going to do in the next video. We're going to focusing on this, right? Uh to bring up this I think we're going to add this here. Probably we'll add somewhere below. Uh and we do it. Now you can also of of course upload a new image if you want. That takes a bit of time. So I'm not doing it here for saving some time. But you can play around it once the code is available. Right now the code is not available but I have a GitHub here. You can see we'll update it on GitHub guys. I know you want to use it. We're definitely going to do it. Uh because this is of course not being completely open source running on a very simple MacBook Air runs on 16 GB of RAM. We are doing already met SAM around weights around like uh 1 GB of size. If you if you count all the safe tensors uh we are using GGUF4 LLM which are which are we are using Medjma right? So you can see we are using Medjma here. That's what we are using medama 4b instruction tuned model and we are we are using this as an API endpoint that being served on this particular port 1234 using lm studio SDK. So that's what we are doing. We are using LM studio SDK for our uh you know LM thingy. That's what we are doing right if you look at here we are using LM studio as LMS. So we have those model that's running uh you can see here. So this is what we do you know to build this completely open source we are using keyboard we are using meds sam we are using medjma completely open source tag guys no APIs nothing to pay here if you have a decent laptop you'll be able to run this and help yourself if you are a domain expertise radiologist pathologist or clinical someone okay who is working or if you are building it for some healthcare providers take this extend it your choice uh wait for the next video where I'm going to bring up that thing that I told you in the tasks you know bringing up you know information discovery or a similar case findings. If you like this video please hit the like icon. If you haven't subscribed the video yet uh subscribe the channel yet please subscribe the channel. Uh share the video with your friends and to peers so they also aware and they can build some cool stuff. Uh if you have any feedback thoughts or if you want me to add some other features that might help you put it in the comment or you can also reach out to me through my social media channels. Find those information on channel banner and channel about us. Let's cut this guys here. Thank you so much for watching. See you in the next one.