Lightning Talk: Biomedical Visual Language Learning

Key Takeaways

hello everyone um Welcome to our talk on a biomedical visual language processing um my name is Stephanie Hyland I'm a senior researcher in the health Futures team at Microsoft research this is my colleague Fernando hi I'm Fernando Perez Garcia I'm a senior research machine learning engineer at the biomedical Imaging group Microsoft research Cambridge um great so yeah thanks for being here over the 20 or so minutes we're going to talk about our work in this space and also show you some demos of our technology okay let me get started so as an outline and what we're going to talk about today is why do we care about Vision language learning in the first place we're focusing on a use case from Radiology which is the area that we're focusing on um in particular we're going to touch on what are the domain specific challenges of vision language processing in the biomedical specifically the Radiology space and then we're also going to talk about how we've addressed these challenges and how we're enabling further research through um sharing code sharing model weights and other open source efforts including Benchmark data set releases so let's start with our Radiology use case so if you're not familiar with Radiology essentially the way it works is an image will be taken of you so let's say a chest Experian MRI a CT this image will be interpreted by a specialist a radiologist who will then write a report summarizing what they've observed their impression and their findings so one of the challenges in Radiology is that there are significant backlogs due to essentially a lot of work that does needed to be done and also the highly specialized expertise of Radiologists and where they're essentially not enough Radiologists to report on all of the images that are being taken and so this is this is resulting in of course delays in people accessing care but also in incredible burden on the existing radiologist to do more work in less time which can result in safety safety issues and worker burnout and yeah various uh terrible Downstream consequences so of course for your machine learning researchers were interested in understanding how we can help with machine learning but of course machine learning typically requires a lot of labeled data and in especially in the medical domain label data is quite scarce so what we're proposing here is of course to do cell supervised learning by exploiting the fact that we have paired images and reports but there are some challenges associated with this so in the general domain there are perhaps access to more let's say Foundation models large models pre pre-trained language models which you could use free trained image models which you could use in a Cell supervised pipeline um we also have smaller data sets so we don't have billions of examples we have of the order let's say 200 000 image report pairs in the data set that we're working with and we also have particular challenges associated with the text domain so Radiology text is quite different to General domain text which means that we can't simply transfer existing General domain Vision language processing methods over to Radiology or to biomedicine and expect them to work so let me talk a little bit in more detail about this last point so Radiology has particular nuances so as you can see this is this is the image and this is an example report some of the challenges that we have are it's very common in a radiology report for the absence of something to be described if you think about a general domain image captioning data set this is quite uncommon you wouldn't write there is no dog in this image whereas in Radiology it's very common to say for example there is no pneumothorax we also have relatively long range dependencies in the language so sentences could be quite long and they could refer even within the same report to different sections we also have data in the text domain which is unrelated to the image let's say so for example they may say I discuss these findings with a doctor or potentially I recommend a follow-up scan these things which may pertain to the image in some sense but may not share semantic overlap which is going to introduce noise during our self-supervised training pipeline so these are some of the challenges associated with doing um Vision language processing in the Radiology domain to address some of these challenges we have we have not a nurse but at eccv in 2022. so this is the work with many colleagues some of whom are here in the audience and who would all be very happy to discuss this work with you if you have particular questions but I'm going to go through the high level approach what we did in this paper in the next couple of slides so uh what's key to the um cida here is about thinking about how can we better adapt our text models to the nuances of radiology tech so that we can enable improved supervision during the vision language self-supervised training so what we do here is we have a domain-specific pre-training phase for our text model so in particular what we do is we train um we train on radiology and clinical and biomedical information from PubMed mimic 3 and mimic chest x-ray we also develop a specific Radiology vocabulary because if you use a general General domain tokenizer you may actually break up important Radiology terms so we have a domain specific a vocabulary and then we do pre-training on all of these corpora we then have a special phase of unimodal self-supervised learning for the text model in particular so as part of this we do both the traditional Mass language modeling and we also have a radiology specific self-supervised objective which is specifically um text section matching for radiology report so Radiology reports are broken into different sections and what we do is we do matching between the finding section and the impression section one of these is more like a summary of the other but we know from Radiology domain that these should have quite a lot of semantic overlap so we're able to define a self-supervised objective of matching these two sections so through this we end up with a radiology specific text model which we call cxr Birds if you're interested in using this model it is available on hugging face if you have any radiology needs so then what we do is we use this text model during a joint multimodal training phase so this if you're familiar with models like clip and the basic idea will be I'm training my image my image encoder and my text model jointly at the same time um so what we have here in a little bit more detail is we have an image which goes through our image encoder we get patch representations we project them into a joint space which will enable contrastive learning with the text embeddings and then we use these with as I said a contrastive loss we do something similar on the text side we use some Radiology specific text augmentations in particular we Shuffle sentences because we know that in Radiology reports although they can be quite long they're often invariant to sentence permutations because they're simply listing findings so we use this as a radiology specific types augmentation we put these through our encoder and then we can do a contrastive loss we also use a math language modeling loss on the text model specifically to make sure it maintains it's Radiology specific knowledge terrain joint training we call this pipeline bioville in case you're interested it is described in more detail in our eccv paper all right I also want to mention that we do both Global and contrastive training which is quite important for improving grounding which I'll get to in a moment and we in this paper we do evaluation both of the image model the text model and on the joint The Joint latent space we learn which I'll now talk about in a little bit more detail so evaluation of multimodal models is obviously an evolving field in Prior work um a lot of papers have focused on Purely unimodal evaluation so you think about joint training as a way to learn a better text model or a better image model but we're interested also in learning a good join a joint latent space so we also want to evaluate that so something that we propose in this paper and we released a corresponding data set is a radiology specific phrase grounding task which is about localization of text fragments let's say we're sentences with image components I'll talk about that in a moment so this is The Benchmark data set that we released for phrase grinding we call it mscxr this is available on physionnet if you're interested and Fernando we'll talk a little bit more about how how we actually work with this data set later but basically um there's some examples here apologies to people standing here I might be on the way um but essentially what you can do is you can put an input query like a sentence like cardiomegaly is noted and you can use similarity between the text representation and image patches to identify what part of the image is most similar to that particular text query and this enables us to do localization of findings on the image so our data set consists of over a thousand um image image text pairs with bounding boxes identifying where the pathology of interest is and across a multitude of different findings and this was manually annotated and curated by two Radiologists so uh we hope you uh you make use of this data setting benefit from it so just as an example of um how well these models are actually working we're doing good for time um on this particular phrase grounding data set and what we're showing is we're reporting contrast to noise ratio and main intersection of reunion so these are classic metrics for how well you're localizing something within a box so higher is better for both of these so what we're able to compare is against Baseline methods which may be trained with let's say non-domain specific text models we can see different text models here um they may be clinical text models but they're not Radiology specific text models we can see the impact of training with our cxr text model during joint training on both of our metrics we further observed that um using the local loss is actually really important for this particular localization task um I also want to stress that unimodal tasks are also benefit from our pre-training pipeline so we're interested in the joint space but we also can demonstrate that both your text specific and your image specific and tasks are improved so for example um CX Albert the model we're releasing non-hugging face it shows a superior transfer performance on a challenging Radiology specific nli data set after joint training so we see an improvement um here we also show that you can do image only pneumonia detection both zero shot and few shot in particular the way zero Shaw works is you can generate a prompt that says there is pneumonia in the image where there is no pneumonia on the image and because we have this joint latent space we're able to identify which one of these prompts is more similar to the image which then enables us to do zero shot essentially open set pathology classification so I've given you a very high level overview of some of this recent work from uccv please read the paper if you're interested just to give you a sense of some of the future work we're thinking about and the open questions of course a lot of what we're doing here is about thinking carefully about how we can make the most out of the text data so we're thinking about how we can make more out of other data sources so in particular um we know that in Radiology often there are multiple scans taken um so the progress of the patient may be tracked throughout time so we are also interested in knowing how perhaps using previous images or multiple images could further improve these models we're also interested in learning from let's say you had a small amount of segmentation data or label data could you make use of that so it doesn't have to be fully self-supervised and we're also very interested in what what can you do if you have a good synthetic data generator so this could open up access to a lot more data but of course now it's synthetic so can we leverage such data sources and then of course I've spoken a lot about pre-training but we always want to use these models for something so we're interested in developing Downstream use cases for example report generation or perhaps error detection so we are hiring interns for 2023 so if any of these things or anything else I've discussed is of interest to you please reach out to us after the talk or find any of our colleagues um around at neuros so with that I'm going to hand it over to Fernando to talk about um demos on open source thank you so I'm going to show you I'm going to demonstrate a little bit the capabilities of our pre-trained models so what we're going to do now this is a tool that we can use after the talk so you can play with it we're loading one of the the cases in mscxr the data set as Stephanie mentioned in this case we can see that there's a clinical finding saying that there is cardiomegalimine in the heart is too large so we're going to prompt a text model with a rephrasing of that we're saying cardiac silhouette is enlarged and what we expect is a strong response of the model around the heart so we have here a heat map of the cosine similarities between the text query and the patch image patches in the in latent space we our data State covers a wide range of findings so yeah we're loading a case with consolidations that look brighter and then it also has purely Fusion that means there is liquid in the lungs and what we're doing here is we are passing a prompt that adds modifiers saying that the uh but the pathology here is bilateral and it's rather in the upper lungs so we would expect a response from the model around around those areas and that's what that's what we're seeing here in uh in red and um what we're going to do next is show how the model is also sensitive to other modifiers such as left and right so we said it's bilateral we're going to write we're going to write here we're going to pass the text query left plural effusion so we should see a strong response on the right side of the image which is the left lung which is what we're seeing here um and then we can do the same with the with the right side um where we we should expect a response in the base of the of the right lung and in the constant final angle claustrophenic angle which is what we're seeing um down here um all our models are publicly available on hacking face everything is uh is ready to build upon um all our code is or we have the code for inference is available on GitHub and we also have a jupyter notebook demo hosted on binder that we'll go through in a minute uh so that can be run online without installing anything and without downloading anything and we have also these resources where you can download take a look at our paper and uh data set on on this unit and the next is this uh jupyter notebook so I'm going to show you how to I think we need to start how to start how to get to it so you just need to search highml multimodal which is the name of a Pi Pi package um you'll reach the the package you'll reach the documentation and you'll find a link to the notebook which is hosted on on binder um and this is so that you can get your hands on the actual models and take a look at the code and and run it yourself um it should load but it's not now technical problems it doesn't matter the notebook is there you've seen how to reach it and you can run it yourself there you go it's it's it's here thank you so here's the notebook so we have an introduction to the paper and then this is the commands that you just need to install the the package that we that we saw before and we load uh python Library modules for text and for image and for joint joint inference which has hang again but it does I think it's okay it's all right basically this is the code that runs under the hood for the demo that we showed before if you want to take a look and uh yeah if you go if you move forward this is our team so this is uh this is work by not only engineers and researchers specialists in specialized in medical imaging and machine learning we have also collaborators from Microsoft research Redmond that I experience in biomedical Network natural language processing um but this is not only a work uh by by engineers and researchers we have also a specialists in ux in user experience and we have strong collaborations with clinicians working working with us to better understand the problems that we're trying to solve and to have a better clinical context and I think that's all so yeah feel free to come to us after the talk if you want to play with our demos and if or if you want to discuss about our research or our intern internship possibilities thank you