Fast Zero Shot Object Detection with OpenAI CLIP

Key Takeaways

the imaginative laws scale visual recognition challenge was a world-changing competition that ran from around 2010 to 2017. during his time the competition acted as the place to go if you needed to find what the current state of the art was in image classification object localization object detection as well as that 2012 onwards it really acted as the Catalyst of the explosion in deep learning researchers fine-tuned better performing computer vision models year on year but there was a unquestioned assumption causing problems it was assumed that every new task required model fine tuning this required a lot of data and a lot of data required a lot of capital and time it wasn't until recently that this assumption was challenged and proven wrong the astonishing rise of what are called multi-modal models has made the what was thought impossible very possible across various domains and tasks one of those is called zero shot object detection and localization now zero shot refers to taking a model and applying it to a new domain without ever fine-tuning it on data from that new domain so that means we can take a model who can it maybe it works in in one domain classification in one particular area on one data set and we can take that same model without any fine tuning and we can use it for object detection in a completely different domain without that model seeing any training data from that new domain so in this video we're going to explore how to use open ai's clip for zero shots object detection and localization let's begin with taking a quick look at image classification now image classification can kind of be seen as one of the simplest tasks in visual recognition and it's also the first step on the way to object detection at his core it's just assigning a categorical label to an image now moving on from image classification we have object localization object localization is image classification followed by the identification of where in the image the specific object actually is so we are localizing the the object now doing that where essentially just going to identify the coordinates on the image I going to return the typical approach to this is return an image where you have like a bounding box surrounding the the object that you are looking for and then we take this one step further to perform object detection with detection we are localizing multiple objects within the image or we have the capability to identify multiple objects within the image so in this example we have cat and a dog we would expect with object detection to identify both the cat and the dot in the case of us having multiple dolbs in this image almost Cuts in this image we would also expect the object detection algorithm to actually identify each one of those independently now in the past if we wanted to switch a model between anyone these tasks would have to fine-tune or more data you want to switch it to another domain we would have to also fine tune it on new data from that domain but that's not always the case with models like open ai's clip for performing each one of these tasks in a zero shot setting now open ai's clip is a multi-modal model that has been pre-trained on a huge number of text and image Pairs and it essentially works by identifying text and image pairs that have a similar meaning and placing them within a similar Vector space so every text and every image gets converted into a vector and they are placed in a shared Vector space and the vectors that appear close together they have a similar meaning now Clips very broad pre-training means that it can perform very effectively across a lot of different domains it's seen a lot of data and so it has a good understanding of all these different things and we can even adjust the task being performed with just a few code changes we don't actually have to is the model itself we just adjust the code around the model and that's very much thanks to Clips focus on sort of comparing these vectors so for example for classification we give clip a list of our plus labels and then we pass in the images and we just identify within that space where those images are with respect to those plus label vectors and which plus label is the most similar to our particular image and then that is our prediction so that most similar plus label that's our predated class now for objects localization we apply a very similar type of logic as before we create a class label but unlike before we don't need the entire image into clip to localize an object we have to break the image into patches we then pass a window over all of those patches moving across the entire image left to right top to bottom and we generate a image embedding for each of those windows and then we calculate the similarity between each one of those windows embedded by clip and the class label embedding returning a similarity score for every single patch now after calculating the similarity score for every single patch we use that to create almost like a map of relevance across the entire image and then we can use that map to identify the location of the object of interest and from that we will get something that's kind of like this so we have most of the image will be very dark and black that means as the object of interest is not in that space and then using that localization map we can create a more traditional bounding box visualization as well both of these visuals are capturing the same information we're just displaying it in a different way now there's also other approaches to this so I recently hosted a talk with what two two sets of people actually so Federico Bianchi from Stanford's NLP group and also Raphael pissoni and both of those have worked on a Italian clip project and part of that was performing object localization now to do that they use a slightly different approach what I'm going to demonstrate here and we can think of it as almost like the opposite so whereas we slide a window over the whole image they slide a black patch over the whole image which hides what is behind in that patch and then they feed the image into click and essentially as you slide the patch over the image you are hiding a part of the image and therefore if this similarity score drops when the patch is over a certain area you know that the object you're looking for is probably within that space and that's called the occlusion algorithm now moving on to object detection which is like the last level in these three tasks we will be identifying multiple objects now there's a very fine line between object localization and object detection but you can simply think of it as localization for multiple clusters and multiple objects with our cat and Butterfly image we will be searching for two objects a cat and a butterfly and with that we could draw a bounding box around both of those objects and essentially what we're doing now is using localization for a single object but then we're putting both of those together in a loop in our code and we're producing this object detection process now we've covered the idea behind my image classification onto object localization and object detection now let's have a look at how we actually Implement all of this now before we move on to any classification localization or detection task we need to have some data we're going to use a small demo data set called James Callum image text demo and we can download it like this so we're using hooking phase data sets here which we can pip install with Pip install data sets and this is the day so it's very small it's 21 text to image pairs okay one of those is the image you've already seen the cat with a butterfly landing on its nose very curious how they got that photo now after you've downloaded that data set we're going to be using this image here and what we want to do is not use the image file itself because at the moment it's a it's a pill python image object but instead we need to convert it into a canister now we're going to be using pi torch later on so what I want to do here is we're going to just transform the image into a tensor and we use toxin transforms use the typical pipeline tool in computer vision and we just use tube tensor okay and then we process our image through that Pipeline and then we can see that we get this okay so what are these values here we have the height of the image in pixels the width of the image in pixels and then also the three color channels red green and blue that make up the image now we need a slightly different format when we are processing everything one we need to add those patches and two we need to process it through a pie torch model and we also need the batch dimension for that so the first thing we're going to do is add the batch Dimension it's just a single image so we just have one in there but we we need that anyway and then we come down to here so this is where we're going to break the image into the patches okay each patch is going to be 256 dimensions in both height and width so the first thing we do here is unfold and we get this here okay there's two five six and there's 20. now the 20 is the height of the image in these 256 pixel patches and we can visualize that here all right so now we have all these kind of like slivers of the image that's just a vertical component of each patch and we use unfold again but in this time in the second Dimensions the targeting what was this Dimension here and we also get another 256 now we visualize that we get our four patches okay like this now if you just consider this here it's like if we look at this patch here it doesn't tell us anything about the image right and even when we're over cats these patches are way too small to actually tell us anything if clip is processing a single patch at a time it's probably not going to tell us anything maybe it could tell us that there's some hair in this patch or that there's an eye in this patch but beyond that it's not going to be very useful so rather than feeding single patches into clip what we do is actually feed a window a six by six patches or we can modify that value if we prefer and that just gives us a big patch to pass over to clip now the reason that we don't just do that from the start we don't just create these bigger patches to begin with is because when we're sliding through the image we want to have some degree of overlap between each patch okay so we create these smaller patches and then what we can do is actually slide across just one little patch at a time and we Define that using the stride variable so if we come down to here we have window we have stride remove this and here we go this is our code for going through the whole image creating a patch at every time step okay so we go for y and we go through the whole y-axis and then within that we're going across left to right with each step and we initialize a empty big patch array so this is our like the full window we got the current batch so okay let's say we start at zero zero X zero y zero we go from zero to six and zero to six here right so that gives us the very top left corner or window of the image and then we're literally going through and and just go processing all of that and you can see that happening here as wine eggs are increasing we're moving through that image and we're seeing each big patch from our image okay sliding across with a single small little patch at a time so that we don't miss any important information now this is how we're going to run through the whole image but before we do that we actually need clip so let's go ahead and actually initialize clip so to do that all we do is this so we're using hook and face Transformers which is using pi torch in the in the back there so we need the clip processor which is like a pre-processing pipeline for both text and images and then the actual model itself okay so we some model ID and we initialize both of those then what we want to do is move the model to advice if possible all right so we can use CPU but if you have a Kudo enabled GPU that will be here much faster so I'd recommend doing that if you can if not then you can use CPU it will be a bit slower but it will still run within a variable time frame so if I'm running this on my Mac I am using a CPU you can actually run this on MPS as well so you could change your device to MPS if you have an MPS enabled Apple silicon device so now returning to that process where we're going through each window within the image we're just going to add a little bit more logic so we are processing like we were before there's nothing different here we're creating that big patch and then what we do is process that big patch and process a text label okay so at the moment we're looking for a fluffy cat within this image so that is how we do this we're returning Pi torch tensors we also add padding here as well for the text although in this case I don't think we need it because we only have a single text item but we include that when we're using multiple text items later and then we calculate and retrieve the similarity score between them okay so if we pass both text and images through this processor we'll pass both into our inputs here and then we just calculate the or we extract the logits for each image and the item just converts that into a array of values or single value and then here we have those scores so what we're doing here is creating the what I earlier called like the relevance map or localization map throughout the whole image so for every window that we go through we're adding this score to every single patch or little patch within that window and what we're going to do or what we're going to find when we do that is that some patches will naturally have a high score than others because they are viewed more times right so if you think about the top left patch in the image that's only going to be viewed once whereas patches in the middle are going to be viewed many times because we'll have a sliding window going over there multiple times so what we also need to do is identify the number of runs that we perform or number of calculations that we perform within each one of those patches the reason we do that is so that we can take the average for each score based on the number of times that score has been capital related because here we're taking the total Awards scores and then we'll just take the average like so now the scores tensor is going to have a very smooth gradient of values from zero completely irrelevant to one now if you consider that we've been going over these scores multiple times it means that the object of interest is kind of like faded out of the window like over multiple steps so that means that the similarities score quite gradually faves out as you go away from the object which means that you don't really get very good localization if you use these scores directly so what you what we need to do is actually clip the lower scores down to zero so to do that what it is calculate the average of scores across the whole image we subtract that average from the current scores what that will do is push 50 of the scores below zero and then we click those scores so anything below zero becomes zero and we can do this multiple times okay one time is usually enough but you can do it multiple times to increase that effect of making the edge of this detected or localized area better defined and then after you've done that what we need to do is normalize those scores okay so we might have to do this a few times but everything's probably going to be within the range of like zero to 0.5 or 0 to 0.2 so then we normalize those scores bring them back within the range of zero to one now to apply these scores to the patches we need to align their tenses because right now that they are not aligned okay for the scores we just we have like 20 by 13 tensor but for the patches we have the the batch Dimension there we have the 20 by 13 which we do want but then we have the three color channels and the two five six for each um set of pixels within each patch so we need to adjust that a little bit so we need to First remove the batch Dimension we do that by squeezing out the zero Dimension which is a batch Dimension and then we permute the different dimensions it's actually just moving them around in our patches in order to align them better with the score tensor dimensions and then all we do is multiply the patches by those scores that's pretty straightforward then we have to mute them again because if we want to visualize everything needs to be within a certain shape in order for us to visualize our matplotlib so we come down and the first thing you do is just get y next here so Y and X are the the patches see here this is y so the height of the image in patches and then 13 which is the width of the image in patches and we come down here and we can plot this okay and we get this it's pretty nice visual localizes the The Fluffy Cuts within that image now what's really interesting is if we just search for a cat we actually get a slightly different localization because here you can see it's kind of focusing a lot on the fluffy part of the cat so if we just search for a cat it would actually focus more on the head so we can really add nuanced information to these prompts and get a pretty nuanced response back now we can do the same for butterfly so we'll just throw all that code together this is just what we've done before we initialize scores and runs and we go process all of that the only thing we change here is the prompt we change it to a butterfly and if we go down and we're going to go down and down and visualize that we'll get this okay so again that's that's pretty cool we can see that it is identifying where in the image that butterfly actually is so that is the object localization set now I want to have a look at object detection which is essentially just taking the object localization and and wrapping some more code around it in order to look at these multiple objects rather than just one but to do that we can't really visualize in the same way that we've done here we're going to need a different type of visualization and that's where we have the bounding boxes so let's take a look at how we would do that so using the I think the butterfly examples of butterfly scores that we just calculated we're going to look at where those scores are higher than 0.5 now you can adjust this threshold based on you know what you find works best so we do this and what we'll get is a array of true and false values as to where the score was higher than 0.5 and not and then we detect where the non-zero values are in that array and what we do is get a load of X and Y values here so position three two we know that there is a score that is higher than 0.5 and we get three and two here so three is the row of the non-zero value and 2 is a column of the non-zero value so at row position three and column two we know that there is a non-zero value or a value or score that's higher than 0.5 our threshold and put all that together we'll get something looks kind of like this so we already we kind of see that localization visual that we we just created and what we want to do is identify the bounding box that's just kind of surrounding those values okay so we know in terms of like a coordinate system we want one and three and four and ten to be included within that so what we do is find the corners from the detection array or or set of coordinates that we got before from NP non-zero and what we do is we just take the minimum X and Y values and maximum X and Y values and that will give us the corners of the box that's pretty simple to to calculate now when we get the maximum value what we want to do is because we basically we get in the position of the patch and the position of each patch we're essentially identifying the top left corner of each patch so when we're looking at the maximum value we actually want not the solved patch but the end of the patch okay so that's why we add that plus one here in order to get that the same for the x max value as well so that gives us a corner coordinates and then what we do is multiply those Corner coordinates by the patch size it's 256 pixels and then we have the pixel positions of each one of those Corners because before we had the patch coordinates now we have the pixel coordinates which we can map directly onto the original image so we can see the minimum values here so we have for x and y two five six and a seven six eight and what we want to do because we're going to be using matplotlib patches and matplotlip patches expects the top left corner coordinates and the width and height of the bounding box that you want to create so we calculate the width and height and that's pretty simple it's just y Max minus y Min and X knives minus X min look at these and what we can do now is take the image we have to reshape it a little bit so we have to move the three color channels Dimension from the zeroth dimension to the final Dimension so we just do that here move axis and now we can plot that image okay so we showed that image with matplotlib and then we create a rectangle patch this is our bounding box okay so we pass X Min and Y Min that's the top left corner and then we also pass the width and height of what the boundary box should be and if we come down we get this visual okay so that's our bounding box visualization and with that it's not much further to create our object detection so let's have a look at how we do that now the logic for this is pretty much just a loop over what we've already done so I've put together a load of functions here which is essentially just what we've already gone through getting patches getting the the scores getting the the box and then the one thing that is new here is this detect function okay so we have detect that's going to get the the patches so it's going to take an image and it's going to split into those patches that we created we're going to convert the image into a format for displaying with matplotlib we did that before and we also initialized that plot and add our image to that plot and then what we do is we have for Loop and this for Loop goes through the image localization steps and bounding box steps that we just went through just multiple times okay so we have multiple primes and we want to do it multiple times so we calculate our similarity scores based on a specific prompt for all of our image Patches from that we get our our scores in that patch tensor format that we saw before and then what we do is we want to get the box based on a particular threshold so 0.5 like we used before you can see over there we have our patch size we just need to pass to that for a calculation of the or for the conversion and we have our patch size which we passed to that for the conversion from patch pixel from patch coordinates to pixel coordinates now we also have our scores and that will return the minimum X and Y coordinates and also width and height of the box we create the bounding box now we add that to the axis okay so now let's visualize all this see what we get so here I've used a slightly smaller window size before using six just to point out that you can change it and depending on your image it may be better to use a smaller or larger window and you can see so what we're doing here we've got a cat and a butterfly and you can see that we get we get a butterfly here and we get the cat here okay it's pretty cool and like I said with clip we can apply this object detection without fine tuning all we need to do is change these prompts here okay so it's it's really straightforward to modify this and move it to a new domain okay so that's it for this walkthrough of object localization and object detection with clip as I said I think zero shot object localization detection and even classification opens the doors to a lot of projects and use cases that were just not accessible before because time and capital constraints and now we can just use clip and get pretty impressive results very quickly all it requires is a bit of code changing here and there now I think clip is one part of a trend in multimodality that is kind of creating a more accessible ml that is less brittle like models were in the past that required a lot of fine tuning just to adapt to a slightly different domain and just more generally applicable which I thing is really exciting and I'm I'm it's really cool to see this sort of thing actually being used and to actually use it and just see how easy it is to use clip for so many different use cases and it work like incredibly easily so that's it for this video I hope it has been useful so thank you very much for watching and I will see you again in the next one bye