Mask R-CNN - Explained!

Key Takeaways

Greetings fellow learners. In this video we are going to talk about mask RCNN. The what, the why, and the how. And we'll also have some code. So let's get to it. So what is mask RCNN? It is an end-to-end trainable neural network that performs localization, classification, and instance segmentation. Specifically, localization is like drawing a bounding box around an object of interest. Classification is determining what that object is along with the potential confidence. And then instance segmentation is creating a binary mask of each object instance separating it from the background and other objects. So that's mask RCNN. Now why do we have mask RCNN? Well, the goal of this network architecture was to create a framework for instance segmentation. And the intuition of how we can build an instance segmentation system or framework was that it was to combine the aspects of object detection and semantic segmentation. So a couple of terms here. So let's get through each of them. So first of all, object detection. This involves the classification and localization of objects in an image. So with an object detection system or framework, we can take an image and then we can you know determine the localization of where objects are as well as a categorization of what that object is. Semantic segmentation involves distinguishing objects from the background. So in this case for example we are able to distinguish like a horse from this human and then we also have a bunch of I guess like cars here that appears as this you know these blobs. Now, while this is impressive, semantic segmentation does not distinguish though between unique objects of the same object class. So, right here, for example, these cars would appear as just one big gray blob. And so, in kind of in order to to handle instance segmentation specifically, what we would do is we would combine the semantic segmentation along with object detection. And thereby like these individual gray blobs are now interpreted as individualized objects. So we have a localization, we have a classification and we have segments where we know which segments belong to like different objects itself and that's why we have mask RCNN and so mask RCNN is structured in such a way that it combines well faster RCNN which was an RCNN architecture that performs object detection along with instance segmentation capabilities. So given this, let's kind of like take a look at the architecture. So this here is a structure of the faster RCNN. Now faster RCNN like we said performs object detection and let's walk through at a very high level how it does this. So we take an input image then we convert it into a tensor of like 14 + 14 + 512 or any such dimension. We then pass this through a region proposal network which is essentially going to create a bunch of bounding box localizations along with a confidence for each of these boxes of how confident we are there exists an object or not. We use NMS non-maximum suppression to get rid of very overlapping bounding boxes and then end up with region proposals. Say in this case it's like 300 of them. And then for each region proposal, we're going to pass it into this structure over here where we essentially take every region proposal, perform something called ROI pooling in order to transform it into a fixed size tensor and then what we're going to do is essentially create, you know, determine the class of that region proposal of what it could be as well as four neurons over here, which is like the corresponding bounding box coordinates or localization. of it. And so you might end up with an image that looks like this. If we do it for every single region proposal, we'll have up to like 300 of these bounding boxes because, you know, some region proposals might represent backgrounds in which case we don't draw a box around it. And then with non-maxim suppression, we'll get like the final um you know object detection that is localization along with classification. So this is like the highle overview of faster RCNN at least the inference part. Now to this entire structure mask RCNN looks like this during inference and it makes two major uh changes to the architecture. Well, the first part is that the like before when we had you know specifically in for each region proposal when we're processing it we were providing class confidences along with like the bounding box for each region proposal but at the same time we will also output a segmentation for every single pixel within that region proposal. So we it's this is like a a binary number that basically says is this a part of that particular object or is this pixel not a part of the object. So that's done with this entire arm over here. Next what we do is also introduce something called ROI align instead of in place of like ROI pooling and this is required in order to get a better segmentation. So that's the main like two changes that we make and we're going to actually you know try to look at this in more detail when we look into like the inference of the U mask RCNN. So that's about it. Now let's actually try to get some more intuition on ROI align versus ROI pooling what it is and why it exists. So ROI line essentially is going to map a variable size tensor which represents like a region proposal to a fixed size tensor that also represents the region proposal. Now ROI pooling as we have seen in a previous video on faster RCNN it does the exact same thing. It maps a variable size tensor to a fixed size tensor. So if it does the same thing then why do we use ROI align? Well, mask RCNN performs segmentation and that has a stricter pixel alignment requirement. So why is this the case? So now in in order to illustrate like what this is and how mask how ROI pooling helps, let's actually walk through you know the comparisons with an example of ROI pooling and then walk through ROI align. So let's say that this you know 5 + 4 matrix represents a region proposal. which we want to map now to a fixed 2 +2 tensor. Now what we do here is you know we want to bin this into you know we want to bin this in such that the output is 2 + 2 and in order to do this well the bin width should be you know it should have two units and the bin height should have about 2.5 units. Now to compute each bin though ry pooling you know this is like a floatingoint number. How do you have 2.5 indices? You can have two or you can have three according to ROI pool. And because of this that this means that we are quantizing the indices making it like an integer value. And so essentially what this looks like in practice is that you know we'll have two rows and two columns that are aggregated to form this number that's like a max pooling. Similarly these two cross two over here is aggregated to with max pooling to get this. And then here we get the remaining three uh rows and two columns in order to get you know max pulling you get 0.6 and then max pulling of the three rows and two columns you get 0.5. So quantization causes misalignment and this is this can be an issue and what this means is that some entries like you know some entries over here represent more or less pixels than other entries. After all, because of quantization, this represents like more pixels than you know this number. And this can cause issues when you're creating segmentations. So how do we deal with this? Well, we just try not to perform quantization of the input indices and instead perform something called bilinear interpolation. And this is the essence of ROI align. And very high level what this means is that you know we need to consider we have to bin this like entire matrix into like a 2.5 rows and two columns for each bin and then perform bilinear interpolation. So this is like 2.5 rows and two columns. You kind of consider a function of these values over here in order to get like this output value and we just you know have the same bin size over here and effectively the entries roughly represent the same number of pixels making it better for segmentation. Now to see how exactly we got those numbers I actually have like code that compares ROI pooling. So this is the logic for ROI pooling uh where we get like the output that we discussed 0.2.8.6 6 0.5 and then we also have the code for bilinear interpolation for ROI align which has the output that we also just discussed. Essentially what we do is you know for each like 2.5 cross 2 bin we compute a center. What we do then is call this bilinear sample in order to compute like the neighboring four points. So in this case the four points would be x0 y 0 x0 y1 x1 y 0 and x1 y1 and of those four points you know the the final interpolated value is going to be like a function of these four points which is given by this equation over here and that's how we kind of you know get this output and so like I mentioned before because you know each entry now represents the same number of pixel it is better the output is better aligned with the input for better segmentations. All right. So let's now talk about the training of mask RCNN. So step one, we want to take like a backbone convolution neural network that is going to be pre-trained on object recognition. That is we take an image and then we want to output an object category and you know we can use something like VGNet or at the time I think ResNet was like typically used. So we can pre-train these the weights of this network in order to get like the functional backbone of the mask RCNN. Next what we do is that we are going to now get our data set. So this is the data set that we want to fine-tune on for you know detection and segmentation. And we're going to be using like the Coco data set here where we have you know each of these are like training examples where we have segmentations of different objects and each segment is going to be associated with like a classification. So this would be the segment of a bear and the actual like bear category and from this information we can infer bounding boxes. So it's pretty easy to computationally draw a bounding box around this too. Next what we're going to do is we're going to train the mask RCNN. So let's now walk through this entire behemoth at a pretty high level. So we take an input image, we pass it through, you know, these pre-trained weights over here from that first stage of object recognition in order to get a tensor that represents an image. This 14 + 14 + 512 tensor. Then we pass this into the region proposal network over here and we're going to get 1764 bounding boxes each of which is going to be associated with a confidence of what's the probability that there is an object in this bounding box. And then what we do is we're going to filter these predictions such that you know from 1,764 predictions in order to you know for the training to compute the loss we're just going to consider 256 positions or predictions of which you know we we select these such that you know the 128 of them actually have a very large overlap with objects and another 128 do not have a large overlap with objects and can be considered as like background And because now we have predictions, we also have a label of like the bounding boxes around each object. We have both. So we can compute a loss. And so first we're going to compute a binary cross entropy loss. This is you know the it's binary because there's only either object or not an object. And we have the probabilities of you know each 256 boxes to understand if it's an object or not an object. And hence we'll have like 256 values which we can take an average or or sum to get a classification loss. And similarly we have you know half of those that's like 128 of them have actual bounding box localizations. So they're going to be like this is going to be a measure of how much overlap there is with the predicted bounding box with the actual bounding box and larger that difference larger the loss. And hence we'll have like 128 of these localization losses of which we can take like an average and then we can add these together or aggregate them in order to get the RPN loss or the region proposal network loss. That's one of our loss components which we will keep aside. Now going back to our RPN processing, we have like 17764 bounding box predictions along with object um confidences. We're going to use NMS in order to get rid of very overlapping bounding boxes and only extract like the top confident uh region proposals. Let's say it's like 1500 of them. Now from this for training purposes we're only going to select 256 of them. This is for now the object detection or classification phase specifically. And so of these 256 proposals, we chose them such that 25% of them are going to have a large overlap with a object category like a dog, a leaf, or whatever it is. And the other 75% of them are going to be like hard negative labels that'll effectively represent a background. And so what we're going to do is now we're going to pass it into this entire region proposal part over here. So essentially we're going to extract from the original image tensor. We're going to extract like the part of the image that represents one single um region proposal and we pass it into ROI align to get a fixed size tensor of 7 cross 7 cross 512. And now from here there's going to be three parallel branches. The first is going to perform a classification determine a classification as well as a confidence. The second is going to determine a localization and the third is then going to determine the b binary values for every single pixel like whether it belongs to the specific object or does not belong to the specific object. So in that first case let's say that you know this is like a class dog. It predicts like it's the we're going to take like 1,00 class classification. It might predict dog with a confidence of like 98% or something like that. Then we have again like a huge uh vector. This is like a vector of size 4,000. So four neurons for each of the thousand categories. Um during the training phase, we're just going to take the ground truth object class. So in this case, like let's say we look over here, the ground truth is going to be uh it's it's a dog. And so we're going to only index specifically the four neurons that represent a dog. And that's going to be this over here. Next, what we do is well, we take this 7 + 7 + 512 tensor. And then, you know, the third one is we're going to reshape this into this 14 + 14 + 128 tensor. We're then going to perform some convolution operations to convert it into a 14 + 14 cross well, we have 1,000 channels. So effectively it's like one channel for every single object category. Once again we will index this and in the training phase it's indexed based on dog because that's like the true label. And so we only have like a 14 + 14 matrix here which we perform sigmoid in order to get like for every single entry it'll be an uh some value between 0 and 1. And effectively this is going to be like representing some sense of probability that you know this uh entry is a part of the dog itself or not a part of the dog. And while that sounds a little bit less interpretable, we can kind of map this to the image pixel space by using bilinear interpolation as well. You know, we looked at that same process for ROI line, but we can upscale it this time into the pixel space. So every entry over here is going to be determining is this pixel a part of the object dog or is it not. And so we have a class with confidence. We have the box localization. We have the categorization of every pixel as being a part of the object and not a part of the object. And that's going to be one region proposal prediction. And we do this for all the region proposals. So we're going to end up with 256 classes, right? That's all of our region proposals with 64 bounding boxes and segmentations. It's 64 because only 25% of these are actually foreground objects. The remaining 192 are background objects. And so like we don't have you know bounding boxes or segmentations for background. And so we have predictions, we have the label and hence we can compute the loss. And this loss has three components. We have a cross entropy loss which will represent you know the 256 classifications. We have the smooth L1 loss which will you know which is the loss of bounding box localizations um for 64 of them. In this case we can aggregate them to get that localization loss. And then the 64 segmentations we have a binary cross entropy loss. And to compute this specifically you know for every region proposal we have each pixel right. So each pixel is going to have a cross entropy loss, right? It would have been like a number between 0 and one. We compare it with the binary label of 0 or 1, which was the ground truth. And then we can get a cross entropy value for that single pixel. We repeat this for every single pixel in that region proposal prediction and then sum it up in order to get the cross entropy loss for the single region proposal and for segmentation specifically and then you know you can do this 64 times because we have 64 of these predictions and you can take the average of them in order to get the final segmentation loss. So we have all of these three losses which we basically add together in order to get this single instance segmentation loss. And now we combine both of our RPN loss along with the instant segmentation loss to get the final loss for our entire network. So this is our entire network now. And once we have that final loss, what we can do is effectively now train the network. So what we're going to do here is this loss is effectively going to be back propagated through the region proposal network updating the weights of the region proposal network as well as fine-tuning these weights of the backbone architecture. And then what we do is we'll also back propagate the loss through the instance segmentation loss to update you know each of these region proposal components again but as well as you know the backbone components too. And so we can effectively train this on multiple examples in this way. And so this network learns. Now in order to understand inference, we really don't need many components over here. Like we can get rid of the lost components in this in you know the region proposal network part. We can also get rid of the loss components in the um in the you know for computing each region proposals loss here. And we can also get rid of some post-processing from the region proposal network as well. And if you kind of like get rid of this and simplify the network, we'll effectively get something that looks like this. And this was essentially like the the same idea that we saw in in the very beginning. And so let's just walk through this very quickly of how inference happens. So, we now have the fine-tuned mask RCNN where we take an input test image and we're going to convert it into a tensor over here, a 14 + 14 cross 512 that represents the input image. We are then going to pass this into the region proposal network in order to generate like 1,764, you know, region proposals. So, it's bounding boxes around potential objects along with like a confidence of how confident are we that this represents some object of interest. And then we'd perform NMS to remove very overlapping bounding boxes and extract let's say in this case the top 300 region proposals. And then for each of these region proposals, we're going to execute this entire block over here where we first perform ROI align to ensure that you know that's you know when we extract the tensor from the you know this input tensor over here it's going to represent some fixed size of like 7 + 7 cross 512 and then there's going to be three parallel branches. The first branch is going to determine a class plus its confidence. So, we're going to get the of the 1,00 categories, we're going to extract the one with the highest categorization value. That will be the output class. Let's say it's dog with like a confidence that's pretty high or something like that in the 90 to 100%. Then, we're going to say that there's like we're going to also um you know, we would have also determined localizations for all 10,00 classes, but we're only going to extract localization for the highest class that we predicted, which is again dog. So this is the localization of that dog. And then what we're going to do in the other parallel mask, what we're going to do is we're going to um you know take this 14 cross 14 uh cross like 128 tensor, map it to a,000 channels for 1,000 object categories. Then we're only going to index a specific object category or class category that was predicted over here, which is dog. So this is like the the dog map. We're performing sigmoid to convert each value here to a a value between zero and one. And then we're going to map it to the pixel space. So each point over here is like a number between 0 and one that represents the probability that a pixel is a part of the dog object or not a part of the dog object in the region proposal. And so we use all of these three pieces of information over here in order to get like the bounding box plus localization plus segmentation. And we do this for each region proposal. So we'll effectively end up with up to 300 boxes because you know there might be like some cases where the predicted class is going to be the background class in which case we have no localization and no segmentation. And for these remaining bounding boxes we're going to perform NMS to remove very highly overlapping bounding boxes per class in order to get like the final segmentation over here. Let's now take a look at some fun code. So basically we're using you know the the mask RCNN that's in torch vision and we're using it with a resnet backbone that is like the network that we pre-trained our image recognition on and then what we do is you know it's pretty simple code but we can walk through like some example outputs where if this is the original image the output kind of looks like this. we have a bounding box plus um the categorization plus a confidence number and a segmentation mask as well. Similarly, you know, if there's multiple objects in the image, it seems like it does pretty well, too. We have a dog and the cat that are recognized with pretty high confidence of, you know, here's like a 99%. Then we have this dog, which was the same kind of image that we saw in the inference slide where we're seeing that, you know, two dogs are recognized pretty well. This pot is recognized as a bench maybe because either one it wasn't in the classification set or two it's just like a mclassification altogether. But overall still the segmentations are pretty clean. This one is pretty interesting because it has just a bunch of cars and trucks which are all detected quite clearly with their segmentations intact as well. And even like some of the smaller objects in the back which would be like these cars over here. They're detected quite well too with high confidence. So I hope all of this makes sense. Interestingly enough, CPU inference time is like 8 to like 9 seconds here on CPU, but on GPU it can be as quick as like 200 milliseconds. And so this makes it just as fast as the faster RCNN for object detection, which kind of makes sense as like we're doing like par we're doing the mask segmentation part in in parallel to localization and classification. And so I hope like all of this makes sense. Quiz time. Have you been paying attention? Let's quiz you to find out why do we perform ROI align instead of ROI pool in mask RCNN. A. Classification requires strict pixel alignment. B. Localization requires strict pixel alignment. C. Segmentation requires strict pixel alignment. Or D. All of the above. I'll give you a few seconds to answer this question. The correct option is C. Did you get it right? Comment your reasoning down in the comments below and let's have a discussion. And at this point, if you think I deserve it, please do consider giving this video a like because it will help me out a lot. And that's going to do it for quiz time. But before we go, let's generate a summary. In this video, we discussed what MAS RCNN is. It is an endto-end trainable neural network and hence also a framework that performs localization, classification and instance segmentation. We also discussed how mask RCNN combines both ideologies of semantic segmentation and object detection to perform instance segmentation. Structurally speaking, mask RCNN is basically faster RCNN with some modifications to instill instance segmentation capabilities. Those two main changes are essentially first we're going to add this mask arm over here for each region proposal and also for each region proposal we make use of ROI align instead of ROI pooling. We also discussed why this is the case and it's because you know mask RCNN performs segmentation that has a strict pixel alignment requirement and ROI align allows us to do this without quantization which was done by ROI pooling and instead makes use of bilinear interpolation to do this. We then discuss the entire mechanism for how this network is trained and also how we can perform inference on this network as well. And we also did walk through like some fun code inference examples too just to illustrate how localization, classification, and segmentation can be performed by mask RCNN. And that's all that we have for today. Thank you all so much for watching. The resources for this video, including the slides, the code snippets that are over here too, will be down in the description below. So do check them out and I will see you in the next one. Take care.