Occlusion Techniques in Computer Vision

Key Takeaways

this is jacob from rebiflow today we're going to talk about occlusion techniques and computer vision first we're going to talk about why you need occlusion techniques in computer vision to build better computer vision models then we're going to talk about where state of the art research is and inclusion techniques now for that we're going to show you how to get hands-on with your own data set to use occlusion techniques to make your own model better so kind of diving into why we need occlusion techniques computer vision models are like all machine learning models that is they often fine-tune and over-fit to a specific training data set and then they're not able to generalize that well when they get into the wild so in this graph we can kind of see a little bit of an example of how that might work mathematically so we have some noisy data which represents the way that a task might manifest itself in the world and then our model is actually fitting to this data in a very tight line and it's getting kind of over fit here to be a very specific function whereas you know once you go into the wild things are actually going to manifest themselves differently than the way they were in the training set so that it's not going to be so easy for the model to generalize when it gets out into uh into a production environment so oftentimes you'll see curves like this during training which shows that you're actually uh suffering from overfitting during training so this is uh where your your training loss is going down so it's learning the task well but the validation loss is actually has a kink in it where it starts going up meaning that uh basically that the the model is over fitting to the training set but it's not learning not continuing to learn ways that it can generalize outside of that so what what does occlusion techniques do for you well occlusion techniques are a specific way to combat against this so basically occlusion techniques are are designed to hide a part of the image and this hiding of the image during training means that your model is going to actually learn around hidden areas of the image so for example here we might have a model that predicts a cat or dog based on a photo and if we look at the cam the class activation map for this model we might see that all of the predictions are happening at the uh the dog's head so like for example in this image for the cam you can see that the prediction dog basically all relies on this dog's head here and that's uh you know not not such a good thing because what if uh this dog was like hidden behind a bush or something it wouldn't be able to predict dog or cat because it's already kind of overfit to this very specific part of the dog's face um so we might use an occlusion technique to kind of hide where the dog is at where the dog is so now kind of diving into the different occlusion techniques that are out there so for example you have the first one is random erase so here's random erase it's basically just taking a piece of the image and erasing it um this is kind of like implemented randomly and then you replace the the rectangle with just noise rather than the base pixels a similar technique is cut out where there's kind of like cut out rectangle out of the image the difference in cutout in the the original paper at least is that uh they're only hiding those pixels from the first layer of the network so they're allowing the future layers to to see um the area that wouldn't cut out um and then another one is called hide and seek so this one just like kind of draws a grid over the image and then randomly hides someone's in in the uh in the grid uh this one's called grid mask so grid mask is uh doing a similar thing except it's not probabilistically uh hiding uh areas of the grid and then so those were kind of the predecessor uh occlusion techniques and now there are like some even better ones that are coming out uh so the best occlusion techniques are there's this one cup mix um so cut mix is uh where you're not actually just occluding you're actually cutting out a piece and then you're putting in a piece from a different image so this does accomplish the same goal as occluding but it also teaches the model to learn how to recognize uh objects and classes within different environments so not even the the context can even be different then taking it kind of one step further is this uh augmentation technique called mosaic so mosaic is not technically an occlusion technique per se um but it is kind of accomplishing similar things so a mosaic is stitching four images together and kind of looking at them in different uh places in the image so this accomplishes a few things uh to help with regularization so to avoid overfitting of the training set in that it's teaching the model to learn to recognize objects in different locations teaching the model to learn if it's sort of slightly occluded on the edge and then it's also teaching the model to learn in different contexts so it doesn't only have the same surroundings it has various surroundings in the mosaic data augmentation so all those things are all pretty powerful and that is a new state of the art augmentation technique so now we're going to go kind of hands-on with an occlusion example so here we're going to jump into the robo platform so let's say we had a chess data set we wanted to build an object detection model to identify chess pieces in an image so you would start by doing this by getting a data set so we have this data set um that's public on roboflow so that's public.roboflow.com and you can see these images here where you have chess pieces that are labeled and you might you know have nice pieces uh like this that are not occluded so this king is like very visible and has nothing in the way but um in this case you know you might have a pawn that actually is like kind of slightly blocked and so if you're training only on ponds that are not blocked and then you get into a scenario afterwards where the pawns are actually blocked they might have the model might have overfit to a specific part of that pawn and to avoid this you're going to want to try to experiment with occlusion techniques to make your model even more resilient so one one way you can do that is by adding an augmentation step so here's uh augmentation steps within the robo-flow platform you add an augmentation step here um particularly we'll we'll look at the cut out one here which is what we were talking about where those rectangles were were hidden from the image we have a slightly different implementation here where we're kind of randomly cutting out different pieces and you can add more than one cutouts to your training set so here's some previews of how it might work so you go here and you can add like more cutouts you can make them bigger um and then it'll kind of randomly generate these cutouts over your images um so you can you can add a lot of these uh you can crank the augmentations up so then you can get a bunch of these occluded images to send through training so to make a data set version we'll just go ahead and hit generate and i'll say this one is just called occluded so so yeah so that's uh basically getting hands-on with occlusion techniques uh to make versions that are even more robust and to make uh models that are even better afterwards then you can export kind of anywhere you want for training um and you can use one click training here too um to see kind of how these occlusion techniques are uh making your models even better so that's all for today looking forward to seeing you guys next time remember to like and subscribe and thanks so much for watching see ya