Panoptic Image Segmentation: Mask2Former explained | Identify all objects!

Key Takeaways

hello community imagine the cameras of your self-driving cars suddenly see this scene and your system has to identify each and every object here on the street now what I want to show you is the theory behind how this is done today in 2023 the technology and I code with your system because this is what you see here is the result of the code that I want to show you we go step by step through the code and you are able to apply this to your vision technology as you can see here we have here each and every person identified with a certain percentage of probability here person 99 percent 100 hand back 95 percent and back 81 percent cardboard yeah this is really cardboard unbelievable person 92 percent and I don't know it was between 10 and 30 seconds on a free collab notebook you see this is what it achieved car 85 percent bus 76 if you have a better compute infrastructure you would be able to resolve here also the people here in the background where we just get a label other merged so this is what we set out to do in this video so let's start what is the vision technology that we have available now at the beginning of 2023 if you are not familiar there are three notations you have to know the first is instance segmentation an instant can be an object a person a car whatever we have here a binary mask classification task and then we have a semantic segmentation here on the left side where you have a per pixel classification problem and maybe you are familiar with algorithms like SEC former or unit or upernet unit a router from the medical applications and now more or less a single started in 2019 somewhere this panoptic segmentation where you have a complete image and everything is identified you have a set of segments each segment has a binary mask like this pink here add a class label on the class label here would be of course a car so you have to know instance semantic and panoptic segmentation and here we go remember in my last video when I showed you the vision Transformer that was just pre-trained and then applied for a downstream task of image classification that this pre-trained Vision Transformer failed to identify to classify the image because there were multiple objects in the image and this was the image on her left side it was the face of a woman and in front of there were lights a light chain and the visual tokens if you want where our self-attention layers we're focusing on their attention they could you see here the heat map on the right side you see they could not distinguish that there is a human face in the background and a light chain in the foreground of course not because they have not been trained for this and you have multiple objects and for this specific case of multiple object in an image or in a movie we have to develop a new technology but it is not only a new technology a new model we also have to have the data sets to train the new model on so when you need huge image data sets hundred thousand millions of of images like let's say from Adobe Cloud with multiple objects in those image and some visual segmentation information so somebody went there and said this is a card this is the kitchen this is the sky and on this image data set we can train our Advanced type of vision Transformers and today in 2023 we have now a universal image segmentation architecture this means in the old time 2022 you had a specific infrastructure to compute only the instant segmentation and you had an architecture only for the semantic segmentation but today with a universal architecture we can solve all three tasks with a unified model with a unified architecture and we moved away from the per pixel classification Paradigm and we are now focused on the mask classification but it is simple they're just more or less three steps you have an image like this credit scene at the very beginning this image is first sense to a backbone it's called now you're not going to be surprised that this backbone is a Transformer this of course is a little bit Advanced Transformer this is a shifted Windows Transformer and I'll show you in a second what it is but we get out a list of feature maps of this image then we enhance these feature maps to high resolution features with a pixel decoder module and then finally and this is where the intelligence comes in we have another Transformer now the decoder stack that takes in a set of queries and transforms them into a set of binary masks and a class predictor your binary mask remember this is the shape of the car and the class prediction is simple core so this is exactly what we have we have here binaural Mouse color coded and and a class prediction person 99 car dark blue 85 percent and if you can identify each and every segment here in this image it's a little bit crowded so I've chosen this image for a reason I'll show you later on we have the technology for this and here we are June 2022 Facebook published something it's called mask attention mask Transformer yeah I know it's Facebook so they are not really genius for Universal image segmentation so what we call it is Mask 2 former and here you have this panoptic and instance and semantic segmentation task and they came up with an architecture now without going into the details I would like to show you that we have in this architecture a swing Transformer now it's the second time we see this so let me explain what it is it is a hierarchical vision Transformer vit like in my last videos using some shifted windows I call it a shifted window based self-attention mechanism now in a non-scientific view you can see here you take a smaller window you shift it along and you only take into account the self-attention and suddenly you get only a linear computation complexity in regard to the image input size this is it there's the architecture if you want jump into the preprint so and this now is our mask to form overview and you see here we use mask attention in the Transformer decoder stack which this is the main idea restricts now the self-attention not to complete image sonar to a localized features centered around some predicted local segments great so at the very beginning here we have our swing Transformer this is our backbone and then we have our classical pixel decoded that enhances our features and then here our Transformer decoder stack L time x layers is where the magic happens if you want and you see here on the right side a magnification of our decoder layer here one and you see here absolutely what you know self attention feed forward Network mask attention and the beauty is to configure the system that it works of course one year before they published this Mask 2 former they published mask if you want one format or mask former where they developed a prototype architecture if you want really interesting to read highly recommend it to you you see here also the loss function how it is put together and this is the most important diagram for you to understand this model it is a little bit out of scope for our video here but please go and read it I leave you the link to the preprint in the description of this video do you have it here also now if we have now the architecture we need the data set and I want that you if you're not familiar you learn two new nomenclatures today Coco and Ada 20K so what is cocoa we go to papers with code you remember they have here data sets you just put in cargo and you get a Microsoft common objects in context this is a large scale object detection segmentation key Point detection and captioning data set of 328 000 images exactly what we need now it was first released 2014 then we had a 2015 Edition then we had a 2017 Edition and we had some additional release so but I just wanted you understand that we have here a data set that we use exactly for the task second cool data set is here and this here is the home page of ADA 20K and you get it it is a semantic segmentation data set and it contains more than 20 000 scene-centric images exhaustively annotated with pixel level objects and object part labels and we have 150 semantic categories like table glass window Sky Garden kitchen whatever and we use this two data set as our main data set when we will code this so now you have a very short introduction what is the state of Technology at the beginning of 2023 for visual technology and it is of course Transformer based it is we moved away from this convolutional neural network Transformer outperform in a lot of tasks here this architecture and in my next video we gonna code now the mask to former state-of-the-art neural network for image segmentation if we have multiple objects in an image we will use here this Universal image segmentation architecture and since I think 18 or 20 January 2023 it is available on hugging face Transformer our beloved open source Library with this code in my next video we will generate exactly this image I hope you enjoyed it and I see you in the coding session