Segment Anything Model (SAM) Breakdown | Computer Vision Breakthrough

Key Takeaways

hey there this is Jacob from revofo here today to talk about the segment anything model from Facebook so in case you haven't heard yet Facebook released the segment anything model which allows you to annotate masks in a zero shot fashion and it's proved to be extremely powerful so many of you uh know that we're pretty excited about yellow V8 at Rebel flow so you can see here a GitHub star history of YOLO V8 verse segment anything and you can see that segment anything is actually shooting off in in a couple days of its release in a way that uh is completely unprecedented and everyone is finding this to be a really exciting new model so we're going to go ahead and break down segment anything into what went into the research that was formulated in it and we'll even dive in a little bit into the repo and stay tuned for a future video we'll be walking through some in-depth code on exactly how to use segment anything so diving into the research so the segment anything model is uh promising to be a new foundational model for computer vision the way that it was trained is that it was trained to detect valid masks in an image so in natural language processing there's been a lot of pre-training routines going all the way back to Bert in 2018 where NLP models are predicting the missing token in a text sequence now Vision never really found its pre-training routine that would be like this for computer vision models Mass pixels just don't necessarily have the semantically packed information that missing word might because words are more densely packed with semantics and so things have come out in Vision like clip where images and text have been embedded into the same space and bringing in the text side of things seems to be doing well but nothing has really figured out how to featurize images extremely well now segment anything has kind of broken this by bringing in a huge and absolutely huge semantic segmentation data set to the pre-training routine and showing that a large Transformer can actually be pre-trained on this to produce extremely rich features so let's dive into the way that exactly they formulated the task so the segment anything task is to predict a valid set of masks given a prompt this prompt could be a set of points that are wanting to be annotated or it could be a mass that's already being worked on or it could be a clip-like vector that's coming in as a prompt and so this means that given the set of things that are being prompted you need to predict the masks so that means that during training you can do a lot of little perturbations of the prompts that you're putting into the model to then get valid uh predictions back as you're supervising the routine so the one thing to note about this task that is different than the NLP routines is that this is not fully unsupervised this needs to be used on a data set and the the routine is actually a supervision routine which is which is slightly different now let's talk a little bit about the way the model is actually broken down so the model is broken down into two sections that is first it's a a Transformer block that takes an image and compresses it into a 256 by 64 by 64 feature Matrix and this is a very futurist feature Matrix because it's then able to use this for zero shot on any prompt that comes in so that means anything that you're clicking in any different kind of image is able to use the same feature Matrix and that then those features are are passed into a decoder head so the points the boxes and the features are all getting pulled together into a model to be then decoded out into a potential uh to put potential masks and then you get a list of valid masks back from the model so this is really a revolutionary architecture because you can see here that a giant Transformer is being put behind of the image encoder and you can put that back on a GPU and then you can run the smaller model up on the web so that means that you can kind of decouple these two steps where you have a very strong featurization process happening on the image and then little things happening uh off of it which is something that I think is going to be something that we'll see more and more image models are doing something like that with a really big Transformer model and a lot of new soda things will probably come out of an architecture like this so uh also another revolutionary thing about the paper is they work on a very large scale data set 11 million images and over a billion masks this is the biggest segmentation Corpus that has ever been produced to date and the way they're able to do it is with a model in the loop so there's three stages of the data set collection process they first did an assisted manual step where annotators are annotating along a small Sam model that was probably trained on you know some other Mass data sets that were already out there and they're kind of moving along and they're creating this data set and then they start to go to semi-automatic where M theaters are only asked to annotate mass that the Sam is unsure of and then they actually want full auto where they were able to actually fully generate data set Mast with the model itself because they already had gotten to a pretty good point where they could sort out ambiguous mass and and do a full sweep where they prompt over the entire image so what are some of our predictions here at Roma flow for the segment anything model well first of all it's obvious that all sorts of labeling tools will be using this we are going to be included in that where you'll be able to run Sam to supervise your models even faster by annotating data faster and then you know there will also be numerous applications that are built off same features uh that uh you know will use its zero shot capability so one prediction I have personally is that I'm pretty curious to see who's going to take these feature matrices maybe to be one of you and training supervising an object detection model on it to see how it does on the Coca data set because these features that prove to be an ex extremely semantically rich thing that also has a very good notion of localization and then uh even more impactful than just like the model itself and the features that come out of it is that it is a new framework of a way to be thinking about the way to model CV problems where you have a super large Transformer that's being pre-trained and then bringing out Rich features and then doing some small things Downstream on top of that so there's going to be a lot of exciting things and I think this also gives us a little bit of a window into the way they did the featurization in the gpd4 model this is probably some large Transformer that's compressing an image down into some sort of feature Matrix but we'll have to see how this all plays out so thanks for watching this video today I hope you enjoyed it stay tuned for new content and especially where we'll go on and we'll do a Hands-On dive into the code of segment anything and you'll see not only how to take apart this repo and how you can run the model yourself but we'll be diving into a collab notebook and uh we'll be seeing all kinds of exciting things on how to combine this with other models so stay tuned for those videos coming out from from us and from Peter soon and and until then we'll see you in the next video thanks so much