SAM 2 is going to transform COMPUTER VISION!!!

Key Takeaways

meta Sam 2 is completely crazy meta has launched a new version of segment anything model which they launched couple of months ago Sam the Sam V2 has been launched yesterday at sigraph I guess and this model has been insane I mean it is a model that can help you do a lot of different things like tracking this object this is a model that can help you segment images explore a lot of different things this model can help you segment images but also can help you track images or track objects if you're not familiar with image segmentation image segmentation is one of the fundamental tasks of computer vision for example if you got a camera and the camera is looking at something now when it looks at something the first thing that it has to do is it has to detect an object like it has to find out that there is something and image segmentation plays a huge role in it whether you are trying to use a self-driving car or you're trying to just track a ball in a football CT or you're trying to track a car or you're trying to track somebody who is running anything what you want to do you first have to segment that part segment that object that is object segmentation and after you do that you can classify that like what kind of object it is then you can track that if it is moving so this is one of the most fundamental computer vision task and segment anything model is quite good at the latest version is completely insane there are a couple of use cases that meta is discussing in this initial launch we're going to see everything including the demo I'm working on the Google collab notebook right now there is a lot of moving pieces so probably in a couple of days I'll actually share the Google goab notebook with you for now what meta has done here is that they have released this new model that they call as segment anything 2 which is Sam 2 Sam was quite revolutionary so if you go to meta's website you can actually see that people have been using Sam on a lot of different things like uh people are using to identify coral reef um this is one popular use case I myself have done it I think it's part of some Global coral reef uh detection system so where you can train these models you can help them create training data using uh Sam and uh you can see that Sam is pretty good with what it does it is good with images it is good with um moving objects as well so for example in this case the boy is continuously moving and Sam is pretty good in noticing even when the boy is actually moving and there is one more demo that is quite impressive So Meta lets you track a part of an object for example the Baseline model that they've got tracks the whole human being but meta at the start you would have noticed that they pretty clearly indicate that they want to only track track the shirt not the head and everything and it does a pretty good job of that and there are lot of different places where the model is absolutely insane whether it is like a football here where you specifically say that you don't want to track the shoe but you want to track the ball it does a pretty good job uh so overall this is an excellent announcement there are a lot of nties details about it but in short this announcement has got meta Sam 2 code and weights they're sharing the model with us the model has been shared under Apachi 2.0 license which means you can do anything you want now whenever I say you can do anything you want you might start thinking that what am I going to do with that the very first thing is for example I don't how many of you have seen this a lot of times you would see YouTubers separating their friend self and putting a text behind them and that is primarily possible thanks to object segmentation So Meta has got a a small demo for that for example if you go here and then you can actually see that there is a bird and the bird is been segmented and then split out of the background and there's a text overlay there and then the bird is been put back and this kind of effect is possible because of models like this and meta seems to be that meta implies that more people will use it in commercial setup and the other use cases for example you've got three balls you can track these balls exactly you can add effects to these balls and there are a lot of other places where you want to add effects like for example you've got a moving object you want to track that person and then add an effect just behind that person and for you to add that effect you should be able to track that person you should be able to segment that person and that is exactly what you're seeing here so in a lot of commercials and a lot of photo editing video editing application this is going to play an insane role in putting an open source model with commercially permissive license into the hands of developers so that is one great thing the second thing is uh they have also shared a very important data set a wide range of data available here so sa data set has got all the things that they've used to train this model so this is allowing a perm large scale video data set so it has got a lot of information and it has got lot of information from different parts of the country so one of the demos that they've got is a lady with a sar which is a dress Indian women weer so they they showed that a lady with a sari dancing so they have got data set from all over the world so that is another good thing and finally they have got this demo that we can all use it so the demo is available for us to use it so if you go to Sam 2. metad demolab see here click the button try now first time they'll ask you to accept something at this point I I don't know what data I'm giving permission to so this is one more data and here you can just upload anything you want for example I can go change the video uh for example I'm going to select this and once we select this the video gets loaded and you can select an object you can select an object here and uh let's not select it let's remove this object let's remove this one and and uh remove remove and we are just keeping that and then we're going to track the object so you can see the object tracking is started and you can pretty much see where the cup is going so at the end you would know that the cup is there so this is a very cool way to create a lot of games and a lot of other things and as you can see here it is pretty fun one thing is like you see here what we are doing is we are trying to track the object and uh the other thing that is what is happening here is that simply that you are able to segment the image for example you got the dog here and first thing first is you need to segment the image and this is honestly like a very important thing for a lot of self-driving cars you often hear self-driving cars not able to read the signs not able to understand the sign the fact that they are going to read the sign in the first place is uh thanks to an image segmentation model and I'm not sure how much you follow self-driving cars as a theme that there are two different School of thoughts one is the lighter based thought way Mo if you see the Google company it's a very heavy ligher based card so you've got like the top crown at the top that sends constantly ligher scans everything makes a 3D image of it if you have got a latest iPhone you know that you know you've got the lighter sensor and it creates a world view but then youve got the second school of thought which is the Elon Musk school of thought which is like okay I don't need to do any lighter stuff uh all I have to do is put a bunch of cameras record them in real time I can segment everything and then I can do um computer vision and this model this model particularly supports that school of thought so it says that I mean it is insane like few years back if you told anybody that you're going to do something like this it would have been insane that people wouldn't have believed that this is quite possible now what you can do is not only just you can segment a static image or part of a video but you can pretty much track that throughout the video like if you have got a video you can track that throughout the video that is pretty insane how effectively you can do that the object goes behind another object and all these things are possible thank to thanks to the latest model from ma the Sam 2 see but the point here is that the model is not without any issue the model itself has some kind of issues still like it might lose track of objects and uh there are certain cases where the model might not do well um especially when the object is moving faster and this is a problem that you would have also heard from Apple Vision Pro reviewer so when you wear Apple Vision Pro when you are in a fast moving train or in an airplane I think airplane is fine I guess maybe in a fast moving train uh because you are in a high high speed motion it loses track of what it sees again this model could be helping there I'm not sure if segmentation would help there but at least tracking and a lot of other things could be helping in a lot of different places in the world where we live in and um I guess this is an exciting model I cannot wait to put out the Google collab notebook and then give it to you all for you to try it out but meanwhile you can go to this particular Deo from Facebook and then try it out let me know what you think about this model if you have felt any use case or any application for this kind of model see you in another video Happy prompting