Label Data with Segment Anything Model (SAM) in Roboflow

Key Takeaways

few days ago we released a video about segment anything modeled by meta Ai and in that video I mentioned that our engineering team was working on the integration of Sam into roboflow annotation tool and today we can take a look at the results of their hard work I selected three different data sets from roboflow universe and we will pick few images from each of them and re-annotate them using some so without further Ado let's dive in and let me show you how fast you can annotate complicated objects with smart polygon in roboflow editor okay so let's start with one of my favorite data sets in roboflow universe and this is the concrete cracks data set I actually used it in one of my previous demo videos at our channel before so before I started recording I created empty project in my workspace and right now I'm just copying images from this already annotated project into mine and hopefully I will be able to recreate those annotations but much faster let's keep the whole copying process as it is not the main topic of this video and jump straight into the editor pick smart polygon tool and hover it over the crack and after split of a second I see the preview of my polygon I just click and I get the first version of it we can see that it only got highlighted partially but while I'm still in editing mode I can just pick one more point this time in the part of the crack that was not highlighted originally and now our annotation is complete we have the full crack mask we can just hit enter it will convert our mask into polygon hit enter once again to confirm that we want to use crack class and we are done it was pretty fast so let's annotate one more we just hover over the crack wait until the editor will show us the preview of The Mask if it's acceptable we click and then hit enter to confirm hit enter once again to confirm that we want to use crack class and that's it now in this image there are actually two cracks the other one is very thin so I try to annotate it hovered over the crack but didn't get any preview that would be acceptable so we need to skip that and annotate it manually so stuff like that can happen especially in case of long thin objects unfortunately not every mask can be obtained with some let's take a look at next image here the case is much simpler got the mask right away so we just click enter enter next image same drill we just hover over the crack click when it's acceptable then enter enter to confirm and that's it I guess you can see that at this point the process is getting quite repetitive so we will jump into next example but like we saw even in case of hard to detect objects Sam is doing absolutely fine and in most cases our work is done in just few seconds next up we have blueberries so this is data set that already exists in roboflow Universe however originally it was annotated using bounding boxes and because there is a lot of objects on every image I thought it's a very good example that we can use so once again we have empty blueberries project in my workspace and I'm just selecting few images copying them into my project and we will use Sam to convert those bounding boxes into segmentations it's actually pretty easy I can just click with right button of my mouse on the bounding box and select convert box into smart polygon and that's it this obviously increase our labeling productivity tremendously and notice how bounding boxes around the objects are not necessarily super tight and some can still figure out which part of the bounding box contain the object that we want to annotate and which part of the bounding box is basically a background that needs to be removed and at the same time even when one object is occluded by another like in this case the blueberry is occluded by the leaf it can still figure out how to properly annotate that which is you yeah super awesome okay so let's speed up the recording a little bit as you can see it's pretty repetitive process so I'm just using the same functionality over and over again but here here is quite an interesting example so you can see that within the same bounding box we had a blueberry annotation but there was also a leaf and interestingly enough the leaf was in the foreground so some thought that in this particular case we are most interested in the leaf not the blueberry because it's kind of blurred and in a background so I could just remove a prediction provided by some and use the manual tool to hover over the blueberry and select this mask instead so because it's not fully automated process you can still fix your mistakes along the way okay so I think that I will spare you looking at me converting those bounding boxes into segmentations and and we will take a look at the final result so here it is the initial image annotated with bounding boxes for object detection on the left side and the same image but annotated with polygons on the right side the quality of the masks is really good but what blows my mind the most that whole process where I converted the bounding boxes into polygons fix few things along the way and even added some annotations for some blueberries that were not annotated before took me four minutes to do I'm pretty confident that with the traditional approach where I'm just drawing the polygons in the editor that would take me easily 30-40 minutes and I'm not even sure if I would be able to deliver the same quality of the annotations in case of both demos that we already saw there is one common factor and that is the quality of the images in both cases we used higher resolution images with easily recognizable objects even if one object occluded another it was quite easy to figure out where is The Edge between them so now let's take a look at a bit harder use case and try to segment blood cells on microscope images similarly as before I copied few images from public data set into my private one and now we will convert those bounding boxes into segmentations let's start with the huge white cell in the middle that was most likely the easiest of them all similarly like in the case of blueberries we just hit the right button and use the convert bounding box to Smart polygon functionality and now you can do exactly the same for other blood cells we can see that even though that the overall resolution of the image is much lower and the quality of damage is also worst it is doing quite well for most of the blood cells from time to time we we need to pick the manual tool to create new annotation because not every blood cell is annotated but the process is very similar to the one that we already covered when we annotated cracks so we just hover confirm the polygon confirm the class and we are good to go the next few cells are interesting because they are close to each other and they also heavily overlap each other but we see that we had no problem with the first one we got a bit weird polygon for the second one and the third one was also generated quite well so now we need to zoom in to fix our polygon we just click on the anchors that we want to delete and drag the and cores that we want to move and after just few adjustments the mask is ready let's speed up the video a bit and take a look how efficiently we can convert the bounding boxes into the masks but also how easy it is to add annotations for objects that were not originally annotated like previously in some cases we need to do some adjustments either by manually annotating or just editing the polygon provided by Sam but after just few seconds we are done so there you have it segment anything model supporting polygon annotation in roboflow editor I highly encourage you to use it and let us know in the comment how it did on your data set remember that it works best on high resolution images but you can certainly give it a try even on those small ones best case scenario you are doing your annotations almost automatically and you just supervise the processed worst case scenarios you still need to do some manual adjustments either way you save a lot of time so if you are doing segmentation annotation in roboflow editor and you are not using some you are certainly missing out okay that's all from me today I hope my tips were useful in the meantime let I can subscribe and stay tuned for more computer vision content coming to this channel soon my name is Peter and I see you next time bye