Computer Vision Model Types

Key Takeaways

[Music] hey everybody this is jacob from rebel flow i'm here today with joseph from rebelflow to talk about different problems in computer vision and the different technologies that are being used to solve them so joseph what are some of the problems you're seeing today and what are some of the techniques that people are using to solve computer vision problems at its core computer vision is making sense of of images right so image recognition identifying contents of what's in an image making sense of video and when we think about the type of techniques that that breaks down into there's kind of a neighborhood of different problems uh so some of the problems that we see range from things like classification to object detection to semantic segmentation to key point detection um and there's a you know host of other types of sub neighborhoods or new problem types that are emerging every day i think it might be useful to maybe double click into what some of these techniques are and problems that you can solve with each of them so maybe we can start with classification um you want to break down what classification is and we can talk through some use cases yeah sure so classification is kind of a traditional ml technique that is used to process data and then transform it into classifying the the data that you're looking at into a series of classes that you want to classify it into and this has been commonly used um in text and just kind of for all different data problems but for an image it's just simply taking that image or frame from a video and then segmenting it with one of the labels that you want to do so for example you might want to decide if an image that you're looking at has a dog or a cat in it and just kind of draw that label across and apply to the entire image that's going through so naturally that's kind of a pretty it's sort of a more trivial task because you only need to make one prediction for the data that's going in but some of the other techniques in computer vision get a little bit more in-depth and a little bit more about localization of where things are so for example like object detection is another technique that is similar to classification but it goes a little bit more granular if you want to go into that one yeah so i mean at its core i think classification is adding tags to things it's you have an image let's add a tag to it and then as you alluded to object detection allows us to drill down with a bit more specificity so object detection is identifying and localizing where in an image an object is right so if you had an image of well say a bunch of dogs that are present in a photo object detection is and you want to find the dogs object detection is drawing bounding boxes around each of those dogs that be present in the image the reason that's different and more powerful is you not only know of course that there's a dog or multiple dogs in this image you actually know where they're at in the image which allows you to do things like count to know where in the image frame a given object is present and provides a deeper level of intelligence of what video or image you're analyzing now you can get even more fine-tuned and so that brings up the other problem type that we were discussing semantic segmentation so maybe i'll pass back to you and you can describe the semantic segmentation task and we can compare and contrast that with an object detection task sure sure so for semantic segmentation it's kind of like object detection where you're localizing objects in an image with semantic segmentation you're actually drawing a mask around the exact outlines of those objects so it's actually even more specific in where it's annotating the contours of different objects in an image so this can be useful if you're having to get precise measurements of area or precise pixel measurements um but naturally you know that's a lot harder of a task for a computer learn so what do you think what are some of the reasons why you might uh compare one or the other or be choosing semantic segmentation or object detection or use cases where those might be more more prevalent yeah so at its core like the technique you want to choose is the one that's that's right for the job whether that's classification object detection or semantic segmentation now in terms of comparing contrasting uh a task that you might be able to do with with each of these let's say that for example you had a a field of plants uh so you grow tomatoes and you want to count and then actually know the size of leaves on those tomato plants right well a classic at the very very beginning like maybe a classification problem for this would just be is there a tomato plant in this photo or not right you could have a a leafy green but is that leafy green a tomato plant or or not in that in that photo at all um but maybe you want to know like where in the photo that tomato plant is because let's say you're making a robot that's going to go down and maybe automatically pick the tomatoes which means you need to know where the tomato plant is well then we would need some level of localization so something like object detection work work really well if we train the model to recognize a box around the plant and perhaps we also need to know how many leaves and the size of those leaves on each of those tomato plants so again we could use optic detection to identify the individual leaves on the plants and we could count and say you know this tomato plant has six leaves but let's say we want to get even more specific and we want to know not just the count of the leaves but the shape and the exact area of those leaves well we do an object detector object detection model object detector that finds the leaves and then once we had just the presence of the leaf we could use a traditional computer vision technique like thresholding to say you know where does the leaf start and stop relative to its background or we could build even a semantic segmentation model that might do a good job of creating a mask around the individual leaf and then we would know how many pixels are in the area which would allow us to basically create a measurement of those leaves so it's kind of like taking one problem and breaking it down into each of those parts of the task but i think we can compare and contrast you know why you might want to choose one of those over another aside from like it fits the problem well so for example like let's say you are counting leaves why wouldn't you just train a semantic segmentation model to count all the leaves why might you want to do an object detector yeah so it really just all comes down to accuracy and costs um basically you know the as you pointed out a semantic segmentation output is going to basically subsume all of the other techniques because you could create a bounding box from the semantic segmentation mask that you um have created with the with with the somatic segmentation output but it's going to cost a lot in annotating because the annotations are going to cost a lot more to create and then training is going to be a lot more difficult because it's a lot more for the computer to learn and ingest and learn how to model the task the modeling problem is going to be a lot more complicated and so along that as you're thinking about migrating up through these techniques you have to consider those those trade-offs as as you're kind of deciding how specific you want to get in our experience the the object detection space has gained a lot of foothold with different technologies there are a lot of easy tools that you can use to move your object detection problems forward a lot faster and a lot of times you can solve problems very effectively and efficiently um just with this technique without having to go up uh to the next level of of specificity um but of course you know as the field evolves these all these technologies will be getting better and you know they'll be getting easier to implement but right now that seems to be um generally the state of things yeah yeah one one sort of like programmer shorthand that i've heard that's kind of funny is like you don't store every numeric input as a float you use an integer like when you want to like enforce you know that there can't be decimal places or maybe it's going to be more memory efficient and that's kind of like a useful way to think about selecting the the right technique i kind of like that and grab onto it um and then one other thing that kind of sticks with me when thinking about this problem is um andre karpathy the head of ai tesla did a talk this last summer on some of the problems and vision problems that tesla self-driving team faces and as a world leader and kind of a world-class model of how to perform computer vision problems i think we have a lot to learn from the techniques that they apply one thing that stuck with me from that talk was carpathi talking about how few semantic segmentation problems they have and how they actually try to frame problems as object detection problems because of the things that you mentioned the cost of getting annotated data and the frequency with which you actually need a pixel map versus just knowing that a parked car is right over here on this side of the street then you know the exact outline of the parked car you know the localization of where it is as the moving car drives past it and i found that to be a really insightful um reason why you know use the tool that's right for the job if object detection models can perform more fat like more quickly more accurately there's a greater array of them and you get data more cheaply then all things considered it might be a better technique for that task at hand as with all things it kind of comes down to how you frame the problem itself as to what technique is going to be most useful and just as you said jacob it's the field is generally going to continue to evolve and these techniques will get better uh and these parameters will kind of change framing each of these decisions so i mean the kind of things we didn't touch on are like key point detection and some of these other techniques but at a high level i think that gives a really good overview of methods in computer vision from classification to object detection to semantic segmentation to a few others example problems of them and why you might choose one over another thanks so much for tuning in to another fireside chat with roblo