The Computer Vision Process: Fireside

Key Takeaways

hey everyone thanks for joining us again today we're here to talk about the computer vision pipeline and some of the steps that are included to take uh a batch of images and result in a trained model um so joseph go ahead and start us off like what what the first step might be yeah i mean so when we talk about the computer vision pipeline uh you got to think about each of the component parts um you have to think about first and foremost what problem you want to solve which is pretty easy to skip that step and just think about oh i have a bunch of images or a bunch of videos but what problem specifically is it you want to solve do you want to monitor the presence of something do you want to look for package theft do you want to count the presence of something like counting the number of cells that ran an experiment or identifying weeds from like aerial imagery so you can more targeted deploy herbicides and things like this um but think very specifically not necessarily narrowly but really specifically in defining your problem and then from there i mean the steps that you follow are pretty common collecting your images getting them identifying which ones should be sent for labeling um labeling your data uh pre-processing your images so standard things that you need to do like resizing or maybe increasing contrast augmenting your data does it increase your data size representation and improve your model's ability to generalize train a model which also involves a little bit of model selection and then check out the inference and see how it does in a given test set deploy that model and then continue to monitor its its performance over time and especially key at that step is continuing to collect image or video data from your inference conditions so that you can continue the process all over again yeah that that seems like a pretty good summary i i definitely think it's you know each one of those steps is uh kind of complex and there's there's a lot involved so maybe um now that we've kind of summarized the whole pipeline maybe we can dive in a little bit into each component part um which do you think is the hardest like if you you're like you're going about building your model which step do you think causes the biggest hurdle yeah so so that's actually a good question and um i i think you know anyone who goes through this process is going to feel like one of those steps is harder for them than you know this is just my unique point of view but i would say the hardest part for me is actually the deployment of a model so that's the step after after you've determined you know through tests and through through different metrics that the model is actually going to be good enough to do what you want it to do but then you actually need to bring it onto a device or onto a server to be able to make those inferences continuously so that's the hardest part from my point of view but what do you think i know but that necessarily that it's the hardest but i think like the most overlooked or at least like the biggest area for improvement and growth in vision and probably machine learning more generally is that computer vision is you know it's only so much about actually the code that you write or only so much about the model that you select and it's all about the data that you use and it's kind of remarkable right when you're debugging model performance understanding why you're getting the inferences you are or the quality performance that you are is entirely about the data that you're training on and so it's amazing that we have such little emphasis on i think even today in our tooling and our understanding of what data are we collecting what data are we sending for labeling do we have balanced classes do we have augmentations that represent the inference conditions uh and basically making this be an ongoing iterative improved process so one of my biggest tips as a result of that is you know you're never going to have a perfect model that's you know it's a trope at this point but people don't internalize that that means that like usually the highest leverage thing you can do is get an initial model to production that solves one part of the problem doesn't have to be perfect but identifies one class that you ultimately want to do many many more classes and then make it really easy to collect additional data and create safeguards in failsafes i should say for when the model doesn't produce an inference that you want if you do those things get a model quickly to production and put in fail-safes for if the model is incorrect and collect additional data your performance is going to cascade and improve on a much faster iterative basis than thinking that you need to have every class or perfect model or a thousand inferences a second or something like this definitely the scoping in the very beginning is extremely important to be able to actually get these things to work and actually deploy them uh all the way through to production so i definitely i definitely agree with that yeah what have been some of the like i don't know maybe unintuitive or tips and tricks or hacks that you've picked up along the way as you've taken a problem and then broken it down into these steps like what are things to watch out for what's like a quick tip that your fifth time self would have told your first time self or your 100th self would tell your first time self well i mean certainly the thing you've already been talking about um with uh narrowing the scope to something that is definitely going to be achievable but i think uh another just sort of general fact about the whole pipeline that i think often ends up being the case is people will spend a lot of time gathering a data set that they believe is entirely representative of the problem that they're going to tackle they do a lot of work to improve their model on this data set choosing models making data augmentations and going through that part of the pipeline um that you were that you were discussing to to get to a really good model but then they take it out and put it into reality and realize that uh the reality of the deployment situation wasn't exactly what they had um when they initially gathered the data set so being very careful and very real with the end state that you wanted to be in i think is very important because otherwise you'll spend a lot of work optimizing something that isn't exactly uh what you wanted in the beginning um so you can use other data sets to kind of get quick uh you know like some quick progress and start to make a model out of things but um there's no no sacrifice for directly collecting images from the exact state um that you want to be ending up in yeah makes a lot of sense makes a lot of sense um why robofall why does roboflow make this process easier or wow that tool yeah i mean so i think one of the i mean so we've we've kind of been talking about these different pieces of this pipeline in uh abstraction you know but at the end of the day each one of these pieces has a variety of different implementations and there's a variety of different uh services that are out there to be able to handle each one of these things for you um the best part about rebelflow is that it kind of sits there in the middle and you're able to actually integrate all these pieces together so you can move quickly throughout the process and experiment with different pieces without having to go back and rebuild everything and so that's that's primarily through data integration um but then beyond that you can kind of use roboflow as a source of record as you're moving through the process to know that okay once i got from here to here i was at this state and here's what my data set looked like and as joseph was elaborating on it's really all about the data set and to use that as kind of a central locus as you're you're working through the process yeah yeah i mean one thing we really care a lot about is interoperability and allowing the best tool to be used at each component part of the process so collect your images from any given security camera raspberry pi video camera whatever it is label them wherever you would want to label them reviews outsource labeling bring them in inspect understand try out a bunch of different image and model formats convert to various places um but yeah we think a lot about how do we accelerate people's workflows and really be the tool to empower a whole generation of applications and developers that are going to build and are building our future of understanding the real world around us um and so when it comes to computer vision pipelines and machine learning pipelines more generally i think the the key takeaways are really get something working make it easy to improve and continue to focus on your data um you follow those those key quick tips and you can pretty quickly start have industry leading applications it's a wide open field well that was a lot in one one fireside chat would you say yeah yeah thank you thank you all for joining don't forget to like this video subscribe for additional verbal full content and as always happy model building good luck see you in the comments