OpenCV + Roboflow: Getting Edge-y - Computer Vision Deployment Techniques

Key Takeaways

[Music] so is it live on linkedin i'm about to hit go live and see what happens my guest this week is joseph nelson back for his uh record record-breaking third appearance i think should we get started let's get started hello everybody welcome to opencv weekly webinar and today we have with us joseph nelson who is the ceo and founder of roboflow and roboflow is also a silver member of opencv they support uh our community so thank you so much joseph for for being so generous over the years and also your style of interacting with the community the value you provide to the community is awesome so we are very very uh happy to have you on uh on our webinar every time you are here uh i mean those shows are very popular uh you know not only on on on uh the live version but also the youtube version people love what you have to say i really appreciate that yeah it's a lot of fun you all make for a great show the audience continues to grow and as i think i said to phil last time we'll continue to make it fun just as always that's great and we also have uh phil nelson on the show the two nelsons are not related much like buster and babb's bunny we are of no relation um that one that one's for the 90s kids out there uh yeah welcome welcome back to opencv weekly webinar everybody if this is your first time joining us there are a couple things you should know one is that we do zoom q a from you and the audience here you on linkedin this week will probably not get to participate in the q a but maybe next week if you're in here on zoom please use the zoom q a functionality to ask your question at any time during the webinar and we'll do our best to answer as many as we can at the end of the webinar or if it makes contextual sense we'll ask it during the show also we'll be doing a giveaway of uh well i'm satia would you like to tell them what they'll win with our trivia giveaway this week do we have a giveaway from roboflow so joseph uh here on mute yeah we have uh our standard giveaway of a few things one is free gpu credits for those that build and and operate with roboflow which gpu increasingly hard to to get access to and then also we have um roverflow swag packs um i do have to note that those um have to be shipped within the us but uh we'll find we'll find the right sort of winners to to get to for both those prizes i appreciate it thank you so much um yeah and so that'll be given away to uh whoever answers our trivia question which will be after the presentation here so stick around you won't want to miss that so yeah and the uh you know the presentation today is very interesting um joseph is going to tell us about deployment uh techniques for the edge this is uh actually not trivial a lot of people struggle with this they train a model um you know usually it's that's the simple part especially if you're doing simple things like object detection or image classification you can train it very easily using keras or pytorch but then starts the difficult part right you try to take this model to the edge and it is slow it doesn't work you cannot get it uh you don't have all the components in place so this uh this session will teach you a lot about how to make do this deployment easily so joseph take it from here happy to yeah looking forward to today's conversation i'll share my screen for starters and as usual uh we'll make today be both a mix of education and interactive content um so the topic today is uh talking about deploying um now we titled it getting edgy right because we can deploy to the edge but i think we're also talking through a number of deployment techniques and we'll walk through ways to get computer vision models into production the benefits and disadvantages of a couple of those ways and tips that are generally useful across each of those those techniques um so as always it's a pleasure to be here and i'm looking forward to talking through some of the the options um of course at opencv you all are experts with partnership with luxonis and the opencv ai kit the oak devices which i like to think of as like the raspberry pi for computer vision we have tons and tons of folks out there that build with roboflow and with oak making all kinds of fun techniques and tools and so we'll talk about those as well um right off the the get-go here i i did uh include a slide that introduced uh at least satia and myself um phil of course is our fearless moderator and an essential part of ensuring the success of today's conversations though for better for worse you'll mostly hear from satya myself as we go through the content today um and as satia mentioned i'm the ceo at roboflow and cetia is the sea open cv ceo um phil it looks like you wanted to come off mute there for a moment too please chime in i i'm i'm biting my tongue i'm trying to stay civil i know that we're live on linkedin i'm kidding we'll get your your bobbing face in here uh next time um but yeah i mean i think the it's always helpful to provide a little bit of context on both opencv and what roboflow are um to trfl you voice over a little bit about opencv of course most of the folks i'm sure know you but they might not know the scale and um breadth of the impact that the opencv library has had over a couple of decades right and especially you know we also have people uh who are your audience overflow audience who may not uh who may not be familiar with so opencv is the biggest computer vision library in the world we have been around for 20 years more than 20 years 21 years actually it started in 20 uh the year 2000 it was open sourced by intel and a lot of people don't know that opencv was also part of the darpa grand challenge in the year 2005 when uh stanley the car that won the darpa grand challenge in 2005. it was the first car that completed the uh grand challenge and won the prize it's in smithsonian museum now uh that car internally was using opencv as the computer vision library it was using a whole bunch of other sensors also but the computer vision library was uh opencv and uh since the last few years let's say let's say the last three years we have been growing beyond the library so opencv has courses opencv also has opencv ai kit uh oak which is a series of smart cameras uh which can do you know neural influence on the device itself and some of the variants can also do depth estimation uh on the device itself in real time and uh as part of that expansion we also reached out to people who have similar values and that's where roboflow came in we you know we saw what roboflow was doing and we invited them to be our silver partner uh which supports the opencv uh community it supports the development of the library etc so yeah that's that's a short bit about opencv i think what's really impressive about opencv is just the longevity and utility of the of the library right i mean well before deep learning and machine learning came to image technology and image processing opencv was there with some of the like more traditional techniques and has continued to evolve and stay with the times as uh increasingly there's newer newer technologies and i think because of that um because of its developer-centric approach which we really agree with as well at roboflow it allows almost having like the building blocks so we have a lot of like users that will build tools and technologies with roboflow and with opencv's libraries so they'll like do machine learning to identify a specific portion of the image but then they'll do say thresholding or a technique like this of the dozens of algorithms that are out of the box in the opencv library as opposed to processing of those detections and so it's been a really um impressive uh and continued growth and as you mentioned we've even actually now had um some customers that have built with opencv consulting and with roboclock which has been cool to see too nice yeah and uh also you know uh talking about uh the longevity we recently in one of the top conferences in computer vision the old feature you know there are these features sift surf and orb which are used for feature matching it won uh it won an award for the big impact you know some big impact award that over the last few years it had the biggest impact on uh in the industry and that feature was actually developed by the opencv team you know gary bradsky is on that paper ethan rubley who is closely associated with opencv uh he was the primary author etc so yeah i mean there are a lot of algorithms etc that that have been around uh developed through opencv and part of opencv i'd like to note that this is the second week in a row that we've talked about an orb of some kind and i am 100 down with that that's great um yeah so lots of respect and for the same reasons we're eager to be supporters of continued open source and implementation of of computer vision for the masses um oops at roboflow um you can think of us kind of like the building blocks that developers use to bring computer vision into their products when you want to use computer vision within a product it requires a relatively wide array of tools everything from how to handle image collection of video and imagery how to organize that imagery in video for identification of what should be annotated actually annotating data and having models an automated annotation of that data or even third-party services that annotate data for you pre-processing and augmentation to increase data set size and variability training of those models whether you want to use open source models that roboflow is interoperable and integrates with like say the yolo family models are quite common in computer vision land and then also deploying and deploying things to a wide array of techniques to the edge as we'll talk about in great depth today but also to microservices whether that's apis mobile devices or even directly in the web browser and then critically i think the thing that roboflow users really get a lot of value from is closing the active learning loop so once you have a process by which you can create and deploy a model it's essential that that model continues to learn from the environment in which you're operating and precisely because you know the images we might have in our training set are not representative of the full world around us it's key that we're able to continuously identify and improve and expand the scope of what those models can accomplish and so roboflow kind of ends up being this machine learning operations tool that orchestrates the ability of you know kind of running that ongoing loop where users will plug in different parts along the way right so they might have their own kind of open source model that they train on they might have a different way of collecting imagery might be using some opencv algorithms we're doing like thresholding or edge detection and it's that interoperability that um you know is really part of our ethos being created and run by developers for other other developers and because of that i mean over 50 000 engineers have built with roboflow including those from over half the fortune 100 i mean we count customers like walmart and cardinal health among those that build with roboflow and um continue to see the enterprise adopt and use these tools in impressive ways so on the right hand side i kind of have like you can imagine it's like a animated in a star wars way i toyed with that but it was a little too distracting so i just have like the floating words on the right hand side of the slide but you can use your imagination to pretend that they're further further going out in this distant space but all sorts of different ways that people have built with roboflow things like gas leak detection like determining if there's like crude oil releasers or high volatile liquids tracking pitches um i could use that i'm pretty lousy unlike the pitchers man roof damage estimation so if you have a drone that flies over top of homes and you identify you know if there's hail damage or if there's an insurance claim to be made satellite imagery analysis everything from like you know counting like swimming pools to determining um if there's given structures tennis ball tracking we actually um roboflow is used at wimbledon each year for doing things like tracking the tennis ball as it comes across um the chord and does improved camera operations uh smart retail checkout technologies measuring fish it's kind of funny like the number of developers that are working on fish problems is insanely high that seems to be everything from like identifying fish to tracking them to seeing their speed to ensuring that there's legal phishing that's happening public transit occupancy counting right so knowing which bus routes or train routes are most common or knowing which parking spaces are being most utilized is all like an intelligent image analysis problem identifying plant health so everything from like you know knowing not just the counts of say the number of berries that are growing but the stage of those berries what yields farmers could expect cataloging inventory and counting things under underneath microscopic cells i mean truly we like to joke um that the use cases that we've seen for computer vision are everything from under a microscope to outside a telescope and it's true everything in between that is the observable world i i had a similar you know i like giving ranges for for our consulting services i like to like to give this range that um we we actually work with uh at one point we worked with a company that analyzed uh horse feces for parasites right so that's one range and on the other side we worked with a company that did high-end fashion uh and we had to identify high-end you know fashion handbags so that was the range but that's that's another reason so you actually almost literally had uh telling ship from shinola i had never used that expression now now i can great that's great that's great i mean recently um on uh roboflow we actually had a curious hacker and they put out a youtube video and i'll be sure that we include it in the email follow-up but um their cat during the pandemic they were concerned that he wasn't getting enough exercise right uh and so you know you can point a laser pointer and a cat will chase it well he bought one of these kind of toy robotic arms and then built a model or collected some data of his cat and then annotated it with roboflow and then built a model and deployed it to track where his cat is and then he did just the very standard rule based method that wherever the cat is add basically like five feet and then shine a laser pointer there so then his robotic arm is just pointing the laser all around and his cats jumping around and doing exercise and so at roboflow you know if you need a cat exercise machinery we're the tool i was actually surprised joseph we cannot obviously mention uh but you have some a very big clients you know uh fortune 500 companies uh using this tool which was surprising i always thought that roboflow was a tool for startups and you know uh companies who that are you know less than a thousand people right uh that's that's the sweet spot but i was pleasantly surprised to know that you know there are a bunch of fortune 500 companies that are using your tool at least different groups in these companies which is a very positive sign you know speaks a lot about because these companies can build their own they are not building their own and coming to roboflow which is great yeah you know what's interesting is that like the same types of problems in fact that like a individual hobbyist might face a large enterprise spaces like which images do i annotate or like how do i get my model deployed in a reliable way and i think the thing that um you know startups are usually quicker to realize that company larger companies uh are coming to conclusions of as well is focusing your resources on where you have comparative advantage is the most efficient use of those resources so for example if you have if you have teams of developers and those developers work at um we'll say like a energy company you know the thesis of that business is to ensure the safe reliable efficient cost-effective delivery of say crude oil or natural gas or highly volatile liquids and so using engineers time for uh you know reinventing the wheel and building your own annotation tool or building your own mo ops pipeline you know you don't it's for the same reason that like we don't build our own data centers right like we use we use the best in class and then focus on the part of the value chain that we can add value to and i think that like for that reason you know like there are companies whom you know vision is their core innovation right so like you know tesla is a famous example carpathia's team has built you know they built their own chips let alone their own models and their own tools their own everything along the stack but i don't what i think was interesting is that you know not every company needs to reinvent the full wheel or the full car if you will to be able to use computer vision in really impressive um and in value-driving use cases and i think as companies continue to realize that they're getting the value more quickly by using i kind of think about it like the lego bricks right like use the lego bricks that allow you to build your your house more quickly and that's why i think developers are such a impressive um group of people because they're creative there's there's no limits whether that's you know finding horse poop in your example or finding like cat exercise in my example but also um curious too and so providing engineers with the tooling to get to value faster i think is kind of a win-win and we're continuing to see that with the adoption just across enterprises as much as startups and hobbyists and students i think um that's a good segue to uh how we kind of think about the deployment portion of where what are we talking about today specifically i mean so we talked at a high level of the machine learning operations pipeline is you know there's everything from collection and organizing and labeling and processing training deploying and displaying we're going to zoom in on perhaps perhaps the most fun icon the rocket icon of deploying and running models successfully and the things we're going to talk about today are not like this is how you need to do it with rebelflow no we're going to talk more generally when you're deploying a model what does that mean where can you deploy it what are considerations to bear in mind what are best practices what are tools that allow you to do that who has the like best hardware along the way um you know of course i'll provide commentary of the tools and best practices we've seen from users but by no means is this something that says oh this is the only the um products that are within the the robovo suite of products in fact more broadly i think computer visions democratization and usage is far beyond any single company and so the more we can empower its use and allow people to be aware of the technologies that are at their disposal the better and so we're going to zoom in on that that deployment bubble um in a moment but satya what if you had any thoughts or comments here before we well i i was just smiling because uh those thoughts are very similar to mine whenever we are talking about you know computer vision in general the pie is growing and we have to contribute to uh the growth of the pie right not try to you know get the biggest chunk of the existing pie you make the pie bigger there is enough for everybody exactly exactly um awesome so let's let's talk about um deployment um i very creatively call this slide deployment let's talk about it so i think there's kind of four things to chat about today i mean what is deployment i think folks probably on this call have a good understanding of it but we can give it a really nice bow tie uh definition then we'll talk about the ways in which we can deploy models um we'll talk about what success looks like and common pitfalls to avoid and then we'll do some audience q a i suspect most of our time will be spent on number two here of like the different ways you can deploy compare and contrasting what does it mean to go to the edge what are given hardware decisions um but of course as as is common this is sort of a loose agenda and the questions that we get and the conversation that's tfo and i have will drive where we spend the most time um when we talk about deployment i think the the way that i kind of think about it is it's basically the process by which we use computer vision models in production right at its core it's like you've got a model now deployment means you you're not in the jupiter notebook anymore you want to have that in a way that you can use wherever you want to use it and you know that might be microservices via apis it might be embedding it on the edge on mobile devices it might be running it directly in the browser um or anything in between one idea that i really want to hit on before passing it to t as well is that when we talk about deployment in a software engineering context it's actually a bit different than we talk about it in the machine learning context in the software engineering context you know you kind of um you you deploy a new release of software and it goes through your ci cd pipeline to ensure that it passes all tests and then the software is out live and deploy in this context is kind of like a one-time i mean you're going to deploy multiple features but you kind of like it's a moment in time in computer vision and machine learning more broadly deployment is a bit more like setting up a system to run and it's less like oh okay i'm going to push this button and i'm done it's more like what's the compute that's going to power my model in an ongoing way and i think that's an important cognitive switch that is uh required because it begs question of of what level of compute do you need where is that model going to run what sort of throughput do you need and um it's much less like a click it and forget it sort of interaction it's much more a how are we going to create a system and include the considerations for computation and accuracy and redeployment that allows us to be successful so my like crude analogy i've been working with i'll workshop this and see if you all like it is i kind of think about it less like a ci cd pipeline and more like uh like maybe like cyber security software or like cdn software um cdn products that you know enable you to have your model or in this context your web page run at the edge and cache your images and things like that and it's an optimization to be made in that context rather than a one time you know push the button deploy the new new pushes software so i i don't know i i welcome your your thoughts here's a tea of like a mental model around this yeah i i agree so it's quite different from you know regular uh software dev uh development deployment because uh traditionally what you we've used to do is you are developing a software right it's in your local environment the environment changes but the code doesn't change right the code is the same when you deploy it it is just a different environment right it needs to uh take care of that and those things can be handled if you have a good uh you know uh continuous deployment pipeline in machine learning though it is there is one level extra added complexity the model you train is not the model model you deploy right and i'll give you an example we built a background subtractor for uh for our or for our own internal as a product and then a client came to us they wanted to license this and uh which is fine you know we have this model but uh which runs great but then uh they wanted it to run on the cpus okay so we have a cpu version that's one deployment they wanted it to run in a browser so we have a webrt based uh deployment so that's the second one they wanted it to run on an edge device so we have an openvino based uh some optimizations we did right so you can see that the model does and these models are not going to give you the exact same results you have to ensure that they are close enough and the deployment environment provides different they are all different environments and the goals are different in some cases accuracy is paramount right when you're running on the server they say that okay use as many resources as you want use the gpu if you want but give us very accurate results uh on the browser for example it's like okay the accuracy doesn't matter as much but you have to make it at least 15 to 20 frames a second so that you don't see the lag as much so you can see that we started with a trained model but the deployment scenarios were very very different and we have to customize the model itself for different scenarios in traditional software deployment you don't think as much you don't spend as much time on deployment you you spend time on the initial setup but in machine learning this is a big deal you know you you have to allocate a lot of time for this piece so those are my two cents i think it's a really really good salient example of calling out you have a model that does something in your case background subtraction but where you're using that model dictates what and how that model makes its way to production in other words it's not like those same exact layers in that same exact architecture and so like people will commonly ask you know like pytorch tensorflow dark net all these sorts of things and actually you know it kind of like the the answer is it will depend on where you're using the model like if you need a really high throughput on nvidia devices you might want to optimize your model with tensor rt if you are um satiated with you know like a near real-time performance and you don't need to be truly like bleeding edge then just deploy it as it is from tensorflow or pytorch will work in the same environment um and i think that uh you bring up a like the way of framing it of the model running or the model being trained is distinct from the actual same exact uh architecture that gets deployed is a good way to highlight that um so there's a tactical difference and i think which is why like when solving a computer vision problem we need to ask ourselves where do we think we want to run the model early on in the process because that will frame some of the decisions that we make um along the way and the other thing i think to call out is so there's that kind of that like tactical element the other thing i think to call out is um we want to include from from day zero how are we going to continue to collect images of the failure cases or even of expanding the scope of what our model is doing let's say we have a model that's like i don't know texts us whenever there's a bird on our bird feeder but then over time we want to actually get texts when it's a cardinal versus a robin versus a more maybe unique bird and you know that requires continuing to collect images of each of those bird species and then labeling a bit more specificity and then redeploying that model then collecting new images of all the various birds that might be appearing at our neighborhood bird feeder and so in this context it's not just you know model getting better but it's like how are we continuing to sample and improve and and and deploy and i think that um this is often the difference of like a production grade system and kind of like a a one-off you know is that like a one-off system like kind of like hobbyist projects or um i kind of like to joke sometimes you end up with like these you know rube goldberg machines uh it's like you know you put you put in like the quarter it rolls down it hits like an egg the egg rolls and falls and splats that's enough weight to cause like a ruler to go up and pretty soon all the way at the end you know you have this like winding series of things that result with i don't know like toast coming out of the toaster or something like this um that's sometimes what i think uh deployment pipelines kind of end up looking like is that like oh yeah yeah yeah of course of course so you train the model and then once the model gets trained just download it and then also go over here to this random script we have that will convert it to openvino but then once it's an open vienna be sure you come back to this other script that we had that john wrote before john left but don't worry it still works because you know john had wrote it and john was a good engineer and then you put it back into here and then it works you know easy and so it's like thinking about um having a robust system is is also important for this interaction right one last thing is um you know in traditional software when you deploy you could say that oh i've done enough testing etc so the deployment is not always set up uh to receive feedback right and it's not necessary that uh feedback comes through but in case of computer vision models it is almost necessary that the the data you're getting the model let's say it's running in the field that needs to make it back into uh improving the system right so that data that you're looking at from the field has to i mean you have to architect protect your system so that you get the data back uh into your system from the field so that you can improve it right that's crucial it's not it's almost not like an option right and many companies do that but i feel that it is crucial um because uh you know you have to train the data uh train the model based on the data that's coming from the field uh and it's a continuous improvement process totally totally yeah so i think that's a pretty good overview of deployment and like what deployment isn't and common misconceptions let's dive into the ways that we can deploy things uh now this isn't necessarily comprehensive to be clear uh it's kind of like an example of some of the most common ways to deploy models and this will be ever evolving of different ways different hardware different configurations um now that said i think that there's kind of like four common ways that that we see deployment happening um and i wouldn't get too hung up on the taxonomy being perfectly correct of separating between these i would more think about them as options and menu of options and so there's things like deploying a model via a web service and using it via api uh so for example you know a simple call and response the model exists on a remote server you send up an image or send a video you get back json of those of those detections on edge hardware and that's you know the uh the getting edgy we'll spend a lot of time discussing that on today's conversation on mobile devices you know i broke that category out but it's arguably just another type of edge right i mean a mobile device is an edge device at the end of the day but there are some different considerations so i thought it'd be interesting to to call out as an option and then directly in the browser this is i think an emerging way to deploy models that we've seen a lot of creative use cases and it'd be important to call out as an option now deployment isn't one size fits all right i mean you know one of these methods is better in some circumstances than others like if you have a website for example that we'll say let's let's say that you're building the website um that might be relevant to what satia was mentioning around um you know people that are selling used goods and you want to know if that used good is legitimate or not well in this context you're building for a web service so a user uploads an image it actually would be quite convenient to have a web api that you just call and say you know what's the probability that this is a true handbag or a uh or a counterfeit handbag and that might be an example where an api works really well now let's pretend that you have people that are working we'll say at like a second-hand retail store and they have this they have this uh the same model of determining fakes from trues of purses on their phone that might be a place for you to play that model to the edge and you know you point your phone at the uh at the purse and the phone the model's running completely on the phone there's no call and response and it's telling you oh you know fake purse or real purse and so hopefully this gives some context of like it depends on the problem of of how you want the models be used in your service as to which of these techniques will make most sense yeah web service is very important uh you know using ai as web service is very useful when um the platform is diverse right when you cannot control uh the platforms and the system doesn't need to be really real time so like if you're dealing with images etc a web service will do just fine and it is the easiest form of deployment as well in some sense um very good choice if uh if you're a one person team don't even try to do uh you know optimize things for uh for separate hardware use a web-based service and that's a that's a legitimate way to do once you get uh you know in some cases that's that's all you have to do but suppose uh uh you feel that oh the performance is not as good etc still you should start with the web api and then once you have traction for your product then you can go and uh customize it right totally totally yeah and i think that like that's a really good point is that like where you start you might you might first deploy via web api because it's like simple and easy so like if we go back to like the first mobile app let's say that you are the second hand retailer and you have this product that tells you if something's real or fake as a purse and you have it in the web products but you don't have it on the mobile app yet well you could still create a mobile app that the user says capture photo and that photo gets sent up and comes back right that would still work and then if you wanted to optimize it so like it processes real time like from the video camera live on the edge well then yes you would have to do a different deployment method and so i think you bring up a really good point that like this this can be a uh you know you can keep it simple to get to value right because we would still be getting value from knowing if it's a real or fake purse even if the user interaction could be made a little bit more seamless there is still value in knowing point to click image gets sent up image comes back real purse i'm gonna stock it yeah there is this thing called in software engineering we call it uh you know uh premature optimization so especially for startups it's crucial that you don't you first establish uh whether the product has legs or not right and for mvps this kind of interface is great uh because it doesn't consume a lot of time you establish value by saying that okay uh here's the service it can be better but this is good enough uh try it out and if you see enough people using it then you can make do the optimizations perfect perfect summary so yeah i think um an example of so we talked quite a bit about like web services and deploying via an api but basically what we mean here is you know call response you can use video or individual images and you uh call that api i mean an example like facebook for example a web service that runs many many apis and images and helps you know when you need to tag your friends because they uh saw an image a face and said who is this and you can add a tag to that that image i thought it might be fun to do a little bit of live example here if we all want to buckle up um get some good grins and we'll see what breaks when i do things live so let me um brave live deployment always always so i'm gonna stop my screen share and then re-share my screen here to see the full screen because then you can see my terminal as well um okay so what i'm going to do is i'm going to take a screenshot of our faces of phil satia and me and phil i'm going to pick on you if you want to make like a funny face we'll see if it'll find like you're you're funny yeah wonderful wonderful okay so i just captured um oh whoops i gotta redo it i'm sorry i don't know how to use screenshot yeah there we go okay perfect so on my uh desktop um i have that uh that image yeah so i have a folder called all the answers and i have this screenshot i'm gonna go to my desktop and um i'm gonna actually go ahead and rename it to i don't know photo dot jpeg is what it's gonna call still the hardest still the hardest problem in uh computers naming stuff yeah you saw me freeze there i was like what do i do photo.png is what it's going to be called and then i separately have um a model here that finds faces um and this model does does all right i mean it's trained on about a thousand about 2000 images um and so i have this use curl command uh and i'm gonna have to um i will be erasing my api key for all you folks out there after this but i will be showing my my api key momentarily for the purposes of uh of doing this this curl command um and so uh this model exists in its live and what this model does is it just finds faces right and so we actually trained it on images of some folks with a team that had masks some public data sets of folks that were wearing bicycle helmets and all those sorts of things right so you get an idea of of what's what's going on um in this data set so 847 examples little under 2000 annotations diverse set and size of images so i'm gonna go to my terminal over here um all right and i'm gonna go ahead and um your image.ping i'm gonna actually make that be photo dot png and so on base64 encoding that image and then i'm passing it up and doing a curl command directly to this model with my api key and then boom it works i get back uh these predictions here um and these predictions are in json of the xy coordinates um that's not as exciting as being able to see the results so i'm going to go actually go over here and click example web app which does the same exact thing it calls the api but it's also going to visualize the results for me which makes debugging nice so here's the moment of truth do we find phil's silly face so here's the photo um punch it in confidence 40 let's go ahead and run in friends hey there what do you mean that's just what my face looks like uh and i get the json here as well right so the json of each of these um on someone's face it's 86 percent confident the other one is about 50 confidence so um cool so that's that's just an example right we just ran the model via api and um quick call response and and got it back so if anybody wants to wants free use of joseph's api key it probably got about 20 minutes so make it quick that's a fun challenge that's a fun challenge yeah i mean you're hiring right we are hiring uh yeah which engineer can rack up our our cloud build fastest yeah so that's kind of the um that's kind of just a simple example of you know call response use the api and you get back json right and with that json you can do things like you know based on the confidence level like for example that image i might actually send back to my training set because i want maybe my face detection confidence to only be like 80 plus and the fact that i had a couple of 50 there i'm like oh you know i need more of phil's silly faces for this to work better um all right so it's time um we should talk about edge deployment um when i looked for edgy meme i got some some some suspect results and so this is where i landed of the edgiest meme that i felt comfortable including in today's presentation yeah thanks for thanks for joseph it may be useful for you know because we have a lot of uh beginners also um in this webinar uh just can you just define edge so that people know uh what an edge device or what do we mean by edge deployment exactly so yes so deploying to the edge when we say deploy the edge we mean basically deploying to a piece of hardware um that exists out in in the field i like to say so typically a gpu um and that gpu runs the model right then and there on site without sending the data anywhere else um and so at the edge means the model's running in a self-contained way um on a specific piece of hardware uh for the purposes of our application so an example of one of those pieces of hardware might be in nvidia jetson so on the left-hand side i have an nvidia jetson picture um and on the right-hand side i have the luxonis uh oak the opencv ai kit this one is the oak d and and then on front i just have like an example of a little docker bubble and the robot logo because we'll talk a little bit about why that that is helpful in this in this context but the broad idea here is deploying to the edge is when you want to have a model that runs perhaps uh offline so no internet connectivity you want to have it run real time so like on a video feed um and with less frequency you send data back to the cloud maybe there's some examples where you are extremely limited and can't send any back but remember from our prior discussion it's always useful to have the ability to continue to improve the model by some degree and also even if we're not sending data back it's helpful to monitor model health right so knowing if like an edge device is live is it is it on or off is it dead or alive is it successfully making detections even if we're not doing model improvement and of course we'll want internet connectivity to say like ping the model and say hey are you awake you're alive are you making detections and then you get back you know yes and these detections that i'm that i'm making on a regular basis so edge deployment is really common in computer vision i mean let's give some places where it's going to be useful there's things for example like if you're working on an application in agriculture you might have a model that's running completely in a disconnected field and it's doing things like identifying weeds versus crops you might have a model that's running on a drone and that drone doesn't necessarily have internet connectivity but when it um sees things from below he wants to capture images or do some sort of business logic with with the data that it sees we have customers actually one of the world's largest manufacturers deploys with roboflow into their factories and inside those factories um they have a combination of hardware that runs offline on their own intranet not making any outside connectivity and that helps verify that when the goods that they're producing come off the assembly line they're of the quality they want it doesn't result in any sort of um malformed uh things in product that goes to to end customers and so there's a number of use cases where edge deployment uh can make sense now of course there are disadvantages too precisely because the model is running in a more offline environment it might be more difficult to monitor model health another example is that it actually might be quite challenging to do um uh development uh for the edge which is why earlier satia and i were mentioning it can be useful to deploy a model to like the cloud like an example i just gave right where i we had our faces and then like i just threw it at an api and saw was it working to do that same demo i'd have to like go to play it's my oak and then like call the oak and then see the results on the oak which you know isn't um too hard or anything but it's more steps than might be necessary at the point that i'm at in my development process um and then of course edge hardware just kind of introduces its own complexity because the hardware on the edge could fail you have to purchase that hardware you have to manage that hardware and those are some some things to bear in mind that might be uh uh complex systems um so that's a high level overview of what edge deployment is now we should shift to talking about like what like how do we get high throughput how do we get the ability for things to run consistently but um i'll toss it back to satia if there's any sort of thoughts you want to add here on on uh so uh yeah so just to add to what you said uh one of the big reasons one i mean in addition to what you said one of one other big reason to use edge is uh privacy that people do not want their photos to be uploaded to the uh cloud so they may you know the edge device can do the processing and lock no photos leave the edge device you can actually configure some of these devices that no photos would ever leave the device so that's that's a big one and so performance uh privacy uh reasons and obviously connectivity when you don't have connectivity uh it's it's very useful and uh the other thing with edge device uh the the downside is also that uh you know you have to have a way of uh updating the model you know when even when it is connected right uh it is much more difficult than updating a web server for example that uh over the wire uh when you're trying to update the model or uh there's always this uh you know what happens when the update fails and things like that so it is more much more complicated when uh you're using an edge device the the failure modes are myriad like there are many more ways it can go wrong yeah um speaking of which we are button up against 9 50 a.m do we want to pause briefly and do our trivia giveaway yeah that's the perfect time um yeah jose would mind on sharing your screen real quick yeah cool so uh for those of you who have never joined us before or if you just need a reminder we do a giveaway here this week we will be giving away a roboflow gift pack thanks so much to roboflow for their generosity i'm going to ask a trivia question and you're going to answer it in the zoom chat window for those of you watching on linkedin don't answer in the comments it's not going to do you any good sorry we'll try to come up with something to some way to include you as well but the first person to answer correctly in the zoom chat window this trivia question that i'm about to ask will win some roboflow credits and uh if they're in the u.s also a swag bag so um thanks again roboflow get ready to answer earlier in the webinar we discussed a specific feature detection method that is part of the opencv library what is the three letter name of that method all right we got it we got some good ones here uh partha you won two recently i can't give this one to you uh looks like chris krishna krishna mohan is the first one i see after partha which is the answer is orb hi krishna krishna used to work with us oh really no congratulations krishna um will please send one email to phil opencv.org and we will make sure that you get your stuff uh go ahead and start the presentation again joseph yeah absolutely um cool let me re-screen so there's a couple other methods that we wanted to chat through um we could do this pretty quickly and probably jump right back into um q a so uh edge deployment uh a couple things that i want to kind of rapid fire mention about edge deployment and then we can move to mobile and web browser on edge deployment um getting high throughput that is getting high frames per second is often a key consideration and that's based on the model size as well as the hardware available to the model right so like electronics oak can often achieve about 30 frames per second uh when deployed um nvidia jetsons come in different levels of style or different um levels of compute there's nano xavier nx and in that order of of computation power and the frames per second is going to differ based on which of those you're using um and so that's an important consideration and there's optimizations to be made so there's a number of frameworks that are available to optimize how models run on given frameworks so for example for intel devices um you optimize models using the framework called openvino for nvidia devices you optimize uh perhaps using sensor rt uh for luxonous oak devices which are based on intel primitives uh you do open cv or openvino and then go to um the depth ai framework that luxonis maintains for deploying to oak devices and so there's these optimizations that can be made uh to ensure performance models but again like while more is always better the sufficient number that is required is usually best right so like you can always eke out more frames per second perhaps but it all comes back to the business logic of the problem you're tackling of what throughput and frame rate is required to tackle our question in mind um i'm going to jump quickly to mobile devices and i mean it's mostly a style of edge i called it out separately because there's different technologies like um you can use create ml for example which is kind of like a very lightweight uh low code way of creating a ml model to deploy to iphones we have a tutorial here that will include an email follow-ups of like doing that it's pretty fun brad goes out to the omaha aquarium he snaps a bunch of photos and then he prints it into create ml and he deploys it back on an app that works at a aquarium for fish identification an example of this might be like a mobile bank app where you take a photo of a check and the check shows up in your bank account that's doing machine learning i think at the edge i don't think it's sending that uh anywhere uh else um and then um the last one that i want to talk about is in browser so this is interesting i just want to add one more thing to a mobile so for uh mobile mobile uh people uh you know who are new to computer vision may not realize uh that um there are two reasons why it is i'm glad that joseph put it as a separate category first of all the computational power available on mobile devices is insane you know uh some of these uh are much more powerful than uh than so an iphone 13 pro max would probably be more powerful than a chromebook uh i i don't know so apple is now putting basically the same uh neural engine in there their high-end pro laptops that they're putting in the ipads and iphones yeah right so so that's one thing that you do have a lot of computation available to you and the second thing is that uh at least in ios there is a neural network uh pre-trained neural network that comes with the operating system right so you have the same neural network on i uh on the watch os on mac os on ios and it is possible to train uh this neural network just the last few layers of this neural network and create an object detector if the problem is you know not very complex not very custom you can create an object detector image segmentation etc using this pre-trained model so what are the benefits the thing is that your neural network while deployment could be a few kilobytes right instead of megabytes or sometimes gigabytes the reason is that uh you did not ship the entire uh neural network you shipped only the last few layers of the neural network the neural network is part of the operating system itself right so that benefit you get um you know it's it's inexpensive the size of the app that you're building could be very tiny it doesn't need to download uh a model so that's one more thing i want and it's and it's hella optimized as well for the for the metal yeah great callouts great callouts um yeah and then the last one to kind of chat about is web browser deployment so this is different than um api because what happens in this context is the model actually loads entirely client-side in a user's browser by that i mean on initial page load everything that the model needs to run gets transferred to the user's session inside the the browser that the page the user has loaded and then from then on there's actually no internet connectivity that's required for the model to continue to run and it makes use of the local hardware that that user has on their local machine so we've like done this with like for example mac m1 versus like older macs and you see difference in frame rate and performance um i thought it'd be kind of neat to show the same kind of face detection model in um in like a webcam type way so what's happening here is um the page is loading and it's a pretty big page to load because all of the model data needs to get sent to the edge uh or to the browser in this context and then now the model is running completely inside my browser uh and not making any responses elsewhere so it's kind of like an interesting hybrid of you know you get the privacy element here none of this data is getting sent back um you get the ability to build it into mobile apps and we've seen a lot of innovative work with this of like for example folks building always on ipad apps like point of sale checkout systems where the iphone app is always running in the browser and they get all the benefits of easy deploy to that edge but yet all the privacy benefits and speed benefits of you know this this kind of this kind of working and like we were talking earlier about privacy the reason i'm showing this model actually is this is actually a privacy protecting model because we have some users who are working on projects where they're not interested in faces in fact they're interested in in products and goods and to know what are what um data they don't want to store in their servers they have to find faces at the edge black it out or blur it out and then they can save those images right which is kind of like a funny oxymoronic thing you have to find the face to then black out the face to then send the image up to somewhere to do and so this is a model that's completely free to use to just get up and going and if you want to do like your own sorts of face blurring perhaps you're like hacking your nest camera and you don't want your friends faces in the data set or whatever it is um but that's kind of like one of the deployment methods that i think is is emergent and increasingly um interesting that we're seeing across roboflow use cases in fact uh you know in traditional computer vision uh there is open opencv.js which runs entirely on the browser uh and uh you know there are some models available there which you can uh do on the browser you know inside the browser you can use opencv it's not the full blown opencv but many functions are there exactly yeah we have roboflow js for the same save oh nice nice yeah um yeah so i know we're coming up against time this might be a good option to kind of turn to a few questions um with just the the final kind of tip slide live phil i'll let you lead us yeah sure so well we've got a couple decent questions here um joel henry would like to know does roboflow have is roboflow hipaa compliant is there a hippa compliant version of roboflow for storing data that's you know compliant with government regulations with regards to health information yes great question yes um cardinal health is one of our customers they're a fortune 50 healthcare company and um to support their use cases and their data sets we both had to ensure hipaa compliance and robust has the ability to execute baas business associates agreements that a number of healthcare partners uh rely on for trusting and ensuring safety of their data so joel feel free to shoot me an email direct for any deeper conversation i'm just joseph at roboflow.com cool yeah uh i mean i'm sure i know that kind of stuff is a big pain like it's it's it's uh tedious to get that kind of certification um so it's cool that uh y'all worked through that um cepha would like to know is there a way to easily store and annotate model inference results on roboflow yes uh that's a great let me just call it that's a great question to ask that's like exactly the question you want to be thinking about is i've got my model it's running can i then continue to improve um the answer is yes so robocle has a pip package um that users will incorporate into their applications and then they'll send images back so uh when the model is running at the edge for example let's say it's running on alexander spoke and you get a detection of phil's silly face that's too low confidence and we say oh you know this is an image that we need you then for what it's worth phil's silly face is very low confidence [Laughter] then uh then you get phil's silly face and then you use the real club pit package to send the image back to the um to your roboflow project where you can do re-annotation of those low confidence predictions so this is something that i think is a great question to ask and something that i think a lot of folks should be thinking about yeah um thanks for that uh partha had a question that i think will be interesting for both setia and joseph um you know we've talked a lot about a different hardware and there's so much different edge hardware especially these days like oak devices and the jetson there's you know there's even more even some of the open mv cameras i think dude inference and stuff now um how do you think about optimizing specific models for the specific hardware because they really do have their own deep optimizations that can be the difference between a model being performant and a model being really sluggish i'll let you speak first and then i'm happy to comment on it because i've spoken a lot yeah i mean for um we have never worked with openmd but uh so i don't know about let me interrupt do you mind unsharing your screen joseph okay yeah so um yeah we haven't uh i've personally not worked with openmv uh myself but with other platforms we use the tools that first of all when you're building for the edge you start with a model that is designed for the edge let's say a mobile version you know mobilenet or something like that instead of let's say inception v3 right so you start with a model which is already small and designed for uh for the edge and then you use the tools provided by the hardware manufacturer they are the best you cannot beat those tools so for example we use openvino for uh intel based processors as well as uh as well as for uh od related uh projects and then uh for nvidia it's tensor rt for uh google also provides uh tools like this right for example if you are if you want to use the google tool chain and uh some of the models and media pipe are great right so you can get things off the bat you don't even need to do the optimization it's already done for you so uh yeah so the short answer is that use the tools that uh are provided by the by the manufacturer of the hardware on which this is going to run and dig deep into it right open we know there is one way of optimizing but if you dig deeper you will see that there are you know there are several layers of optimization it's a whole thing in itself so if you dig deeper will find uh the one thanks yeah uh yosef you have anything to add to that i think all that is um exceptionally accurate and recognizing in um in addition that there are specific frameworks for the hardware you might be optimizing for like we mentioned earlier opendino for intel tensor rt for nvidia there's also model pruning techniques model quantization and to basically reduce the size of the model so it can run more quickly so these are some techniques yeah and we've we've seen we've talked about this before that you know even even running the most basic optimizations for say openvino if you're deploying to a myriad you know based product you get a 2x speed up sometimes for free um so yeah it's definitely and even for your cpu not just mediadex uh even on the cpu you can get that kind of performance boost yeah yeah um good point um joel has another interesting question at least uh i'd like to know the answer to this what sorts of data does roboflow support i mean is it really focused just on still images what about video what about text video for sure uh video and images uh we don't yet sport text we're pretty focused on uh understanding the visual world uh around us and there's plenty to tackle and unpacking that realm of things but all the common file formats you might expect jpeg png mp4 um these sorts of things dot tiff is actually browser dependent so if you're in safari for example tiff will work but um the um i'd be uh obviously feel free to shoot me a note if there's a file format you're looking for that you're not seen as supported webp what about webp i don't think so uh what about it what about hvec no um so those are those are things that will come up now right i mean people are not asking for those two special specific formats because of they are you know 30 smaller than jpeg for example for the same quality so yeah and the hardware encoding is finally good and relatively performant and stuff yeah um flavia would like to know about more about roboflow.js this is actually the first i've heard about it too would you uh in fact i'd like to learn a little bit more about both um opencv.js and roboflow.js um you might maybe go first to you uh yeah opencv.js i mean you go to opencv's repository and you will find that javascript you'll also find some example code there where you use this code in your browser and just put the you know javascript put the model at some specific location and you can try for example phase detection i remember there was also there are also feature detectors for example which you can run inside of the browser so there is a way to convert c plus code into performant javascript and we are using that it's uh escaping my mind what exactly it is called but we use that technique uh to do it and similarly there is tensorflow.js people should try out a lot of these uh work fine you know depending on the application obviously they are not as good as running on the cpu directly directly right browser is uh it limits the amount of processing uh that's allowed but still very very good for many applications yeah joseph yeah roboflow.js yeah robocle.js is for running optimized models directly in the browser um and similar to how opencv has functionality for uh running um opencv algorithms that's that's pretty similar so if you've trained the model um with roboflow then it becomes automatically optimized to run like a lot of those things we're mentioning around optimizing model size ensuring the frame rate as high as possible and so with roboflow js what you can do is basically pass any canvas element inside a browser to your model and then you can get inferences streamed live on frame by frame so you could pass like you know images or video or things like this and the other thing you can do with uh javascript and roboflow is closing the loop so you can add images back to your realflow project where your model is is failing um i'll drop a link to the docs there's both a video of um building with a thumbs up thumbs down detector what it does is when it detects a thumbs down it flips the camera it flips the video feed upside down so it's kind of a fun fun demo that that is cool yeah yeah uh let's see we're we're about getting up coming up on 10 minutes after the hour we want to do one more question and uh and we can we can call it a day here um can you uh joseph first of all um if people want to get in touch with you what are the best ways to do that happily um via email is is great um me and my team will happily help out joseph roboflow.com welcome to it's my real email it's direct um feel free to shoot me a note very accessible um and then also of course the um there's nothing quite like trying the product out and um trying things out at app.roboflow.com if there's any questions there i'm on twitter um i'll put my at joseph that's right you know it uh every well i'm i i've kind of you know i'm pavlovian response to like shake my fist to the screen whenever i see your your name pop up so it's easy to remember yeah so yeah last question here uh today is about a large scale image annotation ronald was asking about a large-scale deployment where they've got millions of images developed with tensorflow slash pytorch they're considering using spark to be able to scale it do you want to talk about other tools for scaling things with these massive data sets that's great that's great ronald i'd love to learn a bit more specifically because we're thinking a lot about this too of when you have this many images selecting down like the most useful images or some things that we're interested in i think for large-scale distributed systems i highly recommend the o'reilly book designing data intensive applications i i really uh enjoyed it i think for images specifically [Music] i think in data science line you know spark certainly is a good place to turn to i think for us what we're focused on is if a user is hosting images with us ensuring that things are performance for loading data directly into a session and then splitting images out so you can almost imagine like a sequel for images like interface where you can select and um kind of remove that abstraction for users but ultimately of course the the systems that you use to determine for scaling up your application for uh large scale systems depends upon a lot of other factors that um i'm sure that ronald you have a lot more intel on than i would be able to comment on right on yeah thanks um do you have anything to add this to you no i think uh uh joseph covered it and i'll have more to add to it in about a month uh you know uh we are working on uh a project which has a large uh data set and we'll be doing research for that so i'll have more to say on that but right now i think what joseph mentioned is uh uh pretty comfortable yeah no substitute for hands-on experience you know um yeah so i think we're about done here you want to take us home satia yeah oh man yeah you've lost your camera there um so i do a little song and dance here soft shoe little heel turn uh thanks again for joining us joseph um it's it's always a pleasure i i want to add one oh god i lost you we'll ask you again man let's let's just uh uh i i'll just want to add one more thing uh that uh and this is not this is not like plugging uh overflow but uh one thing that people need to be very aware of is when they're building a model right um do not forget the data so data annotation is not done once data is uh cleaning up the data data annotation etc it's an iterative uh process so we always have to so that's why tools like roboflow are very uh very useful because uh they combine you know they don't let you forget that oh data is a separate system then uh than your training process right a lot of times people think that oh i annotated the data once and that's it right and now i'm going to train my data never changes that's that's a very wrong approach and uh you know some things came up recently because of which i'm bringing this up uh but there is a very good um uh and i did not we we always come up with you know when we select a model uh we first select a model uh based on our application you know and the way to select it is that uh can you actually uh overfit the model on your training set if you can then the model has enough capacity to train uh on on your set and then we don't change the model actually it's not like we are going and changing model architectures and finding the best model architecture we freeze the model and work on the data right we uh try to make sure that the data is consistently labeled we try to make sure there is no noise in the data and this is especially true when you're working with small data sets right it is exceptionally uh and i did not know the word for this until recently uh you know this is our standard practice unfortunately dr andrew eng uh has a video uh called data it's data centric uh versus model centric approach uh you can find that on youtube it's a very useful uh thing to you know learn or understand a lot of people spend a lot of time trying to find the best model that will work for their application without worrying without actually doing anything about the data whereas you get a lot of gains by uh focusing on the data not on the model right and tools like roboflow you know because everything is in one package you you don't forget that the data is right there you go and fix the data if it's not uh performing well right so uh yeah something just you know from our personal from my personal experience something came up uh recently where i had to make this pitch very strongly to a client so yeah i just wanted to share it with the community as well