Luxonis OAK-D: Computer Vision on Device

Key Takeaways

today we're discussing the luxonis oak d i mean this thing is kind of like a raspberry pi on steroids built for computer vision it's got a 4k camera it's got the ability to do real time object detection it smashed records on kickstarter raising over 1.2 million i mean what makes this device so special yeah we are really excited about this device over here at rebel flow and the reasons for that are numerous first of all it's got a very high resolution camera it has the ability to triangulate depth which we'll go into a little bit more here in a bit and then it also has the ability to do real-time inference by parsing neural network tensors very fast on the back here with the movidius vpu and the movidius vpu is essentially a replacement for a gpu so this is what the computer is going to be using as it's taking images in transforming them into tensors running that through a model very quickly and then spitting inference back out for you at the end and so the other thing you know that i was touching on is the depth component which i think is uh particularly game changing and uh joseph tell us a little bit about how how that works sure yeah so we caught up with brandon from lexanus in it in a prior video don't forget to like and subscribe and he describes how the luxonos team identified if you're able to triangulate the distance of an object not only can you do object detection to identify that for example you have a car on the horizon but if you know the constant distance between one camera and the other two cameras on the side then using the depth ai software platform you can infer and identify measurements and so i mean spatial ai unlocks new capabilities for example if you're a commercial fishery you can only catch and keep fish that are of a specific length and if you want to be able to do say automated identification of both your count of your catch and ensure that you have a sufficient batch that you're able to keep something like this would basically handle both those tasks at once now i mean to be clear not every task requires a sense of depth and object detection and 2d object detection still unlocks massive capabilities as we'll see today but i do think that this unlocks uh new things that i can't wait to see when it ships to everyone in december 2020 but jacob i understand that you've uh got something set up for us to try out today you mind introducing what it is we'll be doing yeah so today we have a tutorial ready for you basically on how to use this technology and a demonstration of what it can do and so the task we've chosen to tackle is actually the identification of the alphabet in american sign language and we're able to do this thanks to a data set provided by roboflow user david lee um but let's go ahead and dive right in and see if we can get the computer to identify uh different letters in in sign language so what do you say should try it out i'm excited let's dive in awesome the first step is to gather a data set here you can see we have american sign language letters hosted publicly on roboflow courtesy of roboflow user david lee let's take a look at a couple of these images the next step is to train a model with our new data we'll go ahead and go through this notebook and form a custom trained model to identify american sign language using state of the art object detection technology while we're working through the notebook we'll also check to make sure that our model can make inference on test images once we're satisfied with our model we'll go ahead and export it to a representation that is rentable on depth ai now we're going to go live here i have alexanus oak d it's plugged into my computer where i have the custom weights loaded in and we're going to go ahead and kick it off so here we go it's just a single python command to kick off our custom model and you can see here that a video pane pops up where the device is actually doing real-time inference so now let's go ahead and test it out to see if it can identify letters like o or v or maybe it can even teach me some sign language that was awesome great work thanks yeah it was pretty exciting to be able to build this so quickly and um it's amazing the state of the technology is in today and i just kind of wonder though like how could we how could we make it better collect more data right i mean if you get more data in your inference condition your model will only improve so data of doing hand signs with different backgrounds and all these different things um i would always keep track of just like you showed in roboflow how many letters you have of each class and then yeah continue to to train and deploy to luxonis um can you go into a bit more depth about how i can do this too at home or anyone else yeah certainly so everything we've done here today is publicly open source to you uh via blog post below we have all the code and all the instructions to be able to do the same thing with your own custom task so we look forward to seeing all the different applications that you bring forward uh both leveraging roboflow depth ai and the luxonis devices so don't forget to like and subscribe and thanks so much for joining us today happy training