How to Use CreateML and Build a Computer Vision iPhone App | AR Object Detection

Key Takeaways

[Music] that's what your iphone might say after today's video because today we're going to train a mobile object detector to find sharks and fish and images at the end of the video you'll have a model that looks a little something like this one of the most frequent questions i receive is what sort of prior knowledge and experience do i need in order to be able to build a computer vision powered application do i really need a phd in machine learning well the answer to that question is emphatically no you don't need an advanced degree and in fact sometimes you won't even need to learn how to code in order to get started with computer vision and that's what this video is all about we're going to take you from nothing to a trained and deployed computer vision powered mobile application all using free and open source tools and without writing a single line of code and while we'll be training our iphone to identify fish at the aquarium you can use these exact same steps to solve any number of problems in fact we've taught thousands of students hobbyists and software developers to use computer vision to train a model for the first time to do everything from protecting endangered species to preventing wildfires and that's just the beginning the only limit is your imagination so if this sounds exciting and interesting you're in the right spot because we're about to level you up [Music] all right so we know where we're heading but how are we going to get there well we're going to use something called a neural network and whoa whoa before you change that youtube channel don't worry this isn't one of those youtube videos where we all of a sudden jump into a whole bunch of complicated math and computer science gobbledygook if you're interested in how a neural network works this is not the video for you there's plenty of other ones i'm sure you can find on youtube for us we're just going to use a neural network as a tool and so all we need to know is what does it do and how can we use it so to answer the question of what does a neural network do we can imagine it kind of like a child learning from flashcards so if we show a child a flashcard it's gonna see the question on it and it's gonna guess a response if it's wrong we say no that's not right the correct answer is actually this and over time after going through those cards many times the children learn what you're trying to teach them so neural networks kind of like that we show it a whole bunch of questions and it learns to predict the answers so for us we're going to show it questions which are images and answers which are the locations of fish in those images and over a period of time our neural network is going to learn to match an image and the location of the fish and so with any luck it will learn to do that not just by memorizing the exact images that we give it but also to generalize to any image of fish that we give it so that it can actually be useful for us um so we'll learn a little bit more about that in a bit but the first thing that we need to do is go collect a whole bunch of images of fish that we are going to use as our examples our flash cards for the neural network and to do that we're going to have to go to the aquarium unfortunately the nearest aquarium is a little bit of a drive so buckle up because we're about to go on a road trip [Music] hey guys what's up i'm brad one of the co-founders of roboflow we're a startup that makes computer vision accessible to everyone not just machine learning experts and we do that in a couple ways one way is content in videos like this but we also make software and tools so that you can focus on the problem you're trying to solve not on reinventing the wheel that's where i usually come in i'm usually behind the keyboard not the camera i'm selecting code not videos but i finally convinced joseph and jacob to let me shoot a youtube video on one condition they said i'd only get to shoot another video if we got more than 100 subscribers from this one so help me out hit that subscribe button ding that bell and i hope you enjoy the show all right we made it to the aquarium so now we're gonna go inside and we're gonna take a whole bunch of pictures remember we're creating our answer key for our model so we want to make sure to take a wide range of pictures of everything that we want our model to learn to identify [Music] so all right i think that does it we've got a whole bunch of photos of a wide range of critters let's go home and see what we've got [Music] those pictures turned out great we have the beginnings of a really cool project on our hands the next step is taking those raw images and turning them into something that our neural network can learn from the flashcards if you will this process is called annotation and it consists of drawing boxes around all the things that we are going to want our model to recognize this means the fish the sharks the starfish the stingrays and any other critter that we find that we want our model to know about there's a whole bunch of tools that we could use to do this but we're going to use roboflow because it fits really seamlessly into the rest of the process as we'll see later so to get started i just go to roboflow.com and click try it now i already have an account but if you don't you can create one with your email or github account then i'll click create a data set and i'm going to call this aquarium and the things that i'm labeling are the animals [Music] then i'll take all my images that i took and drop them into the interface this will process them and then it'll let me upload them to my account so i can label them [Music] now that our images are finished uploading i just need to go through and put boxes around all the things that i want our model to recognize so remember we're going to label all the fish jellyfish penguins sharks starfish and stingrays if you don't want to go out and collect and label your own images just yet you can go to public.roboflow.com to download a pre-labeled dataset that's ready to go so you can follow along with the rest of the tutorial now that we've labeled all of our images we have what's called a data set this is analogous to the flashcards from earlier it's the set of all images and annotations that our neural network is going to use to learn how to identify fish we'll use roboflow's dataset healthcheck to get a closer look you access the dataset healthcheck on the left-hand side of the dataset page it gives you all the high-level information about your images and annotations that you'll need to make your pre-processing decisions so here we have 441 total images and they've all been labeled but we also see that one of the images is missing subjects so this means that we have a photo that didn't have any fish or puffins in it this is fine having a few null examples is actually good it teaches your model that there's not always going to be something that it's looking for in an image but if you have too high of a percentage of null examples your model is not going to learn very well and you might want to filter some of them out we also see that we have 3062 total annotations and 6.9 per image this is probably skewed upwards by some of the photos of schools of fish that we have we also see that we have 10 classes so 10 different types of animals that's a little bit weird because that's more than we intended to label and so we'll see in a second if there's something that we have to correct the other thing we can notice from the top of this page is that our images are absolutely huge because we took our training data from an iphone at full resolution each picture is 12 megapixels and that's going to be way bigger than any model is going to be able to deal with and so we'll want to downsize those later to speed up time at training and also to decrease our file sizes next we'll look at our class balance which is the distribution of our different types of objects amongst our images and we can see a few different things from this chart first we note that we have way more examples of fish than we have examples of anything else sometimes this can be a problem you generally want your classes to be relatively evenly distributed and that's because you don't want your model to learn that it should just guess fish because most of the time the thing it's looking at is a fish you want to force it to also learn things about all of your other classes there's a few different ways to address this the best way would be to go back to the aquarium to get more examples of our underrepresented classes there's another way called over sampling which is using your existing images to create more training examples of the underrepresented classes this is a strategy that we'll cover in a future video we also have three severely underrepresented classes the humans the banner fish and the turtles and it looks like two different things happen there number one we split this data sets labeling task amongst the different members of our team and it looks like somebody didn't get the memo that we weren't looking for turtles that wasn't amongst the initial list of objects that we wanted to label so we'll filter those out also it looks like somebody was a bit more specific on labeling a certain type of fish we actually just wanted to label fish in general not the different species and so we'll rename all the banner fish labels to fish automatically in just a moment finally it looks like we didn't have enough examples of humans to come up with something that our model is going to be able to learn from and we actually didn't care too much about finding the humans in the first place so we'll drop those from our data set as well next we'll look at our dimension insights and this tells us what the different sizes and aspect ratios of our photos are so we can see we only have two different sizes of images this is because we only took photos from one type of camera if we had taken photos from a whole range of cameras or collected them from a bunch of different people we might have a whole scattering of different images for our purposes it looks like the most important thing is that 436 of our images were taken in portrait mode but only five in landscape mode and that might lead us to have to restrict our app to only operating in portrait mode because our model isn't going to learn too much about what landscape mode photos look like next we'll look at our annotation heat map which is a visualization of the position of our object relative to the frame of the photo and as you can probably expect most of our objects are congregated right here around the center if you look at the fish the starfish the penguins they're all kind of in that same spot around the center it makes sense because when you take a photo of an object you're going to try and center it in your photo frame remember this distribution because in a second we're going to use this as a trick to improve our model's deployed performance now that we know what we're dealing with we'll go back into the data set and set up our image processing pipeline we'll apply pre-processing and augmentation steps these are image processing techniques that are going to transform our images so that our model trains faster and performs better so the first thing we're going to do is address those problems with class balance that we identified so we're going to rename the banner fish to fish we'll get rid of the humans and we'll also get rid of the turtles next we'll deal with downsizing those huge 12 megapixel images to something more manageable by our model you'll want to do some research to see what input size your model takes but i know that the one we're going to use uses 416 by 416 so i'll choose that i'll also apply contrast stretching which is going to let my model use more of the color space it makes the lightest pixel white and the darkest pixel black and finally we're going to apply what we learned from the heat map namely that our objects appear primarily in the center of the images to crop out the corners and give our model a much higher proportion of content-rich pixels next we'll add some augmentations this is the process of taking our existing training data and applying transformations to create new training data it means that we'll get a bigger data set without having to go out and take and label more images having a bigger data set is important because it makes it less likely that our model will be able to memorize the exact images that we feed in and it'll make it perform better when we show it completely new images out in the wild so we'll go through these settings and apply the ones that seem to make sense to our images so for example it doesn't matter if our fish are facing left or our fish are facing right we might have examples of both of those so we're going to apply the horizontal flip augmentation but not the vertical flip augmentation because normally fish don't swim upside down now we're going to go through the other augmentation options and add the ones that simulate things that our users might be doing in the wild so we'll add crop in case they're a little bit closer or a little bit farther away from the fish or the fish are a bit off-center we'll add rotation in case the fish are swimming a little bit up or a little bit down or the person is holding the camera a little bit off axis similarly we'll add shear to do the same thing as rotation but on the other two axes of rotation so the fish might be swimming a little bit towards us or a little bit away from us and then we'll change the hue a little bit in case you know the fish are a slightly different color or the coral is a slightly different color we don't want our model to learn the exact color profile of the exact fish that we took pictures of same thing with saturation in case they're a little bit more vibrant or a little bit more dull this could also be affected by the lighting conditions brightness and exposure also could be changed based on the lighting condition of the particular aquarium and the time of day so we'll adjust both of those and then we'll apply a little bit of a blur to some of the photos in case the fish is a little bit out of focus we don't want our model to be honing in specifically on those crisp lines and i think that looks like mostly it we have some of these other advanced augmentations that we're not going to apply for this simple model but that you might want to check out and we have some other content that you should look into for specifically mosaic cutout and bounding box augmentations and your problem might be a little bit different and 90 degree rotation might make sense for instance if you're looking at drone photos from above it doesn't matter whether it's upside down or left or right pointed and you might want to experiment with grayscale and noise but we're not going to add those to this data set for today now that we've picked all of our augmentation options we'll decide how many versions of each training image to create so if i select three here what that means is it's going to take each of my 308 training images and for each one it's going to create three new versions of it by applying a random flip random crop random rotation random all of the things that i've applied combine those together to create a unique output the more examples that i create the better my model is going to generalize to other images that it's never seen in general but also the longer it's going to take to train you might set this based on a number of different things you might observe that your model is not generalizing very well so you could boost this and train it again and see if that improves performance or you might not have enough training images and it's way easier to generate augmentations than it is to go out and collect more images so we're going to try and do 10 examples so that means it's going to take each of our 308 training examples apply 10 different augmentations to them and we'll end up with a data set that at the end has 4 400 different images compared to the 440 that we had originally now that we have our settings just the way we want them we'll go ahead and generate a version this creates a snapshot and time of our data set and this does two things for us first it speeds up the training process because it precomputes all the different outputs of the images with the transformations that we have applied but second and more importantly it gives us reproducibility oftentimes when you're working on projects you'll want to try a whole bunch of different models but you don't want to change a bunch of variables at once and by tying the image processing pipeline to a single point in time snapshot it means that you can compare just the changes that you're making in the models and not have to worry about whether the images were processed differently or whether the random values just happen to make one model perform better than the other to create a version we just click this button and give it a name i'm going to call mine augmented and this will spin up a whole bunch of servers in the background to process all the images this will just take a few seconds while we're waiting it would be a good time to subscribe to the channel [Music] okay our images are finished generating now for the spot where roboflow really shines we're going to download our data set in the exact format that we'll need for training traditionally this is the spot where we'd have to open up the text editor and start writing python but this is a no code training tutorial after all the format we want to use is called create ml so i'll select that and i'll download that file to my computer next we're going to take our data set and feed it into a tool called createml which is a no code training platform created by apple that takes in a data set and spits out a model that's specifically optimized to run really fast on your iphone's processor createml is a part of xcode which is a free download from the app store so if you don't have it already go to the app store and download xcode xcode comes with createml so once you've installed xcode open createml and create a new project we're going to create an object detector so select that and give your project a name you can save it inside of the folder that we just downloaded that has all of our images and annotations now we just add our training validation and test data from roboflow and we should be good to go we'll be training a yolo v2 model which is the full network you might also experiment with transfer learning if you have the new version of mac os once i'm ready i'll hit train and that kicks off the training process this will slow down your computer for the duration of training and it could take quite some time so i'm going to leave this running overnight [Music] okay it's the equivalent of christmas morning for the nerds it's time to check in on how our model did we left it running overnight and it looks like it's finished training so now we're gonna see the results after about six hours of training my model went through all of my images thirteen thousand times that's a lot of flashcards as we can see the loss value or the measure of the amount of error that it was getting decreased over time as we'd expect it ended up at 1.75 but it's a little bit hard to understand whether that's good or bad without seeing some examples on the preview tab i can see how my model makes predictions so i'll load up all of the images from my test set to see how it did importantly these are images that my model has never seen before so here we can see that it found a whole bunch of penguins that's great looks like it found this starfish with 100 confidence got those jellyfish pretty well as well so subjectively speaking i feel like this model did pretty well we can also go in here to evaluation and see how it did objectively speaking this measures the intersection over union which is the percentage of each bounding box that it predicted correctly so it looks like it predicted the jellyfish extremely well some of the other classes like fish and starfish it did medium on but the penguins and puffins it did pretty poorly on if i recall correctly the puffins were a bit of a weird situation where they had a whole bunch of water splatter on the glass and i expect that that is why it didn't do very well on those i'm not sure about the penguins it might be that there are so many of them in the background if i was going to go back and try and improve this model i'd probably go collect more photos of the penguins and puffins to try and improve the performance on these classes right we're almost to the finish line so far we've collected images we've labeled them we've processed them into a data set and we've used that data set to train a custom model now all that's left is wrapping that model in an app so we can deploy it to our iphone apple's already created a sample app for us so if we just search for apple breakfast finder sample app we'll be able to find this sample page that lets us download an xcode project that's already created for us so we just hit the download button and then we'll open this project in xcode once you open xcode you'll have to do two things for setup first plug in your iphone and select it as the device that we want to deploy to and second go to the signing and capabilities and make sure to set your team here if you haven't signed up for an apple developer account yet you'll have to go to developer.app to create a team and then the first time you hit this play button it'll make you authorize your device let's see how the model included with the sample app does it's trained on breakfast items so i pulled out some bananas a bagel and an egg it looks like it's all right it's not perfect but it's passable so i've got a little bit of a confession to make i told a little bit of a white lie earlier in this video i said we wouldn't have to write a single line of code to get things working but technically we're going to have to change one or two configuration lines in our xcode file so that our iphone will know to look at our new model instead of the example model don't worry we'll walk you through it and you don't need to know anything about how to code for an iphone you can just follow these simple steps and your model will be working fine all i need to do is go to the very top left and click the file folder icon to view all the files in this project we can see that there's an object detector.ml model file here that's been trained to identify breakfast items what we want to do is replace this model with our model that knows how to find fish so if i open back up create ml and go to the output tab i can drag our aquarium1.ml model file here and import it with the default settings it should show up here on your file browser and if you click it you'll see that this model knows how to identify the things that we're looking for now all we need to do is update the code to reference this aquarium1.ml model instead of the object detector.ml model and that's in this file called vision object recognition view controller so if i go in there and look at line 24 i can see that they've referenced object detector here and we just need to change that to aquarium one i'll save that file and then hit play and it'll build this for my iphone and now the app is looking for the things in our model the fish success we've now successfully trained our model on a custom data set and deployed it to the iphone now all that's left is testing to see how it does [Music] wow that's looking pretty darn good let's compare those results with a model that i trained with in roboflow that's a bit more powerful you can see that the bounding boxes are a bit more reliable and stable on the roboflow train model especially when the fish are really overlapping but they both did a pretty good job but what happens if we take the app to a different aquarium will it still do just as well yikes no it looks like the model's not doing very well at all turns out this aquarium has different species of fish let's see what happens if we retrain the model with some new training data [Music] much better this just goes to reinforce how important it is to get representative training data that is the flash cards that you train your model with should be as close as possible to the images that it's going to see in the wild well that wasn't so hard was it we just went through the entire process of creating a custom computer vision project in just a few minutes and as promised it didn't even require a phd we went out in the field and we collected images then we took those raw images we labeled them pre-processed them and augmented them to create a data set and then we used create ml to train a custom model on that data set and deployed it to the iphone the next part's a bit more tricky you've got to use your human creativity to figure out what project you're going to build next you can use these exact same steps to do any one of thousands of different things the only limit is your imagination and we're really excited to see what you come up with be sure to share any cool projects you build in the comments and if you like this video and want to see more like it please subscribe and don't forget to ding the bell so you get updates when new videos come out hey there it's from bro