Zero to Hero with TensorFlow (TF Fall 2020 Updates)

Key Takeaways

hi everybody i'm laurence and today i'm going to talk about how you can go from knowing very little or nothing about machine learning to seeing how it works for computer vision scenario there's so much discussion about machine learning with so much information out there it can get very confusing so i want to boil it down to some of the essentials to show you how it all works and then maybe that will help you get started on the road to being an ml developer it's actually much easier than you might think so let's take a look at traditional software development for a moment but label some of the artifacts for example you've likely seen code like this where you read in some data you perform some manipulations on the data and then you return the result that's what the software industry was built on or maybe code like this in a game where you want to determine the behavior of the game how to move the ball what happens when it collides with a brick or the wall etc etc so traditional programming can be summarized in a graph like this you express rules using code these act on data and you get back answers think about almost every computer programming problem and how it could be summarized like this it's what we've been doing for our whole careers the machine learning revolution is just a new way of looking at this graph because it's a new and different methodology for coding instead of trying to figure out the rules ourselves and express them in code what if we gave the answers and the data and then we have a computer figure out what the rules are and that's the core of the entire machine learning revolution i know this might not make a lot of sense right now so let's explore an example consider activity recognition you want to write an app that uses sensors on a phone or a watch or something else to determine a person's activity you could for example use the data of their speed and then write a rule that determines if the speed is below a certain amount then they're walking you have data you have a rule you get an answer and then you could extend this to determine if they're running by a new rule if it's below four they're walking otherwise they're running well great it still works and you could extend this even further to see if they're biking if the speed is below 4 they're walking otherwise if it's below 12 they're running otherwise we can say that they're biking but then how would you handle golfing what role could you write to determine that they're playing golf also by now you've probably realized that the other rules are also that little bit naive you can't just go by speed you might run downhill faster than your bike uphill for example and of course different people go at different paces so let's go back to this diagram and think about how machine learning could solve that problem we can't figure out the rules for golfing and our rules for the others were already a little bit naive so maybe we can use ml what if we give it answers and data and then have a computer figure out what the rules are so for example it could look like this we could consider that all of the sensors on our devices and grab the data from them when we're doing the activity and that will give us data and we could then label that data based on the activity with something like walking running biking or golfing now our problem becomes can we match this data to this label like here for example maybe these parts are always present like this when the person's walking and these values are always different when we're running biking or golfing can we have a machine spot those patterns and when it does spot them can we then use whatever rules it figured out to determine which patterns match which labels and then have that look at future data to determine what activity we're performing and that's probably more reliable than if we wrote the rules ourselves so how do we write a program to do that pattern matching the process is actually quite simple we'll start by making a guess as to the relationship between the data and the labels then we'll look over all of our data and compare our guesses with the correct answers we'll have those answers already when we label the data then based on the parameters of our guess and the accuracy of what we got by measuring against the real answers we have some data that will let us optimize our guess and then we repeat the process logically if we keep doing this our guesses will get better and better when using the tools of machine learning like tensorflow the apis to handle this are available for you so now if we go back to this diagram the idea behind machine learning is to feed in the answers also called labels and the data and then have the machine figure out the rules that match those labels to that data and when you do this you'll get a model the model can then take data and apply the patterns that it learned to match that data to answers to give out a prediction or an inference about that data so in our case of activity detection now someone else could take a device and based on the pattern matching rules that the machine learned and put into the model it could detect if you're walking biking running or even golfing so let's put this idea of pattern matching into practice and we'll explore how it can be used in computer vision where we use the concept of machine learning to help a computer understand the contents of an image when you give an image to a computer it's just a bunch of pixels it doesn't really know anything about its contents so to simulate that here's an image i've blurred it so you probably don't know what it is with images you can apply filters to extract information now you've probably seen tools like photoshop or instagram where you can sharpen images or detect edges and stuff like that those are just filters which are just sets of numbers so what would happen for example if i could apply a filter to this image that does this and detects these things now to you and me they look like human legs and then maybe another filter detects these things and again to you and me they look like human hands but the computer has no idea it just knows that it has detected something and then maybe another filter detects this and again to us it looks like a human face so by applying these filters i can get legs and hands and a face and if i had already labeled the image that works with these filters as a human i can have a computer learn that when it detects these three features using filters then what it sees in the image is a human and similarly if other filters detect things that look like these then when the computer applies those filters to an image and all three of them produce something it's labeled as a horse so over time in the same way as with activity recognition from earlier if the computer can figure out which filters extract the features that determine the difference between the answer of a human and the answer of a horse then we begin to get a set of rules that determine the difference between a horse and human and then in the future if we pass it a new image depending on which filters work with it we can predict what's in the image these filters are randomly initialized but over time using the loop that we showed earlier their values can be measured for accuracy and then optimized and so on the values that match data to labels can be learned over time so when we pass data into the model we can get inferences out and we have a model which contains the learned filters and in the future if we pass it an image it can detect the features that for example determine if the picture contains a horse or a human and in this case it's clearly a human so remember this loop we make a guess we measure the accuracy of that guess and we then optimize for the next guess repeating this across all of our data a number of times in the case of computer vision our guess is to randomly initialize the values of the filters apply them to all images and then try to determine which ones work to extract features that get matched to the labels then we can compare the results of our guesses and their matching with the actual labels to see just how good or how bad our guesses were and then we use the data from this to tweak our set of filters or any other parameters to improve the accuracy and we repeat so for example if my filters look like this and i'm extracting features and i can reliably match these to a label such as human then in the future when using these filters i can confidently predict that the image contains a human and in fact it does like we can see here so what does this look like in code in tensorflow we can define a model as a number of layers where each layer is a piece of functionality and it defines what we want the machine to learn conv 2d stands for convolution 2d which is another word for filter and because this is quite a sophisticated scenario with detailed images applying filters to the image probably isn't enough we have to apply filters to the image and then filter those results and then filter those results again which is why there are three layers of filters in between these layers are things called max pooling which while they aren't necessary they are really useful max pooling is a technique that compresses the image while maintaining the results of the filter this means the next filters will have less pixels to work with but they'll still have the features and this can speed things up when you're processing lots of images after you have all of the filters learned then the idea is to match the filters to the labels and a dense neural network does that really effectively so the results of all the filtered images things that i demonstrated like legs hands and head will be matched to the labels so that those extracted features will be matched to the relevant label so human looking legs get matched to human horse looking legs get matched to horse and so on the last thing is the output of the model and it's defined here and this is also a neuron as we only have two types of class here horse or human we can actually do it with a single neuron where if the value is closer to zero it's likely to be a horse or if the value is closer to 1 it's likely to be a human once the model is defined then we have to define how it works in training so if we remember this diagram make a guess measure accuracy optimize it's done in this code first of all when we define the model all of the parameters the filters the values of the neurons all of that stuff was randomly initialized and that gives us our first guess in other words as we only have two classes horses and humans are randomly initialized model should be expected to be 50 accurate or thereabouts so we can measure that accuracy using something called a loss function now there's tons of built-in loss functions in tensorflow so it becomes a matter of choosing the right one for your model in your scenario i chose one here called binary cross entropy because i only have two classes binary two the next step is to then optimize your guess and in this case we're optimizing using a technique called root mean square propagation or rms prop and again you have a number of different options it depends on your scenario to choose the up the correct optimizer for you the optimizer will take the data that it has the random initialization of the parameters in the network the accuracy and all that and then update all of the values with a new guess and we then repeat we started with close to 50 percent but now because of what we've gathered the next iteration for example might go up to 60 percent remember this is what gives machine learning its name by going through this process it's learning the parameters that best match the data to the labels the next piece of code is called model.fit and it has that name because it's fitting the answers to the labels in order to figure out the rules and the epochs parameter this defines how many times you'll repeat this process as the model gets better and better at matching the data to the labels the process is used it takes time to figure out how many epochs what model architecture what parameters etc will lead to convergence and a skilled machine learning developers should be able to reach convergence quite quickly so let's now give a demo of training a computer to recognize horses or humans we'll do this in tensorflow okay so let's take a look at the horse versus human in action i have a collab here and i'm going to start just by making sure tensorflow is running within it it's connecting to the vm and it works now i'm going to download the data i've stored the data in google cloud storage and it takes just a few seconds to download it first is the horse a human and then is the validation data the data is zip files so i'm just going to create local directories that i can unzip them into and i'm good to go i'm going to import tensorflow and now i'm going to create my model and my model is a convolutional neural network as you can see here a pretty straightforward and simple one i'll just define that there are multiple layers of convolutions and as you can see there are four of them now i'm going to compile the model i'll just compile it using root mean square prop as the optimizer and binary cross entropy because there's only two classes as the loss function this code will import the data using an image data generator and it will do the image augmentation that we discussed setting things like the rotation range the width shift range and so on and now here i'm going to fit the model i'm just going to train it for 15 epoch so it's pretty short and i'll speed it up and now if we scroll down we'll see that it's done the 15 epochs have run and it's got about 86 percent accuracy on the training and about 88 accuracy on the validation set so it doesn't look like it's overfitting and we can run the model just to give it a try so if i run this code it will give me a box or i can choose my files i have some files that i downloaded this one called human what you can guess is probably a human and it saves it and it checks it out and it sees yes indeed that was a human and if i just try one more and i choose my files and i pick one of my validation images which is a horse and we can see horse here and it actually correctly classifies as a horse so that's it that's the horse a human model and that we showed here and just shows how quickly and easily you can get up with computer vision and indeed this model is actually quite a difficult one because training the differences between horses and humans is it can be a very difficult problem to do and tensorflow was able to solve it in this case thank you very much and if you have any questions you can reach me on twitter at el moroney [Music] you