Machine Learning Foundations: Ep #3 - Convolutions and pooling

Hi, and welcome back to machine learning foundations for Google developers. I'm Laurence Moroney and I'll be your host and guide. In the last episode, you took your first steps into computer vision and you built a neural network that could recognize items of clothing. You then did an exercise with a similarly sized data set, but with different content, and that was handwriting recognition. So, before we get on to the new stuff, let's take a look at the answer to the exercise that you did. So, let's take a look at the answer to exercise two. The code is here. One of the things I've done is to implement a callback. I've done that by creating a class called my callback. And in this class, I defined on epoch end, and this takes self, epoch, and logs. From the logs, I can get the accuracy. And if the accuracy, for example, is greater than 0.99, I can say print we've reached 99% accuracy, so the training will be canceled. So, now for MNIST, I can just say MNIST is tf.keras.datasets.mnist. I can load and normalize my data as before. I can create an instance of my callback that I'm just calling callbacks. And then when I'm training, I can add this extra parameter called callbacks and say callbacks equals this instance of my callback. So, if I were to run this, I'll get this warning, but let's see it running. And here you see after five epochs, the accuracy was 0.9917, which is greater than 99%, so it canceled training after only five of the 10 epochs, and it was able to hit that 99% accuracy. So, that's the answer to exercise two. Hope you were able to get it. Okay, now that you've looked at both fashion and handwriting examples for computer vision, you've probably noticed a big limitation for computer vision with these types of example. And that was that you could only have one item in the picture, and that it had to be centered, and it had to be well-defined. So, shoes had to face left, for example. A neural network type called a convolutional neural network can help here. It'll take us a little time to build up to it, and in this video, we'll talk all about what convolutions are, and how they can be used in combination with something called pooling to help a computer understand the contents of an image. It's going to take a little while to put it all together, but let's start with just understanding what a convolution is before you can start using one. The idea behind a convolution is very similar to image processing with filters. If you've used something like Photoshop before, it has filters to do things like sharpening an image or adding motion blur. So, while the effect is complex, the process behind it is quite simple. So, let's take a look at it in action. On the left here, I have an image of a boot from the Fashion-MNIST data set. On the right, I have a representation of nine of the pixels in the image. I'm going to call the center one my current pixel, and all of the others are its neighbors. Here, I define a filter, which is a set of values in the same shape as my pixel and its neighbors. So, for example, if I had one neighbor in each direction for a 3x3 grid of pixels, I'd have a 3x3 grid in my filter, too. Each value in the filter can be called a weight. So, to calculate a new value for my pixel, all I have to do is multiply each neighbor by their weight, my current pixel by its weight, and then add them all up. And I'm going to do this process for every pixel in the image. The result will be a transformed image. So, for example, if you look at the picture on the left here, if I apply the filter in the middle to it, I'll get the picture on the right. This filter has led to a huge emphasis on vertical lines. They really pop now. And similarly, this filter leads to an emphasis on horizontal lines. So, you might be wondering at this point, what does this have to do with computer vision? Ultimately, the goal of trying to understand what an item is isn't just matching the raw pixels to labels like we did with the fashion and handwriting examples. But what if we could extract features from the image instead? And then when an image had this set of features, it was this class. Or if it had that set of features, it was that class. And that's the heart of what convolutional neural networks do. They process the images down into raw features, and then they find sets of features that will match the label. They do this using filters. And just like neurons learned weights and biases to add up to what we needed, convolutions will learn the appropriate filters through an initial randomization, and then using the loss function and optimizer to tweak them for better results. So, for example, take a look at these images and you can see how they've been transformed by filters. And these filters learned how to isolate and abstract features within them. You can see, for example, that something like our horizontal line detector was able to detect a commonality in these images. They're all shoes, and that line is the sole of the shoe. But if you consider these images with the same filter, they had different results. So, if a lot of pixels are lighting up with the horizontal line filter, we could kind of sort of describe that as a sole detector. Now, you might hear that phrase a lot, detector. But what it's referring to is ultimately a filter that can extract a feature that can be used to determine a class. I first heard it in the cats versus dogs classifier, which we'll look at in a later video, when somebody visualized a floppy ear detector, which determined something was a dog and not a cat. One other thing that you might have noticed in the previous slides was also that the resolution of the images was decreasing. This is achieved through something called a pooling. The idea here is pretty simple. If there's a way that we can extract the feature while removing extraneous information, we can actually learn much faster. Now, let's take a look at how this works. The concept is actually quite simple. Imagine this is a chunk of our pixels. To keep it simple, I'm just going to have them as monochrome. We then look at them in blocks of 2 by 2 like this, and then we'll only keep the biggest value, which in this case is 192. We then repeat the process for the next 2 by 2 and keep the biggest, which is 144. And the next, which yields 255. And the next, which yields 168. We then put these four results together to yield a new 2 by 2 block, and this contains what had been the largest values in the 2 by 2 constituents of the previous 4 by 4 block. We've now reduced the size by 75% and maybe, just maybe, we've kept the important information. So, let's see what this actually looks like. And here is the image that we had filtered earlier to detect vertical lines. On the left is what it looks like before pooling, and on the right is what it looks like after pooling in the way that we just demonstrated. Notice how the information hasn't just been maintained, you could argue that it's also been emphasized. Pooling is an important way also of reducing the amount of information your model has to process. If you think about it, say you want to learn 100 filters, that means you would have to keep track of 101 versions of your image, the original plus the results of what it would look like after the filters have been applied. If you have thousands of images, you'll soon start to eat memory. And that's just one layer. What if you had another layer beneath this that also learned 100 filters? That would mean each of your first 100 will also have 100 products from it, giving you 10,000 images for every image you need to train. Any technique that can reduce size while keeping this information is obviously very valuable. Okay, so now that we've seen what a convolution is and how it works as well as how it can go hand-in-hand with pooling, let's take some time to see how to code for them. Here's the URL of a Colab containing codes that can filter an image as well as do pooling. You can play with the filter parameters or try different effects or just use the ones that I've provided. Pause the video now and give it a try. So, let's take a look at some convolutions in action and here's the lab that you'll be doing yourself in a few moments. You might remember some of the images from Fashion MNIST, but I want to talk about for example if you were going to classify a shoe instead of the one that you had in Fashion MNIST. One of the ways of doing this is with convolutions and in this lab you'll just take a quick experiment and just see how the filters of convolutions work. So, we'll start by importing some of the required Python libraries. And one of the things that's built into these libraries is this image called ascent. So, we can use the pyplot library to draw it so we can see what it looks like. And it's a person walking up some stairs. I'm now going to just copy that into a numpy array so that I can manipulate it. And here I've defined a number of different filters. Remember the 3 by 3 filters that we were showing? I'm just implementing them as three arrays of three items. So, for example, we use this one filter here. And one little thing that I've done is added a weight to this so that one of the things is that all the digits should add up to one, but if they do add up to something more than that, then you can multiply them out by a factor to normalize them. So, for example, if they added up to 10, you could set the weight to 0.1 so that final result would be normalized back to one. Now, here's just simply a loop going over the image and multiplying out the relevant pixel and its neighbors by the relevant item in the filter. Once it's done that, we can plot it to see the result. And here you can see the filter that I'm using is really emphasizing the image's vertical lines. For example, I could go and change to a different filter like this one. Run this to set the filter and then run the code again. And now plot the image again. We'll see the one that really emphasizes the horizontal lines. So, you can consider following filter values and look at what their impact on the image is or you can experiment with your own. As for pooling, we saw in the lesson that pooling is taking groups of pixels and just picking the biggest value. So, if we were to do a 2 by 2 pool, we'll take 2 by 2 blocks, pick the biggest value. So, a 4 by 4 block like this one would become a 2 by 2 block. We'd lose 3/4 of the information, but hopefully maintain what's in the image. So, for example, here I've implemented a simple pooling. So, if I'm to run this on the image that we've just seen, it will get pulled to this. If you look at the axis, the axis is now it's 256 by 256. And if we look at our original image, it was 512 by 512. This is the original image after the filter. This is the pulled image after the filter. We haven't really lost any information. So again, have a play with it and try it for yourself. Great. So, you've now had a look at how filters can convolve images, providing the basis of convolutional layers, as well as how pooling can reduce the image size while maintaining the features of an image. There's no assignment this time. Yay! So, in the next video, you'll start implementing convolutions and pooling in code and you'll see how these can improve the Fashion MNIST classifier. After that, you'll take on some more real-world and more challenging images and you can learn how convolutions can help you to classify them. So, don't forget to hit that subscribe button for more great content. Thank you. Mhm.