Introducing Convolutional Neural Networks | Convolutional Neural Networks with TensorFlow

Key Takeaways

Hi everyone, my name is Nathan Pacini. I am marketing manager from data science dojo. Uh and today I have Neil Lizer here with me and he is going to be giving um a presentation on an introduction to convolutional neural networks with TensorFlow. And this will include a little tutorial um at the end. I might refer to it as part two um at the end here. But Neil is a data scientist at Iwaka which is a fintech startup and he's also the host of the AI stories podcast where he brings on data professionals and tech leaders and they talk about you know their careers and how they use AI and day-to-day life to to impact our world. So um so with that Neil why don't you go ahead and um uh and you know things should be good to go. Yeah thanks thanks a lot for the intro. Can can you all see my screen? Is that good? Yes, we can. Okay, perfect. Um, so yeah, hi everyone. Today I will give you an introduction to convolutional neural networks with TensorFlow. Really wanted to thank Data Science Dojo for organizing this event and yeah, hope you will find this useful and interesting. So, just a few words about me first. I don't think you know me quite well. So, as you can hear, I'm not from the UK. I was born and raised in Belgium, but I moved to London seven years ago to study civil engineering at Imperial College. And then in 2019, I kind of realized that I was really interested in data science and machine learning. So I did a master in London and there at the end of my master I did a research thesis on predicting the output of solar panels with satellite images and this is where I kind of went deep into convolutional neural networks because I had to deal with images and so that's also yeah how I got interested in this topic and that's why I'm giving this talk today. As mentioned already, I I'm a data scientist at Iwoka, which is a fintech startup and there I build machine learning algorithms for credit risk. So we are a lending company. We lend money to businesses and so we have algorithms which will decide who we are accepting for a loan, how much money we are offering, also some tools for losses prediction. So this is what I'm working on. And in my spare time, I also host the AI stories podcast where I invite people from all around the world to talk about their career and how they use AI in their day-to-day. Season one is now over and season two will start again in um September, October. So if you're interested, yeah, feel free to follow. All right. So let's start with the talk now. Um just a brief overview of what I want to talk about. I'm first going to give a brief introduction like five minutes on AI and neural networks just to bring everyone on the same page. Then we're going to go through the theory behind convolutional neural network. And again this is just to show you the intuition behind how they actually work. And finally, we'll have the collab where I will show you how you can build your own CNN algorithm with Python and TensorFlow. So, hope you will find this useful and interesting. Let's start with the intro. So in 2012, you may or may not know, but there is this AlexNet paper which is very famous and this is where deep learning becomes well known to everyone and the AlexNet paper basically sets a new record on the net challenge and ImageNet so you have a picture here you don't see things very well but it's just an image classification problem you have loads of images I think 10,000 different classes and you need to classify the images correctly. And so you can see that on the graph at the bottom like in 20110 and 2011 the best algorithms had an error rate of 28 or 26%. But in 2012 suddenly with this new AlexNet deep learning algorithm the error decreases to 16%. And that's like a big boom, a big impressive thing in the world of AI. And lots of researchers from computer vision were doing non- deep learning research actually move to deep learning because people kind of realize that deep learning is very powerful and could actually be the future of computer vision. So there is actually a paper from 1985 which trains a neural network using back propagation and convolutional neural networks have been around since the 1990s. So how come that it's just in 2012 that we realize that neural networks and especially CNN's can actually be so powerful? I think there are two reasons for that. The first one is we actually need a lot of computing power to train a neural network and this wasn't really available or known at the time. And the second reason is we actually need a lot of data and the AlexNet paper showed that with a lot of compute power and a lot of data you can actually train a neural network. Um nowadays it feels easy to train a CNN but back at the time it was much more difficult. So because I don't know how familiar you are and what's your background I'm just going to give a brief intro on neural networks before moving to convolutional neural networks because neural networks is really the basics and neural networks is essentially a network of neurons. So each dot here, each circle is a neurons and it's inspired by the brain and all those neurons are kind of connected to each other by arrows and each arrow represents a different weight. So neural networks have three different parts. They have the input layer, right, which is the input data. There's the output layer which is the prediction essentially. And in between that you've got the hidden layers. Um here we've got two hidden layers of four neurons each. But actually you can have 10 hidden layers, 20 hidden layers with uh how many neurons you want. There isn't really any rule for that. And this is just a toy example. And yeah, each hour is a different rate. So here we've got three times four. So we've got 12 weights in this layer. Here we've got 16 weights. And here we've got four weights. So 22 weights in total and using back propagation you can train this neural network and find what the optimum sets of weights are. Um I'm not going to go into back propagation or deep into neural networks if you're interested. There are lots of readings on that. I will provide some references at the end. But hopefully you get some idea of what is a neural network and how it works. Um, and yeah, I believe we're going to do questions at the end, but or do we want to I see we have a question now or a hand. Um, but yeah, let's leave the questions at the end of the first part because I might answer some questions. Um, yeah, in the coming slides. Um so yeah um that's how neural networks works and CNN are actually parts well you will see neural networks within CNN's so let's move to the interesting part of the torch convolutional neural networks and the first thing that we can see so I described before CNN's were used for image classification right with the imaget paper and an image is actually just a matrix, right? It's a matrix with and each value within this matrix is a pixel or a number. So here you can see it's just a 28x 28 matrix. And so the question is why do we want a neural network like why do we actually need why do we need a CNN and why not just a neural net of the first part? So why not using just a neural network instead of a convolutional neural network, right? An image is just a matrix. So what you can do is flatten this matrix into a vector and you can then feed this input into a neural network, right? And that's it. That's done. Why don't we do that? Well, first of all, as a human, let's say you want to classify dogs and cats. How do you do? You're going to look actually at groups of pixels, right? You're going to look at eyes, at ears, at the color of the skin. And that's exactly what CNN's do. They take spatial dependencies into account and they look at groups of pixel. Neural network will only look at individual pixels and that's very difficult. If you just have like a vector of pixels and need to identify dogs and cats, the task will be much more difficult even just for a human. So that's the first reason why we use CNN's and the second reason is that they are actually more efficient to process images. So here you have some kind of toy example of a CNN. The input is an image and then the network the algorithm is divided into two parts. Um the first part which is in blue is the CNN. So we have convolutional layers and then pooling layers. And the second part after the flattened layer is just a neural network. And so the neural network is going to take the features from the CNN and use those features to make the classification, but the CNN is going to extract the useful features from the images. Um, and so we've got convolution and pooling layers. And I'm going to talk about this in more details in the next slides, but for the moment, just keep in mind that we've got a first part which is convolution and pooling layers and a second part which is a neural network that you yeah hopefully you're kind of familiar with. So the intuition behind neural networks is that um behind CNN sorry is that on the first few layers of the model the CNN will be able to capture tiny details within the image like edges for example and as you get deeper and deeper into the model the algorithm manages to extract more kind of human interpretable features like eyes, head, heads, uh, flowers. Um, and so basically the first part of the network looks at details and the last part looks at higher level features in some sense. So let's dive into convolutions. Here you have a very clear example of what a convolution actually is. And there is a note here, a comprehensive guide to CNN's. Um I really recommend you to read this. I'm taking a lot of images from there and yeah it's very good if you want to have a good understanding of CNN's really recommend uh reading this blog post. So convolution here we have a 5x5 image which is represented in green right with some pixel values which are zeros and ones and the convolution is actually a kernel. So it's this 3x3 yellow square which goes around the image. That's a con that's a kernel and that's the process of doing convolution. You can see the red numbers on the image the times one or time zeros. Those are the weights of the kernel. So those are the weights that the neural network need to learn in order to um well extract meaningful features from the image. And so by kind of rotating this kernel around the image, we get what we call this convol feature um which is this kind of pink box. Um and so the first value for example the four here is obtained by multiplying this kernel by the values on this left top left corner. So here we've got 1* 1. Here we've got 1* zero. Here we've got one times one, zero, one, zero, zero, zero, one. So in total we've got four ones and um six five zeros. And so that's why you get a four here. Um you repeat that and you get the control feature. So that's that's it. That's what a convolution is. It's just a kernel which rotates around the image to extract meaningful information. Now you can again see a kernel here and I want to describe two additional words which are quite important stride and padding. So the stride is by how much you should shift the kernel. So you can define the strides as you want. Here in this case the stride is two because you can see that the kernel shifts by two squares on the right and by two squares from top to bottom. So the stride is two. The second thing that's important is padding. So you can see here those dotted lines um which kind of upsample this the image. That's what we call padding. So here we've got horizontal and vertical padding of one because we're adding one square on each side of the image. And the reason for padding is that sometimes you have interesting features on the corners of the image. And if you don't actually use padding, the kernel might be missing those features that are just close to the edges. And by using padding, you can actually capture better information. Sometimes finally I discussed the pooling. So if you remember here we have a convolution. So we apply the kernel to the image. And here I'm just showing a single kernel. But I don't see any reason why you cannot apply 10 20 or 32 kernels to this image. Right? You will just have 32 different convol feature. And that's what we do in practice. So that's why here you can see 24x 24 by N1 is just N1 kernels that we've applied to the image and that's why you have kind of different images here or uh yeah control features. After the convolution it is quite common practice to apply this pooling operation. And all this pooling operation does is it reduces the size of the convolve feature. That's it. So here you can see it's 24x 24 by n1. After the pooling operation, the dimensions here are are 12 by 12 by n1. So we reduce the dimension the dimensions by half. How does this pooling operation works? Um you have different kinds of pooling. You have max pooling or average pooling. Max pooling is probably the most well-known technique and the one that works usually quite well. Um, and essentially here again you just have a kernel that rotates around this image or this convol feature and max pooling is very simple. All it does is it takes the maximum from a set of pixels or values. So here you've got four values. If you apply the max pooling, it takes the maximum which is 20 and um extracts the 20 here. Um we use a stride of two here. So if we move the kernel two squares on the right, we have another max pooling and here the highest value is 30. And that's how we get this 30 number here. Average pooling is similar, but instead of taking the max, we just take the average. And we use pooling to reduce the dimension of the image. It makes it much easier to process the image, but it also helps to reduce overfitting because it's actually robust to rotations and translations. Um so a short example let's say that you have an eye in this feature um in this square and let's say that suddenly you're rotating this I you will rotate the values within this square but if you apply the max pooling the largest value will still be 20 and so the max pooling will still extract the feature even though you've rotated or translated the picture and so it makes the CNN kind of robust to rotations and translations. And that's basically it actually. You have your inputs, you apply a set of kernels convolutions, you have this max pooling layer, you can have another convolution followed by another max pooling operation. Um here we've got two convolution, but you can have 10 20. There is no rule for that. And then after that you can just flatten the output and feed everything to the neural network which will then do the job for you. Um so that's hopefully that gives you some kind of intuition about how neural networks work. And if it's still not clear hopefully it will be much clearer with part two where we're going to build our own algorithm using Python and TensorFlow. But first I believe we have some time for questions about the theory and then we can go into the code. So I'm seeing one question. Um and this is from Nahal and uh they want to know if you can share you have a source the comprehensive guide to CNN's um can you share that link with everyone? I know um we'll share the slides with everyone with the recording and and so you'll be able to access it through there but are we able to share that source now? Um, so here I've got a few sources here. Um, should I just Yeah, I can. Should I just share the slides on the chat since I'm done with the presentation? Maybe. Yeah. And then, um, data science dojo team, if you can share the Google collab with everyone now, that'd also be time to do it. Um, Sophia, I know you have your hand raised. If you can ask your question in the chat, or not in the chat, but in the in the Q&A tab, I'd appreciate it. Um, and then Neil, whenever you're ready, I have a second question for you. Yeah. So, I'm sure the slides and the references are at the end. Um, but yeah, I'm ready for the second question. Okay. Um, so this is uh from Maruchio. Uh, so they're they're referring to a second step, I'm guessing, in the in the neural network, but why does it have so many layers in the second step? Um, can we Yeah, can we clarify the question? Which second step exactly? Is it the one from I am not exactly sure. Let's wait for clarification. I think uh Maruio I think you're on one of our live streams but if you can just clarify for us um which second step neural network here is it the neural network from this CNN that you're talking about or yeah which second step is it and then I I would be able to answer and then uh Sophia has a question that I think will be a lot easier to answer right now is how do you determine the kernel? So the size of the kernel, that's a good question. Um, so here for example, we've got a 3x3 kernel um that rotates around the image. And the size of the kernel is actually a hyperparameter. So we don't know what a right size actually is. We need to try different things and see what works, which is usually how it works for neural networks. Um you typical sizes are 3 by3, 5 by five, 7 by seven. Um so you should try a bunch of those, see what works. But there isn't a clear rule on what kernel works best. If you have a very small image, I would recommend trying smaller kernels. If you have a large image, just try bigger kernels. That's usually how it works. Okay. And then this one is from Brent and they would like to know how to create the convolved feature. Uh they had a hard time following. So if I just recap here, you have the kernel and you have some weights, right? The times one time zero time one, those are weights that the neural network will learn. Here it's just examples of zeros and ones, but in real life it's not like that. It's just values that the neural network would learn and then you just rotate this kernel around the image. So you see every time you rotate the kernel you get a value here on this control feature. How do you get that value? By multiplying the weights from the kernel. So the zeros and ones that are written in red by the value the pixel values in the image. So to again describe the examples the example that I shared here you would do one * 1 right because that's the first weight of the kernel plus 1 * 0 which is 0 + 1 * 1 which is a one. So we're we have a value of two at the moment plus 0 * 0 which is 0 + 1 * 1 + 1 * 0 + 0 * 1 + 0 * 0 + 1 * 1. So in the end we've got four ones. If we sum all of them together, we get the value of four. And we repeat that for by shifting the kernel around the image to get this convolved feature. Hopefully that's clearer. All right. And then, uh, James, I want you to know I see your question. We're gonna wait to answer that one at the end because I think that one's going to take, um, a little longer to answer. Um, we should be good to go on that. So, let's answer one more question. Um, um, just before we we start the demo. Um, and I want to go back to where is he or she? Uh, Mari Maricio. Um so the the question about um um the layers in the second step uh they are saying yes uh from the the image in the slide the okay I think it was the one that you showed um yeah I see so the intuition here actually you need you can have as many layers as you want basically the convolutional neural networks extracts the meaningful features from the image so for example if If you want to classify um if you want to detect whether someone has a face, there is a face in the image or not, the CNN will kind of Yeah, I know. Sorry, I haven't shared my screen, but there is nothing to really show. I'm going to move to the collab after um or maybe I can well yeah so the first part of the CNN is basically extracting meaningful features from the data and those features could be eyes uh color of the skin things like that but then you still need a neural network to kind of um will detect all the kind of find patterns within all those features. So the CNN will find all the interesting features but you still want the neural network to see okay if he has the person has a mouse long hairs and eyes then it's a girl. If it has short hairs short hairs eyes and a mouse it's a boy. So it's kind of the neural network still has to find patterns within those features but the CNN will find the interesting features and so the job of the neural network is much easier because the CNN does all the hard job of fi of finding the interesting features. So yeah perfect um okay and then I let's go let's start with the collab. So hopefully now you've got some understanding of how CNN's actually work. So let's see how we can build one. And again the collab is divided into two parts. A very short intro to TensorFlow. You will see it's very simple, very similar to NumPy. So if you know NumPy, it's just going to be very similar. and then I will show you how you can train your own CNN using the fash using the fashion amnest data set. So if we go through the code first I'm just installing TensorFlow and importing a bunch of libraries nothing interesting um I import TensorFlow and TensorFlow. Um, Keras is a library built on top of TensorFlow which makes it very easy to build neural networks. So, I'm just going to use that for the tutorial. And here we've just got a bunch of basically cells which show you how TensorFlow work. and tensorflow. The main idea behind tensorflow is this tensor which is very similar to a numpy array. So here I create a tensor of which is essentially a matrix of size 2x3 and if you kind of then print this variable to the screen you can see how it looks like but it's basically very similar to numpy. like numpy, you can create an array of ones. If you call tf ones, you specify the shape, you get an array of ones. Um, and like numpy, you can do some operations between the matrices. So here we've got two matrices. You can multiply uh add them together. You can do an elementwise multiplication. You can do a matrix matrix multiplication. All of this you don't need to know for the tutorial, but it's just some recap and just to explain to you that TensorFlow is very similar to NumPy, but we use this to train neural networks because it works very well. It's computationally efficient and it's built specially for that. So that's why we're using tensors instead of numpy. And like numpy, you can also manipulate the matrices. So here for example I create a variable one which has a shape of 2x3 and then I reshape this variable to have a shape of 1x six. So one row and six columns. Um and you can see that it worked. I get another tensor and also again like numpy you can concatenate different matrices on top of each other. Um well you can have yeah basically here I build this variable one and I concatenate it on top of each other so you get the same matrix twice basically. So the first dimension the number of rows is increased from two to four. Um but again this is just play around with it if you're interested. It's nothing that you need to know for today, but yeah, just find it interesting to show how you can use tensors and manipulate tensors. So that's the first part. Hopefully very easy and yeah, just play around with it in your free time. The second part is the fun bit which is CNN's with the fashion amnest data set. So let's go slowly through this code. The first thing is just load the data. There is this command using keras which loads the fashion mn data set. Um and I have training data, training label, test data and test label. And then here I'm just storing the names of the class. So it's an image classification problem and we have 10 different classes. You can see we've got t-shirt, trouser, pullover, dress, coat, sandal. You can see it's different kinds of closes. Um why which is why we have the name fashion amnest. Um here just some visualization. So you can see that the training data we've got 60,000 images of size 28 by 28. Um the labels we just have 60,000 labels which is the class. And then the test data we have 10,000 images of size 28 by 28. So hopefully here very simple here I have some code which you don't need to understand. It's just to show you what the data set actually looks like. So you can kind of visualize the data set. We've got 16 images here. You can see it's just different clothes. The images are 28 by 28. So, the C quality isn't great, but hopefully you can see that's a sandal. That's a sandal. That's a sandal. That's a pullover sandal again. Um, shorts, ankle boots, um, bags, well, different kind of closes. So, a classic machine learning uh, image classification problem. Once we've loaded the data, the next thing to do is to actually process the data. And the processing is very very simple and only needs three steps. The first step is that I divide the data by 255. And that's just because all the values in the images are between 0 and 255 and I want values between zero and one. So just divide everything by 255. That just very common practice. And then the second thing that I'm doing is that I'm expanding the dimension of my training data. And the reason for that so you can see here I print remember that before sorry this should probably go in front. Um but basically I'm expanding the dimension of the training data and I will talk about this in a few seconds. The third thing that I do is that I'm just split splitting my training set. Remember we had 60,000 um images in my training set and I'm splitting it into training sets and validation sets. Um so that I can I don't touch the test set. I only look at the test set at the end of the process. I train my neural network on the training set and I leave the validation set to see how well my algorithm is performing. So here if you look at the shape of the training set it's now 50,000 rows right because I removed the validation set we have 28 by 28 images and then we have this extra dimension here which is just a one and this is what I've done here I just expanded a final dimension and it is a one and the reason why I'm doing this is because the CNN will expect to get an input of this shape And so if you don't expand the dimension, the CNN is just not going to work. And that's because usually with more complex images, you have actually three. The dimension here is three because you've got three different channels for blue, red, and green colors. So here it's simple. It's just a one, but that's why I'm doing this step here. So processing is very simple. We've processed the data. And now the final thing we need to do is to build the network. So this is where I build my convolutional neural network. And it might look quite complex at the beginning, but we're going to slowly go through that and you will see it's actually very very simple. So the model is I create this tfk.sequential object and this is just a list. Well, it takes as input a list of layers. So each line of code here is going to be a layer from my convolutional neural network. So each line is just a different layer. So this algorithm is divided into five parts. The first part is the input layer. That's simple. You just need to specify the shape of the data that you're going to feed into this algorithm. Right? Your model should know that it's going to be images of size 28 by 28 by one. So that's what I'm doing here. And we've specified the input layer. Then if we just look at the comments and not at the code, we've got the first, the second, and the third convolutional layer followed by the max pooling layer. So that's exactly what I described on the slides. We've got a convolutional layer followed by a pooling layer. Finally, at the end, we have a neural network. So that's just a classic neural network. um first with 256 neurons and then with 10 neurons at the end because we have 10 classes and so each neuron is going to show at the end is going to show the probability that the image belongs to a class. We take the maximum probability and this maximum probability is going to be the class that's the prediction class essentially. So if we go into the convolutional layers in more details, the three layers here are exactly the same or very very similar. And all it does is first of all you have the convolutional layer which is the TF key TF.con2D and so those are just kernels which are going to extract information from your image as I showed in the slides. 32 refers to the number of kernels that we will apply to the image. So in the example, we just applied one kernel with zeros and ones. Remember here we're going to apply 32 different kernels to extract more information from the image. The size of the kernel is going to be 3x3 as I showed you before. And that's because the image has a kind of small dimension. We set the strides to one which mean that we move we rotate well we move the kernel by one square and we have this activation function which makes the CNN nonlinear if you if you're interested in activation functions there is lots of literature on that I'm not going to go into too much details here and then all you do here is have the pooling layer of size 2x two which just reduces the dimension of the convol feature by two. It divided by two and that's it. You do that again with the second layer except that here we want 64 convol feature instead of 32. And then you do that again with the third convolutional layer. So we have three layers here. In the end we feed this into the neural network and finally we get the prediction. So hopefully that's quite simple. All you need is an input layer, a bunch of convolutional layers and with max pooling and then finally the neural network and that's it. That's your model. Um here I have there is some reading on the softmax activation function that I'm using at the end of the network. If you're interested to learn why I'm using this activation function, then there is a very good resource here um with some formula, but I'm not going to go into too much details. So remember what we've done and yeah, don't look at the maths here. You also don't really need it. All we've done is we've loaded the data, we've processed the data, and we've defined the model. We just need one last thing before we can train the model. And it's this cell here. We need to define an optimizer which this optimizer will help us to find the weights of the suitable weights for the algorithms. And we use Adam which is very very standard. We need a metric and the metric will tell us whether the algorithm is performing well or not. And so here we use a sparse categorical accuracy which is very standard. um for when you use classification it essentially gives a value of one if you've classified the image correctly and a value of zero if you didn't and then we also want a loss function because accuracy is not differentiable. So we want a loss function to train the neural network and so here we use the sparse categorical cross entropy and again there is some reference here if you want to know why um but I would say this is not really particular to CNN's that just a classic deep learning approach you need an optimizer a matrix and a loss function so read about that if you're interested what you do after that if you compile your model you set those uh those well you set the optimizer the last function and the metric and then all you need to do is train the model. So here I'm calling model.fit I'm fitting it to the training set. Um you obviously need the labels because the model is learning and I'm using this validation data because I want to see how my model is performing on unseen data to avoid overfitting. Um, here I've already trained the model. It doesn't take too long using Google Collab if you use GPU, but I've trained it already. And you can see here the loss function. Um, and basically as we train the model, we want the training loss to decrease, right? Because we want the loss to be as small as possible and we want the validation loss to decrease as well. So this tells me that okay my model is learning the loss is decreasing. Let's now see how our model performs on the test set. I'm with this line of code I'm just evaluating my model on the test set that we've never seen before. it evaluates and we can see that the accuracy is around 89%. Which means that 89% of the time the model will classify the images correctly. If you want to play around with the code, you can probably improve this by quite a lot. You will just need to maybe add additional layers to the model or change the kernel size or change the activation functions. You can play around with the model. Um, add more layers if you want, but if you want to learn, I will encourage you to try to beat this 89% score on the test set. I'm sure you can improve on what I've done here. We can kind of see the prediction. Um, so again, this is the data set and you can see on top you can see the prediction and at the bottom you can see the label. So we can see that the model is doing quite well for most of the values here. The prediction matches the label, right? Sandal, sandal, t-shirt, t-shirt, trouser, trouser, bag, bag bag. So, it looks like the model is doing quite well. And we can also see, don't worry about this code, but we can see where the model fails actually. So, here you can see examples where the model fails. Um, and you can see that it's mostly around t-shirts, pullover, coat. That's where the model seems to struggle. Um, because here it predicts coat when it's a dress. Here it predicts a t-shirt when it's actually a shirt. And we can see that it's actually quite difficult here. Um, here it's a pullover. It predicts a pullover, but it's actually a coat. So anyway, you can see where the model is wrong. And if you want, you can also print the confusion matrix to see where your model struggle the most and try to improve it. Um, but yeah, um, this just shows that we've basically trained a convolutional neural network on the MNES data set and the model seems to perform quite well with like an 89% accuracy on the test set. Um yeah, feel free to try to beat this and improve on that um during your free time. Um and yeah, I'm gonna take some question now. Hope this was clear. Okay, perfect. So, we do have uh 16 questions and probably counting. So, um Neil, I know yesterday we talked that we might go a little bit over time. Is that still okay with you if we go a couple minutes over? Yeah, I'm happy to stay 10 15 minutes longer if to answer questions if people are still okay with. Okay, perfect. So, um let's try and get and I've been kind of going through these as we've uh as you've been talking and I kind of want to I want to try and get the longest question out of the way first and I'm hoping you're the best person to answer. Um and so one I would suggest opening the Q&A. um read it yourself. I'm going to read it out loud, but it is it's about a paragraph, so I'm gonna I'm gonna read it out loud while you read it. So, this is from James. Um, is there a way to align CNN's with digital twin simulations while updating real-time data Python pipeline? I'm currently using Unreal Engine and Unity in creating dynamic immersive VR digital twins while connecting them with CNN optimizers for real-time data input. Would you understand which optimizer would work best? Also, what is the optimal number of epochs one should use to maximize the accuracy of the CNN? Cool. So, um first of all, I'm not very familiar with digital twin simulation. So, not sure if I will be I'm a twin like I actually have a twin brother. Um I am also not familiar with digital twins. Okay. So even if you're you're you're a twin and you're not familiar that's um but yeah basically but I I still think I can probably help with which optimizer would work best. Um Adam is quite well the classic one I would say. I pretty much always use Adam because it's known to be quite good. There might be better optimizers for CNN. I'm not a specialist, but I know Adam is pretty good. It deals with momentum and things like that. So, I would recommend Adam, although if you if it doesn't work, do some research. I'm sure you can find better ones as well. And the second question, which is what is the optimum number of epochs one should use to maximize the accuracy on the CNN. So, I cannot give you one number because it just depends on the problem. But you can see here. So here I've only used 10 epochs just because I didn't want to train the CNN for too long. Usually we would use more epochs, but at some point so the validation loss here is decreasing. But at some point you will see that you will actually start to overfeit. And what overfitting means is that the training loss will go down because you will learn better and better on your training set, but the validation loss will start to go up. And so what I recommend doing is looking at that and when you see that your validation loss starts to go up after I don't know 10 epochs or so you can stop training your model because it means that your model is overfitting um and won't be as good. So that's how you can find the right number of epox would say and that's called early stopping. if you're interested um yeah just look at early stepping and and I saw James had a second question um a little later on so I'm gonna ask that now as well so um in this tutorial are you using the softmax optimizer as a result of having 10 different class outputs so soft m I'm basically using the softmax activation function in the end um and So it's not really an optimizer. It's an activation function which so basically my network is you're going to have 10 neurons at the end of the network. And what the softmax does is it transform those values into kind of probability. So you're going to have 10 probabilities which is the probability that your image belongs to a class. So the first neuron would be the probability that it belongs to the first class. The second neuron would be the probability that your image belongs to the second class etc. And then you can just take the highest probability right to have the prediction. So that's what the softmax does. Um so I'm using the softmax as the final layer as the activation function of the final layer of the CNN and the neural net basically. Okay. And then this is this one's from Paula. Um, how do you determine the size of padding? So again, that's the same as kernel size. I would say that's a hyperparameter. So here you can see there is no padding. I didn't use any padding at all. Um, that's I think that's the default in KAS. Um, so I didn't specify any padding here. But basically the size of the kernel, the size of the padding, the size of the stride, all those things you need to try different things and see what works. Um, here no padding works well. Sometimes you can use one or two paddings. I Yeah, I haven't seen much more than Yeah, one or two except if you're really expecting to have a lot of uh information. I mean it could be also higher but if you don't expect to have a lot of information close to the corners um yeah I think zero one or two to start with should be all right. Um yeah you just need to experiment. There is no strict rule here. All right and this is from Pedro. What's the main difference between max pooling and average pooling on performance? So usually max pooling performs better um and that's because max pooling extracts the dominant feature. It takes the maximum value so the highest activation in some sense and it extracts this whereas average pooling does some kind of smoothing. Um, so that's why I think max pooling is often better, but I don't think there is a strict rule. Like it could be that sometimes average pooling works best, but from my experience, yeah, I pretty much always use max pooling. It's kind of the the standard because it extracts this dominant feature, I believe, whereas average smooths things out. But yeah. All right. And and this this one's from Anushia. Um, how many images approximately do we need for performing CNN's? I know it's subjective, but is there like some type of guideline or um, if you can provide some type of idea that would be great. Very good question. And yeah, I'm again I'm not really sure. I will try to help and say that first of all, it depends on the difficulty of the problem. uh if you have like if you just want to classify black and white images of numbers for example that's probably easier than if you want to classify very complex large images of I don't know different types of dogs right so it depends on the complexity of the problem and it also depends on the number of classes that you have if you have two different classes you need much less images that if you have 1,000 classes, right? Because you want enough samples from each class. I would say probably want a couple of thousands of example for each class. That would be my guess. So if you have 10 classes, at least 10,000 examples. Um but again it depends on the complexity of the problem and yeah on the complexity of the problem the size of the images and things like that but yeah couple of thousands at least. All right and this is from uh Naomi. Is it possible that using too many convolutional layers could degrade the classification accuracy? I think it could be possible, but yeah, I'm not. It will probably depends what you're doing. I guess if you add additional layers with the wrong activation functions and things like that, I guess it can. Um, you just need to try. It's also I would say depends on the problem. Um sometimes fewer layers might work better. Um it's not like all the time increasing the size of the model is better but you will see now that all those state-of-the-art algorithms are very very big and uh so usually would think that by increasing the size of the layer we will get a better performance. Um, but then yeah, if you want to have a very very big algorithm, you're probably going to need lots of computing power. You're going to need to wait a long time to train your model and so it might not be as good. So it's but I think yes, it could be possible that adding more makes the performance worse, but usually more weights means better performance. Okay. Then sorry I'm just filtering out questions as they get asked. And um so this one's from Zach. Do you need to optimize the hyperparameters? What would you use something like grid search CV? Yeah, exactly. You could use that um some kind of grid search. Um yeah, a grid search with different sets of hyperparameters like kernel size, um number of layers if you want, um the number of channels, right, of kernels that you're going to have in each layer. Um yeah, you can use some kind of grid search to find the best parameters. Um yeah, that's usually good practice. And yeah, remember that you would also need to look at the learning rate, the batch size, all of these which are more common deep learning algorithms would also help. And I'm pretty sure there are other methods than grid search that you could use for kind of smarter searches, but which might be which might be better, but grid search is exactly something that you could do. Okay. And then this is from Morad. Can we use CNN with other types of data such as genetic data? I mean we can classify numeric values not only images or CNN uh will not work properly. So do we have to use images in a CNN or can we use other types of data? You can use other types of data. I'm sure there is some research on language. There's probably some research on language for CNN's but they really work best and they are well known for images. So if you have like tabular data I would not recommend to use CNN's at all because I'm well traditional algorithms like boost XG boost or even regression and random forest will perform much better. So yeah, don't use CNN's on raw data and images and on text. There might be some research, but I don't think they're Yeah, there might be one or two p well a few papers where they've been useful, but it's really not a standard like CNN's are well known for images and in except if you really have a good reason, I would just use them for images or video visual data because that's what they've been built for. Okay. And then this one is from Roya. Um, sometimes TensorFlow takes a lot of time to be run. Is there any way we can decide better from the beginning how many layers to use? So, one trick that I have is, and I'm going to go back to this graph. You can see here that the training loss is going down. and forget the validation loss. You can see that the training loss is going down. So when you start, make sure that that's the case. Like train a big enough CNN such that the training loss is going down because this means that your algorithm can at least learn the training data. And once this is done, you can have some techniques to avoid overfitting like dropouts, um stronger regularization, um things like that. But first, make sure that your training loss is going down by the amount that you want because I see sometimes people training a neural network and it doesn't work. That's because if your model isn't big enough and cannot even learn the training set, it's never going to do well on the validation or on the test set. So first of all, make sure that your training loss is going down and then you can start looking at your validation loss and reduce overfitting. So there isn't a clear size, but just make sure your model is big enough to be able to learn from the training data because if you can't overfit your training data, then your model would not will never be able to do well on the test set. That would be my my advice. Okay, so we're going to try and get through like two or three more questions before we we sign off. So this is from Kumar. Um, how do we decide the initial weights used in the filter? That's all random. Um you the there is some initialization that you can set. I'm not sure what's the default one, but it's basically KAS does this for you and it's pretty much random. There might be some smarter ways. I'm not an expert in that. There there might be some smarter ways to initialize the weight, but random should be fine. And then by training your model, the weight will converge to their optimum value. But initially you don't need to worry about anything. You just set a random value and by looking at examples of or this is a dog or this is a cat or this is a dog or this is a cat and using back propagation your model will learn by itself what the weights should be. So here I believe it's just initializing things pretty much at random because obviously your model doesn't know what the task is going to be. um it doesn't know that it's going to be used to classify closes. So in the beginning just initialize things at random and lets the back propagation algorithm and the neural network do the job. Okay. So I think this one's going to be our final question and we're going back to Roya. Um can CNN use a combination of image pixel array and some other related variables from Excel? For instance, merge MRI pixel array plus patient demographics from Excel. If so, can you please explain how? Thanks. Patient demographics. So I assume that the first one is an image and the second one is raw data. I would assume. Y and so yes, you can actually do that. Um, and what you would do is you would have the CNN, you would train the CNN on the image data only because you don't want to train the CNN on the row data, but then you remember that in the end you flatten the features from your CNN and you feed that into a neural network. Right? So here there is I don't see any reason why you cannot just add additional neurons in the input layer which will be your demographic data. So the CNN will should only be trained on the image but at the stage of the neural network instead of feeding it only the input from the CNN you can also feed additional input. So it's at the second step of the algorithm that I would feed those additional features basically. Okay, perfect. And I am going to cut questions off there because we are 10 minutes over and I want to make sure um we have time to do a closeout. So um thank you Neil. I'm just going to share my screen really quick so we can talk about next week's webinar. Um give me one second everyone. Um, so next week will be July 27th, so seven days from today. And um, we're going a little earlier than we were today as well. This will be happening at 9:30 a.m. Pacific on July 27th. We're going to have a crash course on designing a dashboard, a dashboard in Tableau. And we'll have Irene Daomi with us. She's a data analyst as well as a Tableau ambassador. Um, so we're going to be talking about what is Tableau, how do we design a basic dashboard, you know, how do we include a bar chart in our dashboard and as as well as other things. Um, and how are we going to format it um for good design practice. So um if you want to learn some some quick things on Tableau Tableau, join us next week along with Irene and um and we'll see you all there. Um but Neil, thank you so much for being here with us today. We really appreciate it. We'll make sure to get your uh the recording of this up. It'll be up on our YouTube channel um as well as on our website online.datascience.com. Also, if you are one of our subscribers to our newsletters or um if you RSVPd on our website specifically for this event, we'll make sure to send you an email with the with the link to the recording and the slides and the uh Google Collab and and all the good things Neil shared today. So Neil, we really appreciate you being here. Um, everyone, thank you for joining and I hope everyone has a good rest of their day. Thanks. Thank you.