Batch Normalization in neural networks - EXPLAINED!

Key Takeaways

greetings fellow Learners before we get into this Brazen world of batch normalization I have a thought-provoking question for you do you like hiking and if so do you like one of those irregular hikes with steep inclines or than just deep troughs or do you just like a chill Trek just gradually going up or do you just not like hiking at all I personally love hiking with some challenges in between but please do let me know what you think down in the comments below and I would love to know your hiking Endeavors better also maybe come back to this question after watching this video just to see how hiking relates to everything we're going to be talking about here now this video is going to be divided into three passes starting with an overview then some details and then code so let's get to it let's motivate batch normalization in this past so this is a feed forward neural network and let's say we want to train this network to take in an input image and determine if it is a dog or not a dog now during the training phase we feed hundreds of image plus label pairs after each iteration the network parameters are updated and the model learns now let's try visualizing this this is a contour plot of just two parameters of the neural network in reality the network has possibly hundred hundreds or thousands of these parameters now consider this Contour plot like seeing a mountain from a bird ey view so the blue and the green are the low parts of the mountain and then the red and orange are the high parts of the mountain so blue and green are low loss whereas the red is high loss now say we initialize the network with some parameter values so effectively on the Contour plot we might be starting here somewhere pretty high up the mountain now we perform one iteration of training by showing an example to the network so the parameters are updated and the loss changes which we reflect on the Contour plot now notice that as training continues the loss eventually converges to the blue and green Parts which it should after all this means that the loss is decreasing with training but you'll notice that it's kind of pretty zigzaggy and this is mostly because this Contour plot is very stretched along parameter One Direction and a little bit more compressed along the parameter 2 Direction now this isn't great because that means that a small change in parameter 2 can easily overshoot the minimum point and also small changes in parameter 1 don't even affect the loss at times so that's why hence that we have a zigzag pattern now to resolve this we can normalize each neuron output in the neural network and if we were to use batch normalization during training maybe our Contour plot looks more like this now this Contour plot represents a more smooth terrain so during the training phase we might start let's say here and then after every iteration the jump in this lost terrain converges more cleanly to the lowest point and so training is more stable and it is also more quick quiz time have you been paying attention let's quiz you to find out what are the problems with this Contour plot a possible unstable training as small changes in parameter one can lead to large jump in loss B possible unstable training as small changes in parameter 2 can lead to large jumps and loss C possible unstable training as small changes in both parameters 1 and two can lead to large jumps in loss or D there is no problem and we should have stable training comment your answer down below and let's have a discussion and if you think I deserve it at this point please do consider giving this video a like cuz I will really appreciate it that'll do it for quiz time and pass one for now but keep paying attention because I will be back to quiz you so let's go through more technical definitions of batch normalization why do the Contour plots get stretchy for some specific parameters compared to others well the reasoning is complicated but one reason it happens is because of internal covariance shift so let's talk about it this is a neural network and we want to train it to recognize dog images so in the first iteration let's say that we pass an image and let's say this hidden layer top neuron activation it's five and a few iterations pass now in iteration number 10 we pass an image and the top neuron activation is let's say it's eight a few more iterations pass let's say that iteration 100 we pass an image and the top neuron activation here is 18 now for each of these three cases the distribution of the output of the neuron is different though we're still measuring the output of the same neuron so if we train the neural network we calculate all the activation values of this specific neuron and we might see that the variance of those values might actually be pretty large and if the variance of these values are large that means small changes to the parameter arameters of the network can cause large changes to the neuron output this in turn can cause large changes to the output of the neural network which can in turn cause large changes to the loss so that means even if like the parameters change by a little bit some small change this could potentially cause very large changes in the loss this is why the Contour plot appears stretched out because small changes in parameter 2 can potentially cause large changes in the loss and as we saw in the previous case this can lead to training instability now to correct this we can perform batch normalization batch normalization operates across a batch of samples so neural networks in practice can take multiple samples at once and make the predictions for all of these in parallel now let's say that the batch size is three this means that we can pass through the network we can pass three images and we can get three activations for that specific neuron at the same time now this visual over here I just took a screenshot from the main paper of batch normalization and it tells us the map that's involved and what happens so we would first compute the mean across the batch then we would compute the variance for this batch we would then normalize these values by subtracting the mean and dividing by the standard deviation each neuron will have two learnable parameters associated with it gamma and beta so we multiply the output value by gamma and then add a beta value to it and that'll effectively scale and shift these values for a more detailed math explanation I highly recommend checking out this video that I made on batch normalization 2 just know that the overall outcome of applying batch normalization is that the neuron distribution output Val values they had high variance they are now squished so they vary less and this means that changing parameter values doesn't change the activation itself too much and this has a cascading effect on loss so low variance activation of neurons means low variance output of the neural network which means low variance in the change and loss and that's why the Contour plot looks more smooth like this que time it's that time of video again have you been paying attention let's quiz you to find out gamma will learn to approximate blank and beta will learn to approximate blank of neuron activations a the first blank is true mean and the second blank is true variance B the first blank is true variance and the second blank is true mean or c neither of the above comment your answer down below and let's have a discussion and that's going to do for quiz time number two and also pass to of the explanation but keep paying attention because I will be back to quiz you now for p three let's actually take a look at bat normalization in code we'll compare the performance of networks with and without batch normalization so we first start out with importing torch light libraries so in this case we're going to be using the mnus data set which is a data set of images we want to normalize the input values passing in a mean and standard deviation we'll then load the data set defining the batch size here as 64 we will then Define a neural network here so in defining a neural network we are going to extend the class torch modules and because we are extending in this class we need to override the function forward and in this case we are also overriding the Constructor so the Constructor is going to Define different components of this neural network so in this case we have an input layer hidden layer and output layer over here where this is going to be the the image Dimensions 28x 28 we have some hidden layers over here and then we are going to Output a neur enal Network layer of just size 10 because it's going to be a classification of the images into digits 0 through 9 we then Define batch normalization layers here so this Ed batch Norm is going to be a Boolean value if it is true we are going to use batch normalization layers if it is not true then we're not going to use batch normalization so the batch Norm layers will then take in an input of how many units to actually include in this case we want this Bator layer eventually to follow this fc1 layer of 512 and batch two normalization layer to follow fc2 of size 256 hence these parameters are chosen accordingly now we use each of these components in the forward pass of our neural network so first of all we're going to take the input image through the Network and flatten it we are then going to pass it through the first layer followed by an activation function and if the batch Norm is true that means we're going to use a batch normalization layer and we add it here we'll then pass it through the layer fc2 followed by a relu activation and then apply batch normalization once again if required we'll then pass it through the output layer and this should be a vector of size 10 for for every single example in the batch and we will return the output over here next we Define a function to train this model so for every single Epoch we first zero out the gradients we're going to make a prediction for the batch of inputs we'll get a batch of outputs we then Define the loss Criterion over here which you can see down under it's going to be a cross entropy loss we will then use the loss. backward function which is going to perform back propagation in order to compute the gradient values and then Optimizer step is going to use the optimizer so the optimizer here is going to be an algorithm that updates the neural network parameters we are going to be using the optimizer called stochastic gradient descent and we pass in the model parameters itself along with their initial learning rate and also this is going to be stochastic gradient descent using the concept of momentum now if you want to know exactly how momentum works I have defined this in my video right over here so do check it out on optimizers it's a pretty fun watch so once we update the parameters of the neural network we're going to display the loss itself next you'll see that we Define a neural network with no batch normalization passing in the loss function along with the optimizer and then we perform the training we do the same exact thing but with batch normalization layers passing used batch Norm is true passing in a loss where it's the cross enty loss stochastic gradient descent with momentum and then we will also train this model as well so we're training one model with no batch rization where the loss results are planted here and then one with batch normalization with its loss results printed over here now with the training phase itself you can kind of see that after a few of these iterations the loss values that you see without batch rization are actually higher for the corresponding with batch normalization which just means that the training looks like it's happening fast with bash normalization and now let's actually perform some evaluation over here which we load the mnus data set once again and we are going to get the model predictions we'll then try to these predictions are going to be probability values between 0 and one which we want to just actually map those to classification values which we do right over here and determine how many of the cases we got correct versus not correct in terms of classifying whether the image was recognized correctly and you can see that the accuracy with batch normalization slightly outperforms that without batch normalization so overall you'll see that the convergence was slightly faster with better performance but do note that this results and comparison can vary as you change the architecture of fural network as well as the amount of data that you have in your training set quiz time okay this is going to be a fun one which of the following does bash normalization not address a improved accuracy B faster convergence C decreased model complexity or D decreasing the importance of initial weights comment your answer down below and let's have have a discussion and once again if you think I do deserve it please do consider giving this video a like because it will help me out a lot now that's going to do it for Quiz 3 and pass three of this explanation but before we go let's generate a [Music] summary neural networks can process batches of data in parallel now batch normalization is a technique neural networks use to speed up training make training more stable and reduce internal coari shift and that's all we have for today now the code for this video is linked down in the description below and this code is present with all other videos in this deep learning 101 playlist and also to continue your understanding of batch normalization I highly recommend you check out this video right over here it's a nice supplementary video so thank you all so much for watching if you do think I deserve it please do give this video a like subscribe and I will see you in the next one bye-bye