Batch Normalization | Deep Learning | Internal Covariate Shift | Detailed Explanation

Hello and welcome. I hope you are having a great day. In this video, we are going to understand a very important concept in deep learning called batch normalization. A similar concept called layer normalization is also used in the very famous Transformers architecture. Okay? So, what is batch normalization? How does it work? And how it improves training? All of these questions will be answered in this video. So, let's begin. So, as we know in a deep neural network say for digit classification, your MNIST digit classification. So, what happens? You have the input signal coming in. And each layer will do some transformations and add some non-linearity. The hidden layer. Some sigmoid, tan h, relu, something. Each one will add some non-linearity. So, the distribution whatever distribution is coming in in the input, that distribution will keep on changing over a period of time because every time some part maybe the relu will truncate it. The sigmoid it if it is coming like this, it will change it into S curve. An S kind of curve, right? And tan h is also similar. Relu will just truncate the negative part like that. So, each one will introduce some non-linearity and that way it will like distort the incoming input. So, each layer will have will not have a correct view of what is coming as the input since because each layer is changing something. And if each layer had the exact input or with very less change input, then it can learn different aspects of the input data. So, that was the problem that was observed, right? So, I hope you get the problem. Each layer introduces some non-linearity, some distortions. Because of that, the form of the input distribution, whatever is coming in, that is morphed or changed over passing through multiple layers. And later layers cannot learn properly. Also, each one will change the weights because after the weight whatever is coming we start with some initial weight and sometimes some of these are like relu is positives will be included negatives will be gone. Right? So, such kind of changes and even in back propagation some weights may increase very much. So, that is change the distribution. So, for example, if the weights are increasing very much, so is it Gaussian and some new and sigma squared is coming. So, if the negatives are truncated, obviously sigma squared the variance of this will change. If the values are getting increasing because of the or some decreasing, so the mean will change. So, basically the distribution the probability distribution that was coming in after certain layers it is changing. So, the layer four may not get a correct picture of what was the input signal that was coming in. And this is like layer four only. There can be more layers. Right? So, that's the problem there. And how do we solve this? So, this was the problem that was seen and that was causing this the deep learning models not to train so well or take more iterations. So, I'll just state the problem officially. So, in deep neural networks, so you get the I've told this is the meaning and now you will get the terms now. During training as weights update, so everything is weights, right? So, you give all the neurons will have some weights coming in. It's an input matrix in terms of weights that is coming in. The distribution of the input values to hidden layers changes continuously because of all the back prop back propagation and the non-linearities and the calculations that are happening at each level. At each layer. Hm? It is also called internal covariate shift. We have seen this data data drift also. Basically, the probability distribution that was coming in that is that is not the same and it has changed. It has shifted. Covariate shift. X {comma} Y the relation that has changed now. X has changed. You understood this in data distribution. I'll put the link of that also in the description. Okay, and then we're discussing machine learning classical machine learning. So this causing internal covariance shift. This is what it simply means. The terms look a bit odd, but this is the distribution. So it may be a Gaussian. The mu and sigma of this will change over a period of time by a after passing through multiple layers. And because of that there will be multiple problems. So what are the problems? So this change forces layers to constantly adapt to new distributions. Every time because every iteration every iteration will start and then every iteration the new mini batch will come in. And then it will see a totally new distribution because after multiple layers the distribution has changed. And every time it will see a new distribution coming in. So that is slowing down the training. The neurons are not able to cope up with the change. See? And then because of that that hinders convergence. Convergence means coming reaching the minima. So we are trying to converge to this minima. Anyways, we have non-convex surface, but any local minima we'll try to reach, right? So that will slow it down. We are not able to learn appropriate parameters. So this will hinder convergence and make hyperparameter tuning also difficult. So so many things are changing. The the distribution itself is changing. What will the layers learn? So this was the initial hypothesis given by the in the paper that in which that was introduced. So this was the This was introduced in 2015 by these two researcher scientists. Very difficult names. I will not try to pronounce them because if I get it wrong, the people will get offended. So even these people so I'll not take the names, but these two scientists, okay? So batch normalization is a technique to control the variation in the features such that they do not vary too much. How do we do that? So what are we doing? We are just the changing the means pushing back the mean and variance to what it was. So, if I subtract the mu or mean from each of the data points, so basically I'm moving the mean to mean back to zero. So, I'm subtracting the mean from each of the data point, I'm moving the mean to zero. And if this is the variance sigma squared. So, if I'm dividing by the sigma squared, whatever is the sigma squared, so my sigma squared will now be So, if my sigma squared was five. And if I divide by sigma squared, so end of the day my again my sigma squared will become one. So, basically I'm moving it my mean to zero and sigma variance to one in each in each batch. For each batch I'm doing this. Okay? In each layer, whichever layer I want, each layer I can put this normalization so that my mean and sigma are reset back to new one um new mean to zero. So, the whatever changes has happened, it is reset back to this. Because even at the input layer we are doing this, everything is standardized. Right? Standardization is done. So, mean and sigma anyways at the input layer also we are doing uh um uh um mu zero and sigma one. So, same thing we'll try to continue across all the layers. And that's what it means. So, that my distribution doesn't shift. The distribution doesn't change. That's the idea that was supposed. So, this is the mathematically what is happening at each layer. I'm bringing back the mean to zero so that the weights don't keep on like increasing or very decreasing so much. So, this will control my weights and also the distribution we won't change. It will be kind of constant and each layer we'll see kind of same picture of the data that input that is coming. So, it will basically counter the change that has happened. That's the whole idea. And this epsilon is a very small value for avoiding division by zero. Very small, 10 ^ -5, 10 ^ -8, like that. These values. Prevent division by zero. So this was the idea in 2015. So this is done. Okay, so what are the benefits of this? So this mitigates the main thing that changing distributions internal covariate shift the distribution of the data that is coming in is not changing. So it is mitigating that. It won't eliminate it totally. It will reduce the impact. At each layer I'm just shifting back the mean and variance to zero and one. Prevents vanishing and exploding gradients. My weights won't increase because my mean and because my gradients will be calculated on the weights only. If my weights are not increasing very much because I'm pushing my mean to zero and variance to one. So it won't explode. Similarly for the vanishing gradient also. So this will prevent that. It won't become so less because I'll if the variance has become less than one, if I divide it back with 0.1, so again my data points weight will increase. The reverse will also be true. So prevent this. It will also help that. But this is important point. This was later in later research papers people have seen that why batch normalization works. So that they have seen that this is an important part. Okay, this one. Okay. Many papers so empirically. So the paper research percent that this is doing internal covariate shift. It is handling that. It is mitigating internal covariate shift or it is handling the with I'm reducing the change in the data distribution that has come in that was coming in at each layer. But what later papers published their papers published that told that this smoothing the optimization landscape basically it is saying that the loss function the loss surface that is highly non-convex in nature. Right? The in deep learning the loss function surface is highly it's non-convex. So it is it can have multiple local minima and it will be it is very rugged. So they said that this is smoothing it. Smoothing this means now I'll have a better surface. So it may have like this. So here if I started earlier here, so I may be stuck at this or stuck here like this. Sure freezing here. Here this is smoothing out. So my gradient descent will become smooth now. I can come down easily. This is the whole idea. Later papers were there that this is the main thing that it is doing. So that was the claim. That it is more than handling this inter internal covariate shift, it is handling this. It is making the loss function non-convex surface smoother and that made our gradient descent more faster. So our optimization techniques that we used, right? Our Adam optimizer etc. which had Earlier we have discussed in the videos. Right? So it made the optimization faster now. So that's what was helping our training or optimization. So this was the main claim. And this was This is the latest claim, so this is This holds more authenticity or or power or weight than this claim. Okay? So this is smoothing the loss landscape. And also there's one more effect, it acts as a regularizer because it calculates the mean and variance on the mini batches and each man in mini batch in a stochastic gradient descent is a is a random stochastic is randomly I'm taking batches of the data, calculating mean and variance on that and dividing it with that. So this randomness Okay? This the randomness is causing is introducing this is causing my means it's preventing my data from overfitting. Cuz as soon as I introduce randomness my model will not be able to learn the pattern exactly. And what is happening in the overfitting that it is learning everything. So all this whenever we introduce randomness, means our model cannot learn any everything. Okay, what is happening? Something is happening randomly, so it cannot learn it. So, whenever you introduce some kind of randomness, automatically the model will not overfit. That's the main funda. Keep it Keep that in mind always. Whenever you're seeing that some kind of randomness is being introduced, means that will prevent the model from overfitting. That's what is happening. So, when we're calculating mean and variance on each mini batch, and each mini batch is chosen randomly, stochastically. So, this is this all this randomly taking mean and variance and then the changing the mean and variance of the data. So, that introduces small amount of noise. And that causes the model to not overfit, or it will act as a regularization. Similar to dropout, as because in dropout also randomly we are dropping randomly neurons were turned off. I hope this concept is clear. This is the main thing. So, what are we doing? Scale and shift. So, we always normalize to mean and variance to one, but this may restrict the learning too much. For example, in in sigmoid sigmoid is like this, right? So, if it is in like it is the here it is This is the function. Here it is 0.5. So, if it is one Sorry. Here it is zero. Here it is one. Here this is 0.5. So, in these regions, where it is close to zero or close to close to zero, close to one, here it is kind of linear. Straight line. Here only you have this non-linearity. This curve. This kind of curve. But in this region, it is mostly straight line. So, if it is stuck in this line in linear region, okay? So, this might restrict it too much if my mean and variance is set to zero and one only. So, if it if it is fitting in these regions. So, that means very restrictive because it is kind of linear. It will make it kind of linear only now. If it is coming this it is fitting in these regions only. So, my model become linear it is it will be in a straight line only. So, that is the drawback. Restrict the layer too much, for example, forcing everything in the linear region itself. If it is coming to zero, everything is pushed to zero side. So, this might lose the representational power because all the non-linearities only giving me the power of deep learning. So, to counter that, what was proposed that So, let there be two parameters gamma and beta, which will be learned. So, if gamma is equal to one and beta is equal to zero, means the mean will be whatever is there. The mean will be zero only and gamma is one means the variance will be one only. So, it is it will be same. This is like variance and this is like mean. So, it will mean same. But, if there is some other mean and variance that is that is appropriate for this layer. So, that will be learned. That will be learned. So, this introduces two learnable parameters called gamma and beta. So, gamma is a scaling parameter. Scaling means scale the variance. Variance what is scaling, right? Multiplying. Scaling means multiplying. So, if variance if I don't if gamma is one means it will remain the variance if it was one it will remain the same. And beta is if I'm adding beta beta beta and whatever what was where what am I doing? X minus mu. If I'm adding some more thing here, some more value here. So, that will cause the X minus mu. So, it showed that So, I'm adding plus beta. If I'm if I was reducing the mean from here, so this was causing zero, but say if I add something, so I mean the mean maybe Now, it may not the mean may not be zero, but say this is two, then my mean may shift to two now. And that is the requirement of the layer and that works well. And this will be learned automatically, the network will learn these two parameters. The shifting parameter, what is mean shifting, right? Say for example, this is the So, this is mu equal to zero. And if I have another like this. So, here this mu can be equal to two. Like this. So, this is shifted. If I add plus two to all the data parameters, it will just shift by two. That's what it means. There's a shifting and scaling is the variance. How much is the spread? If this is there, if I spread everything out, then this will become like flatter. The spread out mean. So, variance will increase. Okay, so this is your scaling parameter, similar to this. So, and it tells that let the network decide what is appropriate. If it comes out to be zero and one, no problem. But if it is not, then let the network decide. You need to let it learn. So, it can introduce two learnable parameters for it to learn so that it may not be restricted to zero and one. And this will decide the optimal state for the network. And the learned values of gamma and beta, this will be learned during training. Okay, so this is one important thing. So, what happens at inference time? At run time, we are having many batches. At Sorry, at the training time. And each mini batch will have, say, 256 items are there in a non-sample. So, I can calculate the mu and sigma squared. But what will happen in at run time? Run time I have only one data point. One data point is coming. How will I calculate mean and variance? So, what happens is during training I have the current mini batch, from that it is fine. But at run time, whatever was the mean and variance that was learned or calculated during the training time. So, we freeze that. Or I mean, we take a running average and freeze that and we use the same thing at at at the inference time. So, in short, whatever was learned during training what should be the mu and sigma square. The same thing, we will freeze it and we store it and the same mean and variance will be used at run time for the data point that is coming in. Okay, that's what it that's what is done. I hope that makes sense. So, let's see it in the code what is happening. So, this is a simple data generation here. And this is your batch normalization. So, each layer So, this is a dense layer. So, this is a data creation part. So, here I'm creating the model. So, you're creating sequential. So, I'm creating a dense layer. This is the main part, normalization. So, I'm normalizing. I'm introducing a batch normal. So, each layer I can introduce a batch normalization. Here also I'm introducing a batch normalization, batch norm two. All all this regularization etc. is there, but this is the main thing to focus. So, when I'm creating the model itself, there only I'll tell that I'm introducing batch normalization. And then I can have dropouts also. All those things what you have learned, you can club all of them together. So, I'm just adding concepts one by one. Okay? So, you can just add batch normalization in each layer. And if I if you want if you don't want, that's up to you. So, depending upon what will work for that, you can try and find out. Trial and error. So, this is the code. So, after that we'll compile the network. I won't train it, so I'll just print the model architecture. So, I'll tell you one more thing here. So, this is the model that we trained. So, we have the input layer 1 3 4 4. So, if you see here, there's a 64, there's a batch normalization. So, how many parameters are there in the batch normalization? 64 neurons are there. This is the new number of neurons will be there. And there are two states, means into four. So, there are four parameters in this layer. What are the four parameters? Your mu sigma squared that is for the current for the mini batch plus gamma and beta. These are the learnable parameters. Again, batch normalization two. So, you have 32 and 128. Similarly, you have 32 into four. Mu and sigma squared that is coming from mini batch and gamma and beta. So, four parameters in each batch normalization. For each layer, whatever is the number of neurons, for each neuron there will be mu and sigma squared and gamma and beta. For each batch, mu and sigma squared and gamma and beta for the layer because each layer has a batch normalization layer. So, four four parameters each. I'll tell you one more thing. So, if you see there are non-trainable parameters that is 192. So, there Earlier you have seen that all the parameters are trainable, but what are these non-trainable parameters there? So, if you come back here, here if you see there are like 256 means 128 mu and sigma squared and 128 gamma and beta. Similarly, here there are 64 mu and sigma squared and 64 gamma and beta 32 into two, 32 34 into two like that. So, one So, this gamma and beta will be learned. But this mu and sigma is fixed for a mini batch, right? For each mini batch their mean and variance will be fixed. So, for this 64 neuron layer 128 parameters which are coming from mu and sigma, that will not be trainable because that is fixed for the mini batch. And similarly, for this 64, so 128 plus 64 parameters will be non-trainable that is 192 parameters. I hope you get the point. Okay? So, this is what it means. So, that's all for this batch normalization. Next video we have a very very important concept called back prop back propagation. Very important to understand. It will be applicable throughout deep learning. Very foundational concept. So, I'll tell you to revise different differentiation. I have that in my calculus video in the maths playlist. So, you can check that. So, this is all differentiation. Just revise that. Very simple concept and we'll understand this in detail in the next video. So, that's all for batch normalization. Thanks for watching this video. Have a great day ahead and bye for now.