How to Accelerate Optimization in Deep Learning | Issues with Non Convex Loss | Learning Rate Decay

Hello and welcome. I hope you are having a great day. In this video, we'll understand various optimization methods used to accelerate the optimization process in deep learning. In deep learning, we know that there are multiple layers. In machine learning, we had a single layer. In deep learning, we have multiple layers. So, just training one layer was was taking so much time. So, training multiple layers and that to with so many neurons will take so much time. So optimization means the gradient descent thing you remember right. So there are multiple reasons why this will be very slow and how can we accelerate that in deep learning. So let's begin. So in the context of neural network training usually the since the loss function the surface that is non-convex that is it does not have one single global optima and we use gradient descent and gradient descent. Since there is if it is convex function we know that there will be one global optima and we can reach there but in non-convex there can be multiple local optima and one global optima and we don't know okay so that is very difficult to find the optimum so what we care is we don't want to reach that global optima okay the lowest point the minima okay the global minima instead even if we are able to obtain a good enough generation error a good enough local optima that will be fine for us so that's the target and since in deep learning it will take so much time. So is there a way we can accelerate the gradient descent? So that's the whole point of all the optimization techniques the techniques that you know right so momentum based RMS propom all these we'll discuss in the coming videos. So how can we accelerate the whole process to reach that local optima which is good enough for getting a model which generalizes well. So that's the whole intent of all the optimization techniques and why because the loss surface is non-convex. I'll show you some non-convex surfaces from a real networks. So you see this non-convex surface. So there's one global optima here. But you see there are so many local optima here, here, here you can see even this is local optima. This is a local optima. So it will depend upon okay even on this. So there's one local optima here. There will be one local optima here somewhere down. Okay. Even in this so there are so many local optima here and there. So depending upon where you start if I start from here say from for example if I start here then I'll reach this point this will be my local optimum I will not reach this this will be here the the surface will become flat and I'll tell that okay I have reached but is it good enough okay so it will depend upon the place where I initialize if I start from here I will reach in this local optimum and I'll be stuck here okay if I start here I'll go just go in this and I'll be stuck here so depending upon my starting point I'll be reaching I can if I re if I start from here then I'll just go down in the global optimizer so depending upon where I start and that's a random start and there are other various methods to get the initialization he initialization or zel initialization so that that techniques are there but even with that we can reach local optima which may not be so optimum because of this non-convex nature that's the whole point I'm what I want to stress here that the reaching the global optima is not the point But this non-convexity will cause problems. And even if you reach a good enough, say for example, even if I reach this local optima, I'm good. This will give me a good generation error. Okay. And the model will be working fine. So depending upon the loss etc. We can see how it functions. So I may not be going to the global optim but it will be good enough for me. Okay. So this is the kind of it is not. So in radians we have seen that we got a simple convex surface and we wanted to reach here below the global was a simple a simple curve. So this is a convex function. So yeah it was good no problem but in such a surface so you see such it is so tricky it is like a mountain range. So that is what causes problems. It will cause a lot of problems and how can we accelerate this cuz say for example so what's the another point? Say for example if I'm starting in this region this region is so flat the gradient is like almost invisible there's no gradient itself here very shallow here if you start here this gradient is so steep okay here also if you start here the gradient is so steep but in these areas where the gradient is so shallow this is not changing it is almost flat these regions these are so flat these are flat if my if I start from here the gradient won't change itself so these are the problem if I start from here the gradient won't change for a long time then I'll go here. So how do I get rid how do I accelerate this? Because if the gradient is not changing here much and I'm I'm depending dependent upon my gradient learning rate times my learning rate times the gradient rate this is the old so how will I this is very very small it is like approximately zero how will I move that won't the weight won't change itself I won't come down itself so how do I accelerate this this is the whole point okay so let's see so I hope the problem statement It is clear what is the intent of all this optimization process what is what do we need to accelerate all these flat regions these are like if I'm stuck here I'll be moving very slowly can I accelerate this moving in this flat region can I accelerate this somehow that's the whole point of all these kinds of optimization techniques that we are going to discuss let's move on so whatever I have discussed I'll just put it formally so in gradient descent we know right we take all the points together and then after taking all the points we'll calculate the gradient of all the points and then we move to the next point. That was like batch gradient descent as we know right the batch gradient descent the gradient we take the complete but in deep learning you know we have very huge data set millions of data points so we cannot calculate for calculating the gradient for such a big data set and just taking one step that will make the process even slower so that's the whole idea we go for stoastic gradient descent stoastic means just taking a random point or in this case a random mini batch okay so what we do is instead of taking the whole point it can be accelerated a small subset randomly. So we take many batches randomly from the data set and then we do gradient descent on that. So calculating on a subset say for example I have millions of data and from that if I if my batch size let's say 256 okay so I'll just take 256 uh data points and I calculate the gradient on that that will be faster and then I move move okay so that's what so randomly I'll choose 256 data points and I'll calculate the gradient on that those data points okay so this is your and then average of gradient of randomly chosen m data points out of n say if n is equal to 10 ^ 6 m will be say 256 only or 32 say if I want even smaller it will be in the power sub 2 we'll discuss that also why because of GPU and where my is the learning rate so be like 0.01 01 or something. Okay. So first thing is we go for stoastic with many batches random and then so there's one more thing we do learning rate decay in normal. So to accelerate this say for example if I'm this is my gradient descent you know so here here will move very fast but as we reach near the minima this is the minima right as we reach here the grid becomes very very small movement will be very the changes will be very small right so if I continue with the same learning rate so here the gradient will be high so I'll make take big big steps but if I continue with that same learning rate there may chances that I can overshoot the minimum if this is my minimum. So what we do is we decay the learning rate. So we start with the big learning rate that we have. Say for example I start with 0.1 cuz initially I know that I'll be on some top point and I want to move faster down the gradient. But as I come close to the local the optima the minima right I want to slow down because the gradient will be very small. This gradient will become very small. So the learning rate also I want to take small small steps. when I'm at the top I can move fast but when I come towards the bottom I want to take small steps. So that step size is dependent on my learning rate. So what do we do generally in practice is that we decay the we remove the so we decay the learning rate. Okay. So we decay how do we do this? So this is a linear decay. So the learning rate at kth step this k is the kth step is given by this 1 - alpha into n. N not is the initial learning rate that we start with. So initially I'll start with big step size 0.1 and gradually I'll move to 0.01 and say my tow number of iterations I I'm doing is 100 and after that it will remain at 0.01 itself. Okay. So this is a linear. So say for example on in the very first step so the k is equal to 1. Say for example k is equal. So what will be my at first step? So alpha we how we calculate alpha. So for toao iteration these are the number of iterations. So k is equal to 1 and toao is 100. So this will be 0.01. So what will be ea 1? This will be 1 minus 0.01 into ea. So it will be very close to the initial learning rate only. And the this one for at what is the desired this will be 0.01. So initial learning rate will be very close to the the starting learning rate itself. Okay. So it will be very close to this and as K increases so if K is equal to 100 at 100 step. So K is equal to 100. So alpha will become one now. So this 1 - alpha this is 1 - 1 will become zero and now it will become 1 into 0.01. So it will start from 0. It will come start from 0.1 and gradually decay to 0.01. What is the desired at the 100th iteration? I hope you get the math. This is simple. Okay. At kth iteration 1 2 gradually at one it will be very close to beta not and at alpha or at the 100th step at tth iteration 100th iteration it will become close to it t such kind of combination is called convex combination. I I'll discuss in detail when I come to that later. But generally these values will be ranging between it not to these are the two variables which in which these values will uh move these are the bounds of these two values depending upon the value of alpha I'll explain sometime later okay but these are called convex combinations and you will see a lot during this complete optimization techniques all of every place you will see this same combination this convex combination okay so if this learning rate decay so this is also a practical thing that we do so that We don't miss we don't overshoot the whatever optimize there local local minimize there so that we don't have to come back that will increase our iterations unnecessarily and after reaching the desired small step size we let it be constant that now moves slowly slowly slowly so let me reintroduce you to some terminologies that we have discussed but I'll put it formally now say we have a sample size of n we have n sample size n is equal to say 10 ^ 6 million samples in my data and I want I have to choose a mini batch of m less than n size say for example m is equal to 32 that is my mini batch size 32 samples in one bit so one iteration is weight update after computing the gradient of one mini batch so for one mini batch randomly selected for all 32 data points I'll calculate the gradient and that will be my one iteration one iteration means one weight update one time I'll update the weight w new is equal to W old minus EA* the gradient that I calculated for one mini batch so that is one iteration and now one epoch you will listen it a lot epoch right one epoch is one complete pass through the entire training data set that will be n by m iterations so if it is like 10 ^ 6 is my complete so if I have 32 in one mini batch so how many such mini batches to be created 10 ^ 6 by 32 32 so I have to cover all the data sets okay all the samples in my data set so once I have covered all the data points in my data set then I say that that is one epoch that at least I visited all the data points one once so that is your one epoch and l epo or multiple times of epoch is like l into number of times into n by m iterations so that many iterations so that's the relation between basically the relationship between epoch and iteration. So n by m iterations and n is the total data point and m is the number of samples in the mini batch. Okay, I hope this is clear. The size m for mini batch is decided based on the computing resources at hand. The size of the GPU, RAM, how much you have the RAM of the GPU. Okay. So the GPU is RAM also or in Google it is tensor processing TPU. What is the RAM of that? So how much my memory does the GPU have to do calculations. So for example my H00 GPU has 80 GB RAM. So depending upon that that I can decide my mini batch size. So each each data so there will be n samples and each data will have say k d um dimensions are there and each dimension will be an integer say for example h so depending upon the dimension each row will have some size and depending upon that I can see how many rows can I load in my RAM right so each one one row will have say for example one row is say um 1 MB So how many I can load depending upon this. Say for example if I can in the multiples of two say if one row is even bigger if it is 100 MB then and I have say 2 GB RAM then I can load only 20 such mini batches right 120 will be 2,000 like that approximately. So that's the whole idea. How much is the RAM and what is the size of my one row of the data one sample. Okay. So in practice the mini B size is chosen to be the largest possible power of two. It will be in the powers of two because of the GPU. The GPU RAM will be in the power of two. we can that many number of coursees will be there in the GPU also that fits within the available GPU memory or RAM and all the things should be will be used for computation and still allowing for good model performance. So apart from the matrix calculation which will be done on the GPU, there are other things also that needs to be computed. Okay. So that memory also we'll have to leave. Say for the system to run and the program applications to run that memory should be left aside. Even for the model other things to train on the model. So those things for that we'll have to leave that and so that it performs well and apart from that how much max memory that is remaining on that the in power of two. So generally the batch size will be like 32 64 like that is 768 we see like that. So that's the size of my mini batch and depending upon the ramat I hope that is clear. So these are terminologies you should know okay and the samples in the mini batch are randomized in every epoch. So every epoch we will randomize the samples also within that we'll shuffle all the all the samples. Okay. So I hope the terms are clear. iteration epoch and power of two for GPU. It will depend on the GPU RAM by mini bath size. Okay, let's move. So what are some issues with the non-convex surface that we have seen because of that the optimization will need some special handling so that it can accelerate. So law is it is nonconvex we have discussed that and finding for finding a local minima is difficult because of the because of multiple reasons. First thing is there are multiple local minimum. We have seen that in the diagram there are multiple local minimum. And then there are saddle points. Saddle point is like maximum. It is maximum with respect to one dimension and minimum with respect to one dimension. So we'll reach this. So we'll say that okay I've reached a minima but it actually it's a maxima with respect to the other dimension. So it is not a local minima basically. So it's a subtle point. You'll be stuck there. Okay. So here it will because the gradient will become zero. Maxima or minimum what is there? The gradient will become zero. It will become flat. The gradient is zero. At the maximum or minima it is maximum. Minimum maximum gradient become zero. Right? That's the thing since point also it will be the gradient will become zero but it will be maximum with respect to one dimension and minimum with respect to one dimension. So it will not be a local minima. So that's the point we have to get past that point so that we reach a local minimum h and then there are flat regions as we saw. So if there's a region like this, so we have a surface like this. We have surface like this and then it goes down. So goes down like this. Say for example H. So if we have a flat surface here like this almost flat. So gradient is very very low. So traversing this will take a lot of time. How can we accelerate this? So these are my flat regions or kind of plateaus. How do I get past this fastly? That's the momentum approach we'll discuss. And then different parameters. So there may be parameters which are like some parameters may be sparse. Okay. Only some data points may have and other parame parameters are frequent. So sparse parameters for example if I'm predicting housing prices and there is a feature like uh near near airport or near a mall. So very few houses will have this feature enabled one or 11 one in that and most of the features most of the houses will not have this feature near airport or near a mall. So that's what I mean. So these features these will this will require a different learning rate because they'll be updated very because these features since there is no entry itself. So the the value will be zero. So it will not be updated at all. So they will need other features which have all the entries are filled they will move fastly and this will not be changing fastly right. So handling of such parameter this past parameters need special handling and that we'll discuss in Adam and before that add also. So these are the problems major problems that will that will stop the acceleration or that will slow down my acceleration so that I reach a local optima local minimum. Okay. So what are so this let me revise saddle point. So this is saddle point if you remember. So I've taken this lecture from the maths from calculus part. So here if you see with respect to y here there is a maxima at this point this graph is the maxima and if you see with respect to x here this is a minima here this a graph is like this. So that is a saddle point. It's like a horse saddle which we use to we can we can sit on this right. This is put on a horseback and someone a person can sit like this here. The legs can be like this and horse head is somewhere here. Okay. So this is like a horse saddle. That's why it's called a saddle point. Hope this is clear. So here you will get stuck. We have to get past this. So what are the optimization techniques famous that we use? So now we use Adam only but this is the evolution that happened over a period of time. Each one added one new feature and then finally currently we are using ad optimizer that is kind of default in the industry. So we start with momentum based that was that was initially invented in 1964 but that was improved in 2013 by SSK. He is a student of Jeffrey Hinton father of AI. Okay. And then adaptive gradient that came in 2011 that this was for uh that sparse parameters that I talked about. How do we handle sparse parameters because they will not be so frequent. So the weight updates how do we handle that? Then RMS prop this was an advancement on adagrad because uh there was some issues it caused something called vanishing learning rate problem at a grad. So that was solved in RMS prop. And then finally came Adam which clubbed both the things. So it club momentum based plus RMS prop and the features of both of them were present in Adam and it is the current best optimization technique which is the fastest. So this came in 2014. So this is how it all developed. So we'll discuss all these one by one in four separate videos and we'll build one will build on top of the other. So this is momentum based is there. So the same thing this logic is used in Adam also had a grid and RMS prop is an advancement of that and all these this momentum plus RMS prop is used in Adam. So this is the logical advancement. So we'll discuss this in separate videos one by one. So next video we'll discuss we'll start with momentum based optimization technique. Okay. So that's all for this video. Thanks for watching this video. Have a great day ahead and bye for now.