Learning Rate Decay (C2W2L09)

Key Takeaways

one of the things that might help speed up your learning algorithm is to slowly reduce your learning rate over time we call this learning rate decay let's see how you can implement this let's start - an example of why you might want to implement learning rate decay suppose you're implementing mini batch gradient descent with a reasonably small mini batch maybe a mini batch has just 64 128 examples then as you iterate your steps will be a little bit noisy and it will tend toward this minimum over here but it won't exactly converge but your algorithm might just end up wandering around and never really converge because you're using some fixed value for alpha and there's just some noise in your different mini batches but if you were to slowly reduce your learning rate alpha then during the initial phases while your learning rate alpha still lasts you can still have it to be fast learning but then as alpha gets smaller your steps you take would be slower and smaller and so you end up oscillating in a tighter region around this minimum rather than one ring far away even as training goes on and on so the intuition behind slowly reducing alpha is that maybe during the initial steps of learning you can afford to take much bigger steps but then as learning approaches convergence then having a slower learning rate allows you to take smaller steps so here's how you can implement learning rate decay recall that one epoch is one class through the data right so if you have them a training set as follows maybe break it up into different mini batches then once the first pass through the training set is called the first epoch and then the second pass is the second epoch and so on so one thing you could do is set your learning rate alpha to be equal to one over one plus a per hour originally called the decay rate times the epoch num and there's going to be times some initial learning rate alpha zero note that the decay rate here it becomes another hyper parameter which you might need to tune so here's a concrete example um if you take several epochs so several passes through your data if alpha zero is equal to zero point two and the decay rate is equal to one then doing your first epoch alpha will be 1 over 1 plus 1 times alpha 0 so your learning rate will be zero point one that's just your evaluating this formula when the decay rate is equal to 1 and the epochal on this one on the second you pop your learning rate the case to 0.67 on the third 0.5 on the fourth 0.4 and so on fearful evaluate well these values yourself and get a sense that you know as a function of your epoch number your learning rate gradually decreases whereas this according to this formula up on top so if you wish to use learning rate decay what you can do is try to provide your values of both hyper parameter alpha 0 as well as of this decay rate hyper parameter and then try to find a value that works well other than this formula for learning rate decay there are a few other ways that people use for example this is called exponential decay where alpha is equal to some number less than 1 such as 0.9 5 times epoch num times alpha 0 so this will exponentially quickly decay your learning rate other formulas that people use are things like alpha equals some constant over EPOC numb square root times alpha zero or some constants cave another hyper counter over dr.mini Bosch number P square root 2 times alpha zero and sometimes you also see people use a learning rate that decreases and discrete stats where for some number of steps you have some learning rate and then after a while you decrease it by one half after a while by one half after a while by one half and so this is a discrete staircase so so far we've talked about some using some you know formula to govern how alpha the learning rate changes over time one other thing that people sometimes do is nanyo decay and so if you're training just one model at a time and the dual model takes many hours or even many days to Train what some people will do is just wash your model as this training over your a large number of days and then annually say oh it looks like the learning rate slowed down I'm going to decrease out for a little bit of course this works this manually controlling alpha really tuning alpha by hand all by hour day by day this works only if you're training only a small number of models but sometimes people do that as well so now you have a few more options so how to control the learning rate alpha now in case you're thinking wow this is a lot of hyper parameters how that select amongst all these different options I would say don't worry about it for now in next week we'll talk more about how to systematically choose hyper parameters for me I would say that learning rate is usually lower down or the list of things I try setting alpha just a fixed value of alpha and getting that to be wealthy and has a huge in time learning rate decay does help sometimes it can really help speed up training but it is a little bit lower down my list when in terms of the things I would try but next we want to talk about hyper parameter tuning you see more systematic ways to organize all of these hyper parameters and how to efficiently search amongst them so that's it for learning rate is hey um finally I want to also want to talk a little bit about local optimal and saddle points in new networks so you can have a little bit better intuition about the types of optimization problems your optimization algorithm is trying to solve when you're trying to train these in your network let's go onto the next video to see that