The Problem of Local Optima (C2W3L10)

Key Takeaways

in the early days of deep learning people used to worry a lot about the optimization algorithm getting stuck in bad local optima but as the theory of deep learning has advanced our understanding of local optima is also changing let me show you how we now think about local optima and problems in the optimization problem in deep learning so this was a picture people used to have in mind when they worried about local optima maybe you're trying to optimize some set of parameters and we call them W 1 and W 2 and the height of the surface is the cost function so in this picture it looks like there are a lot of local optima you know in in all those places and it'd be easy for gradient descents or one of the other algorithms to get stuck on a local optimum rather than find this way to a global optimum it turns out that if you are plotting a figure like this in two dimensions then it's easy to create plots like this of a lot of different local optima and these very low dimensional plots used to gather intuition but this intuition isn't actually correct it turns out if you create in your network most points of 0 gradients are not local optima like points like this instead most points of 0 gradients in the cost function are actually saddle points so that's a point with a zero gradient again this is maybe W 1 W 2 and the highest heightens the value of the cost function J but informally a function in a very high dimensional space if the gradient is 0 then in each direction it can either be a convex light function or a concave light function and if you are in say a 20,000 dimensional space then thread to be a local optima all 20,000 directions need to look like this and so the chance of that happening is maybe very small you know maybe 2 to the minus 20000 instead you're much more likely to get some directions where the curve bends up like so as well some directions where the function is bending down rather than have them all Bend upwards so that's why in very high dimensional spaces you're actually much more likely to run into a saddle points like that shown on the right then local optimum oh and as for why the surface is called a saddle point if you can picture maybe this is a sort of shadow you put on a horse right so maybe if this is a horse I guess there's a head of a horse as you I have a horse you know I guess and right well another great drawing of a horse but you get the idea then you the rider will sit here in the saddle so then so that's why this point here where the derivative is zero that point is called a saddle point it's really the point to understand where you're sitting s and that happens to have you know derivative zero and so one of the lessons we learned in history of deep learning is that a lot of our intuitions about low dimensional spaces like what you can plot on the left they really don't transfer to the very high dimensional spaces then our learning algorithms are operating over because if you have twenty thousand parameters then J is V a function over a twenty thousand dimensional vector and you're much more likely to see saddle points than local optimum if local optima aren't a problem then what is a problem it turns out that plateaus can really slow down learning and the plateau is a region where the derivative is close to zero for a long time so if you are here then gradient descent will move down the surface and because the gradient is zero or near zero the surface is quite flat you can actually take a very long time you know to slowly find your way to maybe this point on the plateau and then because of a random perturbation to the left or right maybe then finally I'm gonna switch pen colors for clarity your algorithm can then find this way off the plateau but then to take this very long slope off before it's found this way here and they could get off this plateau so the takeaways from this video are first you actually pretty unlikely to get stuck in bad local optima so long as you're training and reasonably launched new network save a lot of parameters and the cost function J is defined over a relatively high dimensional space but second that plateaus are a problem and they can actually make learning pretty slow and this is where algorithms like momentum or our most proper atom can really help you learning algorithm as well and these are scenarios where more sophisticated optimization algorithms such as atom can actually speed up the rate at which you could move down the plateau and then get off the plateau so because your networks are solving optimization problems over such high dimensional spaces to be honest I don't think anyone has great intuitions about what these spaces really look like and our understanding of them is still evolving but I hope this gives you some better intuition about the challenges that the optimization algorithms may face so that congratulations on coming to the end of this week's content please take a look at this week's quiz as well as the exercise I hope you enjoyed practicing some of these ideas with this week's forum exercise and I look forward to seeing you at the start of next week's videos