Optimizers - EXPLAINED!

Key Takeaways

you got this believe in yourself Kevin easy easy almost there talk about optimizers optimizers define how neural networks learn they find the values of parameters such that a loss function is at its lowest keep in mind that these optimizers don't know the terrain of the loss so they need to find the bottom of a canyon when line folded essentially let's start with the one the only gradient descent hop hippity hop hop wait too far huh too far again oh come on the original optimizer gradient descent involves taking small steps iteratively until we reach the correct weights theta the problem here is the weight is only updated once after seeing the entire data set so this gradient is typically large theta can only make larger jumps and it may just hover over its optimal value without actually being able to reach it the solution to this update the parameters more frequently like in the case of stochastic gradient descent stochastic gradient descent updates the weights after seeing each data point instead of the entire data set but there's a problem here too you see wait that example was weird no okay easy wait wait easy nope nope no no no oh hell no this may make very noisy jumps that go away from the optimal values it's influenced by every single sample because of this we use mini-batch gradient descent as a compromise updating the parameters only after a few samples huh another way to decrease the noise of stochastic gradient descent is to add the concept of momentum the parameters of a model may have a tendency to change in one direction typically if examples follow a similar pattern with this momentum the model can learn faster by paying little attention to the few examples that throw it off time to time but you might see a problem here bigger bigger bigger do they choosing to blindly ignore samples simply because it isn't typical it may be a costly mistake reflecting in our laws adding an acceleration term though helps your model is training gaining momentum the weights are becoming larger it finds an odd sample because of momentum it thinks very little of it though but discarding it leads to a loss decrease that wasn't as drastic as you thought this is where we decelerate our weight updates the weight updates become smaller again allowing future samples to fine-tune the current model we go big or we go home way will meet the lawsuit decrease as much they thought it would slow down haha not too shabby but this is the loss function for a single predictor using multiple predictors that the learning rate is fixed for every parameter autograph allows an adaptive learning rate for every parameter I'm on a 3d surface plot iran octomorg cool site to plot out equations this is a plot of Z is equal to X square minus y square Z is the value of the loss and this loss has a minimum value of y tending towards negative or positive infinity if I were to start somewhere up here on the saddle point my optimizer would go down in one direction of the y axis like how my cursor is moving with an adaptive loss I have more degrees of freedom to increase my learning rate in the Y direction and decrease it along the x direction in fact this is what we see here adaptive learning rate optimizers are able to learn more along one direction than another hence they can traverse this kind of terrain in the optimizer update the capital gtii is the sum of squares of the gradients with respect to theta i parameter until that point the problem with this is that the G term is monotonically increasing over iterations so the learning rate will decay to a point where the parameter will no longer update and there's no learning we can actually see this effect here for the outer grad point as the iterations go on it learns slower and slower even though the optimal trajectory is quite clear add a delta to the rescue it reduces the influence of past squared gradients by introducing a gamma weight to all of those gradients this reduces their effect by an exponential factor so the denominator doesn't explode and this prevents the learning rate from tanking to zero cool so we actually have learning rate updates for every single parameter well if this is the case why not just go even further and have momentum updates for every parameter and this is what Adam does the only change you need to make from out of Delta to Adam is just add the expected value of past gradients what does it mean it means that we are slow initially but pick up speed over time and this is intuitively similar to momentum as you build up momentum over time in this way Adam can take different size steps for different parameters and with momentum for every parameter it can also lead to faster convergence because of its speed and accuracy I think you can see why Adam can be used as a de-facto optimizer for many projects of course we can go even further introducing acceleration in Adam natum and I could go on it might seem like a ton of optimizers are out there and there are but we've literally just added a term to each algorithm gradually making them capable of more things but with all of these optimizers which is the best one well that depends on the kind of problem that you're trying to solve instant segmentation semantic analysis machine translation image generation so many problems out there with different types of losses the best algorithm is the one that can traverse the loss for that problem pretty well it's more empirical than mathematical I hope this video helps you better understand the role of these optimizers and clear some things up too if you liked the video hit that like click subscribe and also watch some of my other videos on the channel you won't regret it take care