Adam Optimization Algorithm (C2W2L08)

Key Takeaways

during the history of deep learning many researchers including some very well-known researchers sometimes proposed optimization algorithms and show their work well in a few problems but those optimization algorithms subsequently will show not to really generalize that well to the wide range of neural networks you might want to train so over time I think the deep learning community actually develops some amount of skepticism about new optimization algorithms and a lot of people felt that you know gradient descent with momentum really works well is difficult to propose things that work much better so rmsprop and the atom optimization algorithm which to talk about in this video is one of those rare algorithms that has really stood up and has been shown to work well across a wide range of deep learning architectures so there's only algorithms that what it hesitate to recommend you try because many people have tried it and seen it work well on many problems and the atom optimization algorithm is basically taking momentum and rmsprop and putting them together so let's see how that works to implement atom you would initialize vgw equals 0 s DW equals 0 and similarly V DB s DB equals 0 and then on iteration T you would compute the reserve it is compute V WB be using current mini-batch so usually you do this with mini-batch gradient descent and then you do the momentum exponentially weighted average so VT w equals beta but now I'm going to call this beta 1 to distinguish it from the hyper parameter beta 2 we'll use for the RMS portion of this so this is exactly what we had when we're implementing momentum except that I've now called the hyper parameter beta 1 instead of beta and similarly you have V DB as follows 1 minus beta 1 x DB and then you do the rmsprop like updates as well so now you have a different hyper parameter beta 2 plus 1 minus beta 2 DW squared again the squaring there is element wise squaring of your derivatives BW and then s DB is equal to this plus 1 minus beta 2 times DB so this is the momentum like update with hyper param 2 beta 1 and this is the rmsprop like updating with hyper parameter beta 2 in the typical implementation of atom you do implement bias correction so you can f be perfected corrected means after bias correction DW equals v DW divided by 1 minus beta 1 so power of T if you've done T iterations and similarly B DB corrected equals V DV divided by 1 minus beta 1 to the power of T and then similarly you implement this on bias correction on s as well so that's s DW divided by 1 minus beta 2 to the T and s DB corrected equals s DB divided by 1 1 is beta 2 to the T finally you perform the update so W gets updated as W minus alpha at times so if you're just implementing momentum you'd use v DW or maybe VG w corrected but now we add in the rmsprop portion of this so we're also going to divide by square root of s DW corrected plus Epsilon and similarly B gets updated as similar formula the DP directed divided by square root s corrected DB plus Epsilon and so this algorithm combines the effect of gradient descent with momentum together with gradient descent of rmsprop and this is a commonly used learning algorithm that's proven to be very effective for many different neural networks of a very wide variety of architectures so this algorithm has a number of hyper parameters the learning rate hyper parameter alpha is still important and usually needs to be tuned so you just have to try range of values and see what works a comment or is really the default choice for beta 1 is 0.9 so this is the moving average wrote an average of DWI this is the momentum light term the high parameter for beta 2 the authors of the Adam paper inventors the Adams album recommend 0.99 induces computing the moving weighted average of DW squared as well as DP squared and then epsilon the choice of epsilon doesn't matter very much but the authors of the advent paper recommended 10 to the minus 8 but this parameter you really don't need to set it and it doesn't affect performance much at all but when implementing atom what people usually do is just use the default values of beta 1 and beta 2 as well as Epsilon I don't think anyone ever really choose epsilon and then try a range of values of alpha to see what works best you can also tune beta 1 and beta 2 but it's not done that often among the practitioners I know so where does the term atom come from atom stands for adaptive moment estimation so beta 1 is computing the mean of the derivatives this is called the first moment and beta 2 is used compute exponentially weighted average of the squares and that's called the second moment so that gives rise to named adaptive moment estimation but everyone just calls it the atom also invention and by the way one of my long-term friends and collaborators is called atom codes far as I know this algorithm doesn't have anything to do with him except for the fact that I think he uses it sometimes but sometimes I get off that question so just in case you're wondering so that's it for the atom optimization algorithm with it I think you really train your neural networks much more quickly but before we wrap up for this week let's keep talking about hyper parameter tuning as was getting some more intuitions about what the optimization problem from your networks look like in the next video we'll talk about learning rate decay