Adam Optimizer Explained in Detail | Deep Learning

adam is one of the most powerful and most widely used optimization algorithm in deep learning now you may ask what is optimization if you know about mini batch gradient descent then you know that the weight updation in the mini batch korean descent takes place in the zigzag form so a lot of training time is wasted by moving in the vertical direction instead of just moving straight to the local minima so to reduce the time taken to train the model and to have a much straighter path instead of the zigzag path we use something called optimization optimization as it stands it reduces the time taken to train the model thus optimize the training and one of the best technique or the algorithm that is most widely used in deep learning is called atom optimization and it is formed by combining two another optimization algorithms namely rms prop or root mean square propagation or momentum or mini batch korean descent with momentum so to understand adam you will need to have the understanding of these two techniques or algorithms first and if you don't know about these algorithms or if you haven't watched my videos on these techniques then you can find the link to these down in the description box as well as by clicking on the upper i button so make sure to watch those videos out first before you can understand adam but if you already know about these algorithms then we will understand what is adam in this video and if you are new to this channel consider subscribing because i upload new machine learning videos like this every week in which i provide mathematical details behind every single model as well as the intuition behind how they work so make sure to hit the red subscribe button also in the bell icon and without further ado let's get started with this video if you know about momentum then you will know that the weight updation is given by this equation where we use this vdw here and vdw is given by this and if you know about the rms prop then you will know that the weight updation is given by this equation where sdw here is given by this a small note here as we take dw square here we represent it with sdw and when we don't take the square we represent it with vdw for adam optimizer we combine both the momentum as well as the rms prop into one single equation so the weight updation in atom will be given by combining both the v d w of the momentum and this term from the rms prop into 1 and it will be given by vdw upon or divided by sdw plus epsilon and similarly for b it will be given by v d b upon s d b plus epsilon and as we are having 2 beta here we will be taking this as a beta 1 and we will be taking this as a beta 2 value of the beta 1 is taken as 0.9 usually taken as 0.9 and value of the beta 2 is usually taken as 0.999 and epsilon here is same as as we took it for rms prop which is 10 power minus 8. the full form of adam is adaptive moment estimation and in short it is called adam as the atom is found by combination of two optimization algorithms which are momentum and rms prop it has the advantage of both these optimization algorithms and thus the resultant is better than momentum or rms prop alone thus it is found out that the atom usually works better than momentum and rms prop and thus it is almost the best known optimization algorithm known till now this way by using adam we can get rid of that zigzag path and the resultant path will be much more straighter because this way by using adam we can greatly reduce the time taken to train our model and thus the training will be much more faster also we need not worry about implementing these from scratch but we can use a programming framework like tensorflow or pytorch and in that we just have to mention that we are using adam and and we will need to pass the value of the hyper parameters beta1 and beta2 and it will automatically implement it for us in the background so this was about adam if you found this video helpful then please hit the like button and also share it among your friends so that they can also be benefited from this i am making an entire course on deep learning and machine learning so continue watching my video if you want to learn machine learning so subscribe to this channel hit the bell icon so that you don't miss any update when i upload my next video and i will see you again with another such machine learning or deep learning video