RMSprop Optimizer Explained in Detail | Deep Learning

rms prop is an optimization algorithm which speed ups the training of our model if you already know about mini batch gradient descent then you will know that in the mini batch gradient descent the weight updation takes place in this zigzag manner and what happens is that a lot of times get wasted in moving in the vertical direction and taking the zigzag path instead of just moving directly straight towards the local minima so optimization algorithms are those algorithms which speed up the learning of the model and make this path much more straighter instead of taking the zigzag path and one of such algorithm is aramis prop which works very well rms prop stands for root mean square propagation and that is what we are going to understand in this video and in the previous video i also talked about one of such optimizer algorithm which is known as momentum or mini batch korean descent with momentum or stochastic gradient descent with momentum make sure to watch that video out first because that video will give you a much better understanding of how the optimization works in deep learning it also gives you an idea about exponentially weighted moving average which we are going to use in this video as well in rms prop as well so make sure to check that out video first if you haven't already i will provide its link down in the description box or you can find it slick by clicking on the upper i button and if you are new to this channel consider subscribing because i upload new machine learning videos just like this every week in which i provide mathematical details behind every single model as well as the intuition behind those so hit the red subscribe button also the bell icon and without further ado let's get started with this video we know that the weight updation in the gradient descent is given by these two equations now in rms prop instead of this dw what we do is that we take dw divided by square root of sdw and similarly dv divided by square root of sdb here this sdw is given by the exponential weighted average of dw square which will be equal to beta times of sdw previous plus 1 minus of beta multiplied by d w squared and similarly s d b will be equal to beta times of s d p previous plus 1 minus beta dv square now let's see what will happen by dividing this term with square root of sdw now in the mini batch gradient descent as the steps are like this zigzag path what is happening is that it is taking a big movement in the vertical direction while a very little movement in the horizontal direction let's say the vertical direction is of the direction of w and the horizontal direction is that of direction b here i am taking here i am assuming that w is a single parameter and not of matrix of parameters and similarly b is a single parameter and not a matrix of the parameters so what will happen is that as this movement in the vertical direction is higher this d w is going to be higher and thus the sdw is going to be higher and as we are dividing it with the square root of sdw which is a higher this overall quantity is going to be lower thus the w is going to be updated by a smaller quantity thus this vertical movement will reduce and as the movement in the b direction is smaller this db is going to be smaller thus sdw is going to be smaller and as we are dividing it with its square root this entire quantity is going to be higher thus the movement in the horizontal direction is going to be higher and thus the net movement is going to be much more straighter and much more larger towards the local minima thus we are going to have a faster training of our model and as we are taking the square of dw and then taking its root it this algorithm is called root mean square propagation now there is one more thing to note here it is possible that the value of the either sdw or sdb can go much much smaller and if its value is highly smaller and or very close to zero then what will happen is that we will be dividing it with the 0 and thus this entire value will shoot up very highly and thus we might be encountering the problem of overshooting thus to overcome that what we do is that we add this term epsilon to this and this and the value of the epsilon will be taken as 10 power minus 8. so we add this epsilon so that the entire value does not go very small and thus we don't end up overshooting also the value of this hyper parameter beta is usually taken as 0.999 so with these two chain so we can implement rms prop with just these two changes and it will speed up the training of our model now we need not to implement these from scratch but we can use a programming frameworks like tensorflow and pi torch with the help of that we just need to mention that we want to use rms prop and pass the value of the beta and the epsilon in it and it will automatically implement it for us in the background so i hope that i made this concept of rms prop clear to you if so please hit the like button and share this video among your friends who can also benefit from this and in the next video we will look at another such optimization algorithm which is even powerful than the root mean square propagation and it is called atom optimization atom optimization is formed by combining rms prop as well as the momentum so we will look at that optimization in the next video so i will see you in the next one