Optimizers in Neural Networks - EXPLAINED!

Key Takeaways

greetings fellow Learners welcome to another video in a playlist of videos on the fundamentals of deep learning and neural networks but before we get started with this wonderful world of optimizers I have a thought-provoking question for you how do you usually learn new things are you a visual learner where you prefer visuals to understand Concepts or are you more of a Hands-On learner where getting your hands dirty would be the best way to learn for you or is there another method that you prefer as a learning mode please comment down below your thoughts on this and let's have a discussion now this video is going to be divided into three passes where in the first pass we're going to start with an overview of optimizers themselves then go into their types and some other details too so let's get to it this here is a simple feed forward neural network now we want this neural network to take in an image and be able to classify it as either a dog or not dog and so the neural network needs to learn specifically these Edge weights or weight parameters need to be learned let's illustrate a specific optimization algorithm called stochastic gradient descent so first we pass an image into the network then the network makes a prediction of whether the image is either a dog or not this network will determine like a probability value that's between 0o and one we compare this to the ground truth which is going to be one since it's a dog we'll pass both of these prediction and the ground truth into a loss function to generate a loss for this classification phase this loss function could be a cross entropy loss now we take this loss and back propagate through the network in order to compute the gradient of the loss with respect to the individual parameters of the network for more details on this back propagation process check out this video on back propagation we use the gradients computed in the optimization algorithm in the case of stochastic gradient descent we update each weight parameter using this formula this here is one iteration of weight updates and we repeat the process hundreds or thousands of times during the entirety of the training process then during the inference phase we will pass an unseen image to the network and ideally it will be able to make the correct prediction of whether it is a dog or not a dog squeeze time have you been paying attention let's quiz you to find out what is the difference between back propagation and an Optimizer a back propagation computes gradients while optimizers compute updated model parameters B back propagation computes updated model parameters and the optimizer computes gradients C back propagation and optimizers are interchangeable terms describing the same process or D back propagation optimizers are unrelated processes in the neural network training comment your answer down below and let's have a discussion and at this point if you do think I deserve it please do give this video a like because it will help me out a lot that'll do it for quiz one and pass one of the explanation but I'll be back so keep paying attention optimizers Define how neural networks learn this here is a non-exhaustive list of optimization algorithms and while it seems that there's so many of them they're all actually kind of related to each other so let's talk talk about each of them a little briefly for example sarcastic gradient descent this is exactly what we described in the previous pass so for every single input label pair we will update label parameters step by step now the concept of momentum helps speed up learning and this is done by making larger learning jumps for common examples and smaller learning jumps for outlier examples now in the nestra accelerated gradient or nag method this is the equivalent of stochastic gradient descent plus momentum plus a concept called acceleration acceleration ensures that the algorithm better anticipates the direction of the gradient for stable training next is adagrad this is an Adaptive learning rate algorithm and this means that the learning rate can be different for different parameters in the out ofr case the learning rate depends on the gradient so in this case the parameters that have large gradients will have a smaller learning rate but parameters with smaller gradients will have a larger learning rate but there's a little bit of a problem with this here the denominator is a sum of squared gradients over time which can increase to an arbitrarily large number and so the overall learning rate will become close to zero to combat this two optimizers were developed independently of each other one of them was adad Delta and the other was RMS prop both have slight modifications to ensure that this denominator does not explode over time and this allows stable convergence now next is adom Adam is another adaptive learning rate algorithm that does RMS prop but also adds some momentum it performs well on many problems and it's one of the most common op optimizers that we see today and then we have natam which is a combination of the nestra accelerated gradient and atom and it offers fast convergence and good General performance so for more information on each of these Optimizer algorithms and how they interplay with each other digging into the math so you understand why things happen the way they do I recommend checking out this video on optimizers quiz time [Music] it's that time of video again have you been paying attention let's quiz you to find out why is Adam used over stochastic gradient descent in practice a Adam is computationally more efficient and has fewer hyperparameters B Adam is less prone to overfitting compared to stochastic gradient descent C adom generally converges faster and requires less less manual tuning or D Adam has a more straightforward optimization landscape comment your answer down below and let's have a discussion now that'll do it for quiz time for now but do keep paying attention because I will be back to quiz you for this pass I wanted to get a fun take on the types of optimizers and so I created a short video that we can watch together hey want to hike Sunday sure where to L Canyon oh the usual yeah it's a fun downward hike to the lowest point I don't know I kind of struggle to find the end well maybe you should take baby steps like me but you take forever y'all do know there is a middle ground right true and we can even run down some parts since their trajectory is obvious but for another perspective it would be better to go off Trail and adapt to your surroundings is that fun more degrees of freedom oh nice so you're the best one here nah he tends to slow down when the hike gets long I'm more adaptive and faster so you're the best one here cute and you are at a Delta but faster are you the best hi there's no end to this no you know what keeps me up at night what knowing that if we all started a hike together we'd end up alone and some of us may never be found never reaching the bottom so Sunday squeeze time okay this is going to be a fun one what are the primary roles of the loss function and Optimizer in the training of neural networks a the loss function computes gradients while the optimizer quantifies the model's performance B the optimizer determines the model's architecture while the loss function guides parameter updates during training C the loss function quantifies the model's performance while the optimized adjusts the model parameters during training or D the loss function is responsible for hyperparameter tuning while the optimizer computes model predictions comment your answer down below and let's have a discussion and once again if you do think I deserve it please do consider giving this video a like and it'll help me out a lot now that's going to do it for the final quiz and pass three of this explanation and before we go though let's generate a summary back propagation is used to compute the gradient of floss with respect to each parameter of the neural network optimizers Define how we use this gradient to actually update the parameters there are many types of optimization algorithms and each algorithm overcomes some shortcoming of a preceding algorithm now that's all we have for today thank you all so much for watching and like I mentioned before if you do like this video please do consider giving it a like if you're interested in getting more information on optimizers in a more relatable way and you're not afraid of some math do check out this video where we get into some details thank you all so much once again and I will see you in the next one bye-bye