Main Types of Gradient Descent | Batch, Stochastic and Mini-Batch Explained! | Which One to Choose?

Key Takeaways

hi this video will cover the three main types of gradient descent if you haven't watched our previous video do that now to have a better understanding of the algorithm batch gradient descent uses the entire data set to compute the gradient and update the parameters meaning you update the parameters once for the complete data in each Epoch stochastic gradient descent on the other hand takes one random observation from the data for the parameter update thus if you have 5,000 observations in the data set parameters will be updated 5,000 times during each epic mini batch gradient descent is the mix of the previous two it takes a userdefined number of observations as a random sample for updating the parameters called batch size batch size is commonly taken as a power of two batch gradient descent provides an accurate estimate of the gradient so the error confidently decreases however often it finds a local minimum since batch gradient descent minimizes the air collectively for all observations in the data set it can follow a deterministic path if you are close to a local Minima the algorithm would probably converge to it thus batch gradient descent is problematic for non-convex functions Additionally you will need huge resources to fit large data sets in the memory during stochastic gradient descent an update favoring one observation can lead to higher errors for other observations thus we expect the error function to sometimes increase temporarily additionally random selections can help the algorithm avoid local Minima which seems a better approach for non-convex functions due to frequent updates it potentially converges faster while due to the noisy updates it may never converge exactly to the minimum but can hover around it mini batch gradient descent is the most common type it has some Randomness to help avoid being stuck at local Minima and better approximates the gradient of the entire data set in practice batch size is a hyperparameter that should be configured for each separate task experiment with all to see which one works better for your problem if you want to learn more about artificial intelligence subscribe to our channel to be aware of the new videos press the like button and let's discuss AI in the comments section