Saddle Points - Why Gradient Descent Doesn't Get Stuck

The classic worry about gradient descent is local minima — that training will settle in a shallow valley and stop. But modern networks have hundreds of millions of parameters and still train just fine. So either we're getting absurdly lucky, or the story is wrong. The real picture is about saddle points. In one dimension, every critical point is a minimum or a maximum. In two dimensions, a third shape appears: x² − y² has zero gradient at the origin, but it's a cup along one axis and a hill along the other. The Hessian's mixed signs are the fingerprint of a saddle. In high dimensions, a critical point is a true minimum only if all n eigenvalues agree in sign — and the chance of that vanishes exponentially with n. The typical critical point is overwhelmingly a saddle. And saddles are not traps: any step off-center gives you a downhill direction, so SGD noise and momentum carry you off the ridge. Training a network is mostly a tour through saddles, and the few minima you actually reach turn out to be roughly comparable in quality. *Related Videos* ▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ Backpropagation is Just the Chain Rule: https://youtu.be/VCGlYxGJZ04 Newton's Method is Just Tangent Lines: https://youtu.be/8uO7ZOZ9JGE What the Determinant Actually Measures: https://youtu.be/KRPyY76faRs PCA is Just Eigenvectors of the Covariance Matrix: https://youtu.be/MNbJTYnYN0E Fisher Information is Just Curvature: https://youtu.be/pYyot2-KRyI Jensen's Inequality - Why the Average of a Curve is Not the Curve of the Average: https://youtu.be/R9oq9f_XdqI Maximum A Posteriori (MAP) - Why L2 Regularization is Bayesian in Disguise: https://youtu.be/Ovuszur2uzk Activation Functions in Neural Networks - Explained: https://youtu.be/slp222E_0d4 *Contents* ▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 00:00 - The Old Story 00:44 - Where the Story Came From 01:16 - A Third Shape 02:03 - What Happens in High Dimensions 03:01 - The Escape 03:40 - The Real Picture *Follow Me* ▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 🐦 X: @datamlistic http