Partial Derivatives explained (Maths Behind AI)

Welcome to Neurom Monk. I am your host and today we are going to unlock one of the most important concepts in all of mathematics and artificial intelligence, the partial derivative. If you have ever trained in neural network, opened a paper on back propagation, or wondered how a model actually learns from its mistakes, the answer always comes back to this single idea. Partial derivatives are the engine inside every AI model that learns. In this video, we will build the concept from scratch. We start with pure intuition, move to the geometry, then the algebra, and finally land at the exact role partial derivatives play in training real neural networks. By the end, you will not just know what a partial derivative is. You will understand why it is unavoidable in AI. Let us start with the simplest possible question. What is a partial derivative? Here is the intuition. Imagine you are standing on a hilly landscape. The height at any point depends on two things. How far east you are, which we call x, and how far north you are, which we call y. So, height equals f ofx and y. A function of two variables. Now, here is the key question. If I take one small step to the east, keeping my north position completely fixed, how steeply does the ground rise or fall? That rate of change, the slope in the x direction, while y is held completely still, is the partial derivative of f with respect to x. We write it as df over dx using the curly d symbol to signal that other variables exist but are being frozen. It is exactly like an ordinary derivative but with one crucial rule. All variables except the one you are differentiating with respect to are treated as constants. Freeze everything. Else slide along one axis. Measure the slope. That is a partial derivative. Let us now compute actual partial derivatives step by step. Our function is f ofx and y = x^2 + 3x y + y cubed. To find the partial derivative with respect to x, we treat y as a plain constant just like the number 3 or 5 and differentiate normally. x^2 becomes 2x. The power rule 3xy since y is a constant becomes 3 y and y cubed which contains no x at all contributes zero. just like any constant would. So the partial derivative of f with respect to x is 2x + 3 y. Now for the partial derivative with respect to y. This time x is the constant. x^2 contributes zero 3xy with x as a constant becomes 3x and y cubed becomes 3 y^2 by the power rule. So the partial of f with respect to y is 3x + 3 y ^2. Let us check at the point x = 1, y = 2. The x partial gives 2 + 6 which is 8. The y partial gives 3 + 12 which is 15. Those are our slopes in each direction at that specific point. Now let us see what partial derivatives actually look like geometrically because this is where the real intuition lives. Any function of two variables f ofx and y defines a surface in three dimensions. Think of a bull, a hill, a saddle, any curved surface floating in space. Every point on that surface has a specific x, a specific y and a height z equal to f ofx and y. To visualize a partial derivative, we slice the surface. We pick a fixed value of y, say y equals 1, and look at the resulting curve where that horizontal plane cuts the surface. We get a two-dimensional curve. Height as a function of x only. The partial derivative of f with respect to x at any point is simply the slope of that curve. The tangent line to this cross-sectional slice. In the same way, if we fix x and let y vary, we get a different slice and the slope of that slice is the partial derivative with respect to y. Two slices, two slopes, same point. That is the geometric picture of partial derivatives. And this idea scales to any number of dimensions including the thousands or millions of weight dimensions inside a neural network. Now, we take the single most important step in this entire video. We collect all the partial derivatives into one object. The gradient vector for a function of two variables. The gradient is simply the vector containing both partial derivatives. The partial with respect to x as the first component and the partial with respect to y as the second. This vector has a remarkable geometric property. It points in the direction of steepest ascent on the surface. Wherever you are standing on the landscape, the gradient vector points uphill and it points in the direction that gains height the fastest. Its magnitude tells you how steep that slope is. A large gradient means a steep hill. A small gradient means nearly flat terrain. For machine learning, this is everything. The loss function is a surface in the space of all model weights. We want to roll downhill on that surface towards smaller loss values. The gradient of the loss with respect to all the weights tells us which direction is uphill. So we move in the opposite direction. Every step of gradient descent is take the gradient, negate it, scale it by the learning rate, update the weights, repeat until convergence. Simple in principle powered entirely by partial derivatives. Now we come to the chain rule. The machinery that makes back propagation possible. Suppose Z is a function of U and V. But U and V are themselves functions of X and Y. We have a chain of dependencies. X and Y affect U and V which in turn affect Z. To find how Z changes when X changes, we have to trace all the paths through the chain. The partial derivative of Z with respect to X equals the partial of Z with respect to U times the partial of U with respect to X plus the partial of Z with respect to V * the partial of V with respect to X. We sum over every intermediate path from X to Z. Every route that X can affect Z through via U, via V, or via any intermediate node contributes one term to the sum. This is exactly what back propagation does. A neural network is a deeply nested chain of functions. The input flows through layer one, then layer two, and so on up to the loss. Back propagation applies the multivariable chain rule starting from the loss and working backwards one layer at a time, multiplying partial derivatives all the way back to the first layer weights. Every single gradient in a trained neural network was computed using the chain rule applied to partial derivatives layer by layer. Let us now see partial derivatives at work inside an actual neural network with concrete notation. Consider a small network with an input layer, one hidden layer and an output neuron. The weight connecting input neuron 1 to hidden neuron 2 is called W12. We want to know how does the loss L change if we nudge W12 by a tiny amount. That question is answered by the partial derivative of L with respect to W12. To compute it, we trace the path from W12 to the loss. Applying the chain rule at each step, the partial of L with respect to W12 equals the partial of L with respect to the output activation times the partial of the output activation with respect to the hidden activation times the partial of the hidden activation with respect to W12. Three partial derivatives multiplied together. That is one way to update. Now scale this up. GPT3 has 175 billion weights. Every training step computes one partial derivative for each of those weights. That is 75 billion partial derivatives per step computed efficiently by the back propagation algorithm using the chain rule. Every AI model you have ever used was trained this way. partial derivatives all the way down. So far we have only taken a partial derivative once but we can go further. We can differentiate again and get second order partial derivatives the second partial of f with respect to x written d^2 f over dx^2 tells us the rate of change of the slope itself the curvature in the x direction. We also have mixed second partial derivatives differentiating first with respect to x and then with respect to y or vice versa. For smooth functions a beautiful theorem due to clara tells us these two are always equal. The order of differentiation does not matter. Collecting all four second order partials into a matrix gives us the hessen. This two by two matrix contains the full curvature information about the function at any point. The hessian tells us things the first order gradient cannot whether a critical point is a local minimum a local maximum or a saddle point. In principle hessenbased methods like Newton's method converge much faster than plane gradient descent in practice computing and storing the hessen for a model with millions of parameters is computationally impossible. So deep learning sticks to first order methods gradient descent and its variance like atom which only use partial derivatives never second order information. There is one more powerful idea we should explore the directional derivative. A partial derivative measures the slope in one of the coordinate directions pure x or pure y. But what if we want the slope in some arbitrary diagonal direction? The directional derivative in the direction of any unit vector u is simply the dotproduct of the gradient with u. This is a remarkable result. It tells us that all directional slopes are completely determined by just two numbers. The two partial derivatives that make up the gradient. As a consequence, the direction that gives the maximum possible slope is exactly the direction of the gradient vector itself. The gradient is always the direction of steepest ascent negated to get the direction of steepest descent which is precisely what gradient descent follows. Now let us look at saddle points a phenomenon that matters a lot in deep learning. A saddle point is a location where all partial derivatives are zero just like a minimum but the function goes up in some directions and down in others. On a two variable saddle shaped like a horse saddle, the function curves upward in the x direction and downward in the y direction. The gradient is zero, but the point is neither a minimum nor a maximum. In highdimensional networks, saddle points are extremely common. Early researchers worried they would trap gradient descent forever. In practice, stochastic gradient descent tens to escape them because the noise in mini batch gradients prevents the optimizer from sitting perfectly still at zero gradient. Let us bring everything together. A partial derivative measures how a function changes when we move in one variables direction while holding all others fixed. It is the slope of a one-dimensional slice through a multivariable surface. Computing one is straightforward. Apply the standard differentiation rules treating all other variables as constants. Power rule, product rule, chain rule, they all apply one variable at a time. The gradient vector collects all partial derivatives into a single object that points in the direction of steepest descent. Gradient descent moves in the opposite direction downhill to minimize the loss function. The chain rule connects partial derivatives through nested functions. This is the foundation of back propagation. The algorithm that computes all the gradients in a neural network in a single backward pass. Second order partial derivatives describe curvature. The hessen matrix collects them all but is too expensive to compute for large models. And the directional derivative, the dotproduct of the gradient with any unit direction tells us the slope in any direction we choose. Every AI model that learns from the simplest logistic regression to the largest language model relies entirely on these ideas. Partial derivatives are not background mathematics. They are the heartbeat of machine intelligence. Thank you for watching Neurom Monk. If this video helped you, please subscribe and share it with someone learning AI. The maths behind AI series continues and I will see you in the next one.