Backpropagation in 5 Minutes (tutorial)

Key Takeaways

Hello world. It's Suriraj and today we're going to learn how back propagation works in five minutes. Gradient descent is a popular optimization technique and can be used in many different types of machine learning models. It's used to optimize or improve the accuracy of our models predictions. One implementation of it that is particularly popular is for neural networks. A neural network is a learning model that can make predictions. We give it some input data. The x values represent the input and the y values represent the expected output labels. Our network's job is to learn this mapping so that given some arbitrary input, it can correctly predict its output label. So our three-layer neural network will first have an input layer. Each neuron represents a different row from our input data. Then it has a hidden layer. Data will flow in one direction from our input layer to our output. And the way it does this is by having weights that connect each neuron in one layer to every neuron in the next layer. We can initialize these weights as matrices with random values to start off with. We'll multiply each row of the input by each column of our weight matrix. The resulting values from this operation results in our hidden neuron values. We'll take each of those values and convert them to a probability value. That is a value between 0 and 1 by applying an activation function to it. The type we'll use in this example is a sigmoid. So each neuron receives a set of inputs, performs a dotproduct, and then applies an activation function to it. We'll just repeat this same process again to calculate the output prediction. We compute the dotproduct of the hidden layer neurons and the next weight matrix between the hidden layer and the output layer. Then we once again apply our activation function to it. This resulting value is our prediction. And this process that we just completed is called forward propagation. If we compare this to our expected output, we'll see that our prediction is incorrect. We want to find the absolute best weight values that given any input, they would help calculate the correct output. To do this, we'll first want to calculate an error value. We want to minimize this error. If we were to create a simple graph of the error value versus some random weight from our network, it would look like this. Too small a weight value and our error is high, but too big and our error becomes high. Again, we want an optimal value for each weight in our network where our error is smallest. Starting at some random weight value, we want to take a step in the direction towards the minimum error. This direction is the opposite to the gradient. If we take many steps descending down the gradient, eventually the weight will find the minimum of the error. We call this process gradient descent. So, how do we do this? Well, we'll need to use calculus. Let's do a little refresher on three terms from calculus we'll need to know. The derivative is a term that means the slope of the tangent line to a curve at a specific point. It measures the rate of change of a function. A derivative of a function f ofx gives you another function that returns the slope of f ofx at a point x. For example, the derivative of x^2 is 2x. So at x= 2, the slope is 4. A partial derivative of a function of several variables is its derivative with respect to one of those variables with the others held constant. And the chain rule is the process we can use to compute derivatives of composite functions. A composite function is a function of other functions. That is we might have one function that is composed of multiple inner or nested functions. Say you have some function f ofx and another function g ofx. Using them you form some composite function f of g ofx. The chain rule states that the derivative of f of g ofx is equal to the derivative of g ofx times the derivative of f ofx. A neural network is essentially a massive nested composite function. Each layer of a feed forward neural network can be represented as a single function whose inputs are a weight vector and the outputs of the previous layer. The purpose of back propagation is to figure out the partial derivatives of our error function with respect to each individual weight in the network. So we can use those in gradient descent. It gives us a way of computing the error for every layer and then relating those errors to the quantity of real interest, the partial derivative with respect to any weight in the network. We can use the chain rule to compute the partial derivatives that is the gradient of the error with respect to each weight. Back propagation at its core simply consists of repeatedly applying the chain rule through all the possible paths in our network. Our ultimate goal in training a neural network is to find the gradient of each weight with respect to the output. We do this so that we can update the weights incrementally using gradient descent. We reuse multiple values as we compute the updates for weights that appear earlier and earlier in the network. After we have the error for the output layer, we calculate an error for each neuron in the hidden layers going backwards layer by layer. The error for a neuron in a hidden layer is the sum of the products between the errors of the neurons in the next layer and the weights of the connections to those neurons multiplied by the derivative of the activation function. We will use those errors to calculate the variation of the weights as a result of the current input pattern and ideal outputs. The variation or delta of a weight is the product of the input neuron output value with the error of the output neuron for that connection. This process is repeated for all the input patterns and the deltas are accumulated. At the end of a learning iteration, we change the actual weights with the accumulated deltas for all the training patterns and we multiply it with a learning rate which states how fast a network converges to a result. When we run our code, we can see this process in action as our prediction gradually increases in accuracy. Please subscribe. And for now, I've got to go derive the meaning of life.