1D convolution for neural networks, part 5: Backpropagation

having equations for convolution is a great start they are specific enough that we can turn around and implement them in code the other thing that we have to be able to do in a neural network that uses back propagation to learn is we have to be able to differentiate these equations specifically with respect to the error gradient the lost gradient that's going to be passed back down through the layers if these concepts are unfamiliar there are some links in the text up above that you can use to get familiar with them for now I'm going to assume that they're not totally new and kind of jump in back propagation is how we take the sensitivity of the loss function to changes in each of our layers output values so this is the partial derivative of the loss with respect to the outputs Y and we want to be able to propagate that back and calculate the partial derivative of the loss with respect to the inputs X this is one link in our chain and we back propagate this sensitivity of the loss to each set of outputs and inputs all the way back through the network by the chain rule of calculus the way that we get the partial derivative of loss with respect to X is we take the partial derivative of loss with respect to Y and multiply it by the partial derivative of Y with respect to X now in our case x and y are arrays they're not single values they're a whole long line of values the list of values so really what we want to calculate is the partial derivative of the loss with respect to each one of the inputs individually and the way we calculate that is by the chain rule the partial derivative of the loss with respect to Y times the partial derivative of Y with respect to each input element and each of our outputs are so individual values not one element and so to get to really break this down to individual elements we have the partial of loss with respect to each input element is the sum of the partial of loss to each output element times the partial of that output element to the input element added up over all of the output elements so because it's so verbose we'll use as a shorthand for a partial derivative of loss with respect to the output we'll just call that the output gradient and the partial derivative of the loss with respect to the input will call the input gradient so to go from the output gradient to the input gradient we have to know this quantity here the partial derivative of each output element with respect to each input element that's what we mean when we say that the layer is differentiable we have to be able to calculate this bit to be able to do back propagation so in order to do that we can go back to our definition of convolution we can explode it back out and now say each y sub J is equal to X sub J minus P times W sub P etc etc we expand that summation out and we expand out for all of our potential elements J so this is just a longhand way to represent that summation sign and to represent the various elements J then we can take in each of these cases we can take the derivative of that Y sub J with respect to each of the X's that occur in it so in this case the very first element if we take the derivative of Y sub J with respect to X sub J minus P the derivative if J is W sub P the very next term the derivative of Y sub J with respect to X sub J minus P plus one is w sub P plus one and so what we get is a bunch of small partial derivatives of individual output values with respect to individual input values so this is starting to look really good this is the type of thing that we're looking for