1D convolution for neural networks, part 6: Input gradient

Key Takeaways

our next step is to regroup these so that all of our partials with respect to a particular input value are all grouped together so this top two lines include all of the parcels with respect to X sub J so it's just all of the little small expressions from the previous set of equations just rearranged just reordered but now we're starting to gather up all of the contributions of this partial of Y with respect to an individual X element an individual input and if you look at them you can represent them then with the pattern so we see that the partial of X sub I plus K with respect to X of I is W sub K this is a shorthand way to represent all of the equations here and you can see that the pattern holds for any X sub I for any input element if we gather up all of the partials with respect to it we can take and represent all of those expressions with this shorthand so for any input X sub I the partial of the output X of I plus K is equal to W sub K so that's a neat little way to condense that and something that we're gonna make good use of now we can actually plug this back in to our chain rule where the input gradient is equal to the summation of the output gradient with respect to each of these partials we can then substitute in this expression W sub K we have the input gradient with respect to the output gradient times W sub K so this is a fairly slick way then to do our back propagation there's one more step we can do if we take W sub K and flip it left to right which we're going to represent with this left-handed arrow above it then everything that was minus K becomes plus K so we can change the sign on the K index in our output gradient and everything else stays the same so we just did a little trick by pre flipping this w sub K now this is a sliding dot product so it is an array which is our output gradient and we have this kernel our flipped W sub K and we're summing it over the full length of that kernel and for each value of our input X sub I so then we can represent that even more concisely as our input gradient is our output gradient convolve with the reversed version of our kernel so this is a really slick little result it says that the derivative of a convolution is a convolution with the kernel flipped there's a pleasing symmetry with that math is beautiful exhibit 673 very very slick