L13.5 Cross-correlation vs. Convolution (Old)

Key Takeaways

Yes. So, this video is not going to be terribly important. It's just a little bit of nitpicking in case one of you has been wondering, maybe one or more have been taking a signal processing class in previous semesters and were wondering why do we call the procedure convolution and not crossorrelation. So, in fact actually convolutional networks are performing something called a crossorrelation. So, what's the difference between convolution and crossorrelation? So yeah, crossorrelation or convolution is another one of these jargon terms. So in deep learning um we sometimes use words from other fields that have a different meaning and convolution would be one of those. So the convolution in deep learning is actually what is more commonly known as crossorrelation. So the crossorrelation is essentially the sliding of the dot product over the image. So when we have this one here so this region and we um multiply the pixels in that region with the weight the filter right so this is essentially a dot product right so we we wrote it as as follows right when we do this calculation but now so when we yeah write it down more concretely so we can write this procedure down as a crossorrelation. So in in in essence it's computed like this where we have these two sums. So we could actually um so the dot product is if we would actually yeah reshape that into two vectors right. So if we have this one let's um write it like this. And let's say my first row is green, my second row is blue, and my red last one is red. This is for let's say the weight matrix um W. And I have the same one for the inputs. So let's do yellow, maybe purple and black here. So in order to do the dot product I could just reshape them as vector like this right and then for the other one like this and then maybe taking the transpose and then compute um the dot product between the two. But I could also, of course, instead of doing it like this, I could do it the other way around, right? I could do it I could have the vector like this. It's just vice versa. And then if I multiply this vector with the same x, it gives me a different result. Right? Also, should be like this. So this one and this one they will give me different results. So that is mainly the difference between crossorrelation and convolution. The way we we do the multiplication in which order. So um here this is like the way we do the crossorrelation. It's maybe hard to stare at this formula. So let me um illustrate how this works. So essentially is think of the index zero as the one in the center. So if this is your let's say your um weight matrix and um you multiply each one with a corresponding value in the image. So how you do that is um you consider index i as the center point and then the negative indices. So the one minus one would be the left uh upper left corner and one one would be the lower right corner. And then you multiply this weight with the first. So say we have our let's say this is w and this is x here where this whole thing is w. So we multiply this one with this one, this one with this one, this one with this one. So these indices correspond to yeah the position in the input essentially. So that's how we do the crossorrelation and this is essentially how how we did that in the previous videos. But you can also do it the other way around. So this is known as convolution. Notice the only difference is that I'm flipping here some signs. But again, it's probably hard to stare at this equation or formula. So it's probably easier to see it here in this image. So again, this is our w here. And then we multiply it with the values in x. So in X we have still the same um the same order but now so let's say 1 2 3 4 5 6 7 8 9 but now we multiply this one with the first position and then this one with a second one and so forth. So really the difference is you can see that it's inverted. So here in crossorrelation we go like this and in convolution we would go like this. That's the main difference. Um if you don't believe me I just also have an example here just using code to just in a practical way show you that this is indeed the case. So here to demonstrate this I'm initializing a torch tensor just some arbitrary values and then I'm initializing the convolution I'm setting the bias to zero because let's not worry about the bias here it's just distracting here is the weight matrix corresponding to the convolution the 3x3 we have a 3x3 kernel and when I then apply this so first of all I have to um what I'm doing here is I'm um converting it to numpy. So I can use the crossorrelation function and the convolution functions implemented in scypi because it expects numpy arrays. So yeah, I'm just um creating a numpy placeholder that I'm going to use and you can see this is just here the pytorch weight inside the convolutional operator and here this is the numpy version. It's exactly the same values. It's just a little bit more precision because I think this is 30 bit precision and this is 64-bit precision. But you can see um the values are exactly the same. Now when I call this conf PyTorch the conf PyTorch again this is my confd operation. It's a 3x3 kernel applied to a 3x3 input. So what I will get is a single value. the single value from applying PyTorch convolution is this minus 1.1027. So what is this? I'm just reshaping it because usually in a neural network what we have is we have the number of batches and the number of channels. Here it doesn't matter. We are just looking at a matrix. So I I'm setting them to one because otherwise this function will complain. In any case, so the value I get is min -1.1027. Now if I use the crossorrelation function here from cypi, you can see I get exactly the same value. Of course, okay, we have 64 bit precision here. So there are few more digits, but you can see overall it's also minus 1.102. So it's um exactly the same value. So the convolution in PyTorch is the same as the crossorrelation in Cypi and vice versa. So this is the same as above. If we look at the real convolution, if we use the convolution implemented in Cypi, you can see what we would get is minus um 0.26. And if I want to get that result in PyTorch, what I would have to do is I have to either um invert or not invert but rearrange the weights or the input. It doesn't really matter. So here you see I'm rearranging it here. I have the 3.3. So I'm I'm kind of creating the matrix backward. You can see 3.3 is here now at the top 3.2 and the 3.2 here. So I'm just putting in a different order. And then I also get the same result as with this convolution. So in fact the convolution in PyTorch is a crossorrelation. So why does it even matter? I mean I was just highlighting the difference between the two operations. Yeah, in practice it does not really matter at least not in deep learning. So maybe in traditional or in other fields of computer vision and signal processing, it's actually useful to have this associative property of the real convolution. But in deep learning to be honest um that doesn't really matter. And I think crossorrelation is easier to implement especially for the backward pass. But we don't worry about the backward pass. I will show you well explain a little bit about the back propagation in the next video. But that is usually automatically handled in PyTorch. It's maybe like a small efficiency reason. Uh yeah, but then when we or if we implement crossorrelation for these networks, why do we call them convolutional networks? Why don't we call them crossorrelational networks? Yeah, my theory is just that crossorrelational network sounds a little bit weird and convolutional network sounds slightly cooler. So that's maybe also one aspect about the whole thing. Anyways, so this video was not really important. I was just highlighting the difference in case someone was wondering. In the next video, I want to briefly go over CNN's and back propagation, but we [clears throat] won't go into any mathematical details. So don't be afraid.