Backpropagation in CNN - PART 2

what's going on everyone this is Jay Patel and welcome back to the part two of the back propagation in CNN now in the part one of the video we saw a back propagation for a small unit which was just the convolution operation as you can see in here now in this video we will be looking at the pack propagation for the entire CNN architecture in part one of the video what we did is that we take an input X and convolved it with the filter K and added a bias B to get this output Z now and we also did the back propagation for this unit and we obtained DL by d k DL by D B using the DL by d z that we had and if you haven't watched the part one of the video then I will then I will link that video down in the description box or you can also check that out by clicking the upper IE button and if you new to this channel consider subscribing because I upload new machine learning videos like this every week in which I provide a mathematical details behind every machine learning model and then we also Implement that in Python and there are very few resources online which teaches you back propagation for CNN most of the courses and the resources out there just implemented using tensor flow and it does not teaches the mathematics behind the back propagation in CNN but I have made a video on it and and that's what I do in this channel I provide mathematical details behind models and we study them carefully and we understand it properly so make sure to hit the red subscribe button so that you do not miss out on such great videos when I upload next time and also it took me a lot of effort to actually make this video so please do hit the like button show some support so that I get motivated to make more such videos and let's not wait further let's get riding with this video let's first see how our CNN architecture looks like this is the first layer of our CNN in this we take this input image and convolved it with the filter K now here I'm assuming that we are only using one filter but in a normal CNN or in or in any other network we actually use many such filters but for the simple explanation of the back propagation here I'm just taking one filter and the exact same process is done for the other filters after convolving this input image with the filter K1 we add a bias B to obtain Z1 which is this node then on this Z1 we perform Rel operation to obtain C1 which is this equation then we perform the average pooling operation to get the P1 then in the second layer we take this P1 as an input image conver it with the filter K2 add a bias B to obtain this Z2 and then perform the Rel operation then Max pulling operation to obtain this P2 then in the third layer what we do is that we flatten this P2 image now this P2 might be a multi-dimensional vector we flatten it to a single dimensional Vector which is represented by F then this F is given as the input to the third layer which is the fully connected layer here I'm only taking one fully connected layer fc3 which has a single node here and that single node has sigmoid activation function which gives us the vibrate as our output now once we have our by which are the predictions we can obtain the cost function for a sigmoid activation function or a binary classification we can use this cost function and you might already be familiar with the loss loss is just the error for the single observation which is this thing now in this video for keeping the Simplicity I will be dealing with only one input image thus we will be dealing with loss and not the cost our parameters here is W W3 B3 K2 B2 and K1 B1 so we need to update the parameters so that they can take appropriate value to minimize the error and to get the better predictions so to update these parameters we need to find DL by dw3 DL by db3 and similarly DL by D K2 DL by D B2 and then DL by D K1 DL by D B1 so our goal in this video is to obtain these gradients so as to update these weight parameters and we will obtain these by propagating backwards and once we have these gradients we will be just applying these WAP updation equations where the W3 will be updated by W3 minus of alpha which is learning rate time DL by D W3 and similarly all these will be updated okay so let's start by deriving these gradients and as we are propagating backwards we will first start with the third layer now the forward propagation for the third layer looks something like this and if you have watched my playlist on the neural Network or a Ann then you might have seen my video on obtaining the back propagation forn now now everything from this layer to the end is same as that we had in the Ann because these fully connected layers can also be represented as the layers that we had Inn thus this part can be just represented by Ann and in the Ann we had obtain these equations for the back propagation if you haven't watched my video on obtaining the back propagation for the neural network then I will link that video down in the description box please do watch that video If you haven't in that video we had obtained these gradients which will be same as what we are going to have in this CNN as well thus for the back propagation equation for the CNN will come out to be exactly this here we are just dealing with one example so here DZ will be y minus of Y here this was ypr and this Y is our actual labels then in that video the previous layer was represented by A2 but here the previous layer input is represented by F which is our flattened input thus here instead of A2 we will be having F here and db3 will be same as this dz3 for a single observation it will be just dz3 but for M number of observations it will be a summation of all those M observations so obtaining this D L by dw3 and D L by db3 is very simple now it's time to move a step backward and find the gradient of this F which is D L by d f now in my enn video you might have seen that I have obtained this dz2 and this dz2 was nothing but the product of D A2 and this term here A2 was the input but in our case f is the input right so DF is obtained same as this da2 which was nothing but W3 transpose multiplied with dz3 and here this is the matrix multiplication so we are done with the third layer and it's time to move to our second layer now in the second layer we first need to find DL by D P2 then moving backward we need to find DL by D C2 then DL by D Z2 and once we have this DL by D Z2 we can obtain DL by D K2 and DL by D B2 because in the part one of the video you might have seen that we had this DL by d z term and then input X and these K and B so once we have this D L by d z term we know how to obtain d l by d k and D L by D B so let's first start with finding d l by d p now this can be obtained from DL by d f that we had in the previous slide now F here is just the flatten of P2 right so all the values in the P2 and the F will be the exact same the only difference will be of the dimension so DL by D P2 can be obtained by just reshaping this D by d f into the shape of P2 so the equation for dp2 which is d by D P2 can be given by this where we just need to reshape DF into the shape of P2 once we obtain DL by D P2 we can obtain d l by D C2 but hey hey wait here there's one challenge here here we are performing the max pooling operation so how does the back propagation works for Max poing let's try to find that out to understand that let's take a hypothetical example where we have this C2 given by this 1 2 3 4 thus the P2 which is just the max poing operation of this C2 will be just four because the max value of this 1 2 3 4 is 4 and let's say while moving backward hypothetically we might come up with DL by D P2 as 2 so what will be this DL by D C2 to answer this question let us examine this thing carefully if we keep this as the maximum value any change in these three values will not affect P2 and thus will not affect our final y bread and thus it will not affect the loss if these term change it is not affecting our loss which means that our d l by d c for these three terms will be zero thus our d l by D C2 will actually come out to be this way how because it is because any change here will be reflected as it is in this P thus any change in this will be reflected as it is in this it is like having yal to X where dely by D X is 1 which means that if this two number changes by let's say x * this two number will also change by x times th the final equation for DL by D C MN for any value MN will actually be written by this here in hypothetical example I had only taken C2 of 2 cross2 Matrix but let's say if the C was of 4 cross 4 dimension and the filter size f is 2 cross 2 Dimension then this P will be of the dimension 2 CR2 and thus DL by D2 2 will also be of the dimension 2 CR2 this D L by D C2 will also be of 4 CR 4 dimensional Matrix which will contain all these values of this DL by D P2 at a position where this C2 had the maximum value which means that there will be four such numbers in a 4 CR 4 dimensional Matrix which will have a value at a certain position where this C2 had the max maximum value and all the rest will be zero now I know that writing this way is becoming a mess but but I wanted to explain you so that you get better Clarity so with the help of that we have also obtained our d l by D C which was here now it's time to move a step backward and obtain DL by d z now for d l by d z we already know this DL by D C and we know that C2 is the Rel operation of Z2 so this can be obtained by Chain rule multiplication of d l by D C2 and D C2 by D Z2 and we can obtain this D C2 by D Z2 with this equation the graph of relu looks like this so the derivative of this graph will be given by this where this will take one value for all the positive input and zero for all the negative input which means that if our Z C2 let's say is given by the these hypothetical values then our D C2 by D Z2 will be a matrix which will have one at a position where Z2 had a positive value and zero at a position where Z2 had a negative value and its equation can be represented with the help of this and thus d l by D Z2 which is D Z2 can be obtained by multiplying d l by D C2 which was this term and D C by D Z2 which is this o so we have finally obtained DL by D Z2 now we know from the part one of the video that once we have this DL by d z term and we know the input X and the filter K and B we can easily obtain DL by d k and D by DB from the part one of the video we know that our DK2 or DL by D K2 can be obtained by by performing the convolution operation between this input image and this dz2 which is this term right here so we can obtain DK with the help of this equation and similarly we can obtain DB with the help of this equation obtaining this was the most challenging part and that's why I had explained it in the first part of the video so that the understanding back propagation for the entire architecture becomes quite simple and with the help of this we are also done with with our second layer and now I think you might already have a proper Clarity on what is going to happen in the first layer so now we will move on to our first layer which is layer one for layer one again we need to find DL by D P1 DL by D C1 DL by D Z1 and once we have this D by D Z1 we know how to obtain DL by d k and DL by D P1 now the question is but we don't have DL by DP right so how will we obtain this DL by D P1 you might have known that in the part one of the video I had explained deriving DL by D X well X was the input here instead of X we have P1 as the input so this DL by D P1 is same as this DL by D X that we had in the part one of the video and if you remember the equation of that came out to be the convolution operation between the padded DZ term and with the 180° rotated filter K obtaining this was also very challenging part right but thanks that we had already covered that in our first video now once we have our d by D P2 let's move backward to obtain d by D C1 now here we have average pooling operation we already saw how the how it would look like for the max pooling operation let's see the similar for the average pulling operation let's say hypothetically we have the same C1 which is given by 1 2 3 4 value thus the P1 will be obtain by the average of these four values which will be 2.5 let's say hypothetically again we have d l by D P1 as two now our challenge is to obtain DL by D C1 so what will that be now see here that any change in these value will affect and thus it will affect byad and thus it will affect loss and any change in these values will be actually equally distributed among all these values what do I mean by that is that if if we have a filter size of 2 CR2 any change in this value will be affected by 1/4 times here because the contribution of these numbers is 1/4 times to this number thus any change here will also affect 1/ 14 times to this D by D C th the DL by d c will came out to be this Matrix which is nothing but 1x4 * of 2 so whenever we have an average pooling operation the D by d c will be obtained by taking the 1/4 of the value at L by d p now again here I took an simple example of 2 cross2 dimensional C but if this was a 4 cross four dimensional then this would have been 2 cross two dimensional and this would have been two cross two dimensional and this would have again be a four cross four dimensional which will have four such 2 cross two dimensional matrices and the value of these would have been 14 times of the value at this Matrix now I know I explained it roughly but I think you understand that right these four values will be 1/4 times of this two like and these four values would be the 1/4 times of the number at this position and similarly for this and this and this and this thus we have obtained DL by D Z2 whatever I explained here can in short be written in the with the help of this equation now we have our d l by D C let's move move one more step backward and obtain DL by d z now you know the deal that DL by d z can be D by D Z1 can be obtained by Chain rule multiplication of d l by D C1 with d C1 by D Z1 and previously we had already obtained d c by d z which was this thus it is thus this term will be multiplied with DL by D C which we already obtained here to obtain DL by d z which will be written with the help of this equation and again you know the deal that once we have this DL by d z we can obtain DL by d k with the help of the convolution operation between the input image this in with the dz1 and this DB can be obtained by summing the DZ Matrix so now the entire summary of this video can be summed up with the help of these many complicated equations but still let's try to summarize that these equations are obtained same as that we saw Inn if we had multiple fully connected layers the equations would have been similarly again obtained as we had that Inn once we have all the fly connected layers we move to this these convolutional and Max pulling layers and here we saw how the gradients can be obtained when there was a Max pulling layer and in the part one of the video we already obtained this D by d k then we saw then we also saw how our gradient will look like when we have our average pooling layer and so on and once we have all these dks and DBS we just can update the weights by using a for Loop and iterating it iterating these equations for many number of times unless these parameter takes appropriate values now if we had multiple filters here these equation would have been the exact same just that the dimension of this DK would have increased oh okay so after a lot of complex mathematics we finally saw how the back propagation for convolutional neural network looks like I hope you found this video helpful if so please do hit the like button if you have any suggestions then let me know that down in the comments below show your support in the comment section and also by liking this video and in the next video we will start with our tensorflow Series so I hope to see you there thank you so much and I will see you in the next video