Vectorizing Logistic Regression's Gradient Computation (C1W2L14)

Key Takeaways

in the previous video you saw how you can use vectorization to compute the predictions the lowercase A's for an entire training set all sort of at the same time in this video you see how you can use vectorization to also perform the gradient computations for all M training examples again all sort of at the same time and then at the end of this video we'll pull all together and show how you can derive a very efficient implementation of logistic regression so you remember that for the gradient computation what we did was we computed DZ one for the first example riskily a one minus y one and then DZ 2 equals a 2 minus y 2 and so on so on and so on for all my training examples so what we're going to do is define a new variable D capital Z is going to be DZ 1 DZ 2 right DZ m again all the D lowercase D variables stacked horizontally so this would be a 1 by M matrix or alternative VA M dimensional real vector now recall that from the previous slide we'd already figured out how to compute capital a which was this a 1 through am and we had defined Capital y as y 1 through ym also you know stacked horizontally so based on these definitions um maybe you could see for yourself that DZ can be computed as just a minus y because this is going to be equal to now a 1 minus y1 through the first element a2 minus y2 in the second element and so on and so and so this first elements a 1 minus y1 is exactly the definition of DZ 1 the second element is exactly definitely Z 2 and so on so with just one line of code you can compute all of this at the same time now in the previous implementation we've gotten rid of one for loop already but we still had this second for loop over training examples so we initialize DW to zero to the vector of zeros but then we still have to loop over training examples where we have PW plus equals x1 times d z1 for the first training example DW plus equals x2 DZ 2 and so on so we do the M times and then you know DW 2y equals by M and similarly for B write DP was initialized as 0 and then DB plus equals V Z 1 DP plus equals D Z 2 down to you know DZ m and then D be divided equally M so that's what we had in the previous implementation we'd already gotten rid of one for loop so le is now DW is a vector and we went separately you know updating DW 1 DW 2 and so on so we've gotten rid of that already but we still have a for loop over the m examples in the training set so let's think these operations in vectorize them here's what we can do for the vectorized implementation of DB what is doing is basically summing up all of these disease and then dividing by M so DB is basically 1 over m sum from I equals 1 through m of DZ I and well all the disease are in that a row vector and so in Python what you do is in Flynn you know 1 over m times n P dot sum of D Z right so just take the variable and call the NP but some function on it and that would give you DB how about DW I'll just write out the correct equations you can verify the right thing to do DW x will be one of M times the matrix x times DZ transpose and to kind of see why that's the case this is equal to one of M then the matrix X is X 1 through X M right stacked up in columns like that and DZ transpose is going to be DZ one down to DZ m like so and so if you figure out what this matrix times this vector works out to be this turns out to be one over m times X 1 DZ 1 plus dot dot dot plus XM DZ m right and so this is a n by 1 vector and this is what you actually end up with what DW because DW was taking these you know x ID zi and adding them up and so that's what exactly this matrix vector multiplication is doing and so again with one line of code you can compute G W so the vectorized implementation of the difference calculations is just this you use this line to implement DB and use this line to influence DW and notice that without a for loop over the training sites you can now compute the updates you want to your parameters so now let's put all together into how you would actually implement logistic regression so this is our original highly inefficient non vectorized implementation where um so the first thing we've done in the previous video was get rid of this volume right so instead of looping over DW 1 DW 2 and so on we have replaced this with a vector value DW and just say this is DW + equals X I which is now a vector times DZ I but now we'll see that we can also get rid of not just a for loop below but also get rid of this volume so here's how you do it so using what we had from the previous slides you would say Capital Z is equal to W transpose X plus B and code you write is capital Z equals NP dot W transpose X plus B and then a equals sigmoid of capital T so you've now computes it all of this and all of this for all the values of iron mix on the previous line we said you would compute DZ equals capital A minus capital y so now you compute all of this for all the values of I then finally DW equals 1 over M X D Z transpose and DB equals one of em you know and P dot some DZ so you've just done for propagation and back propagation really computing the predictions and computing the derivatives on all M training examples without using a for loop and so the gradient descent update then would be you know W gets updated as W minus the learning rate times DW is just computed above and B is updated as B minus a learning rate times DB so sometimes it's really cool inserted it notices on assignment but I guess I haven't been totally consistent to that notation but with this you have just implemented a single iteration of gradient descent for logistic regression now I know I said that we should get rid of explicit folders whenever you can but if you want to implement multiple iterations of gradient descent then you still need a for loop over the number of iterations so you if you want to have a thousand iterations of gradient you might still need a full loop over the iteration number because and also most folders like that and I don't think there's any way to get rid of that for loop but I do think it's incredibly cool that you can implement at least one iteration of gradient descent without needing to use a full loop so that's it you now have a highly vectorize and highly efficient implementation of gradient descent or register direction there's just some one more detail that I want to talk about in the next video which is in our description here I briefly alluded to this technique called broadcasting broadcasting turns out to be a technique that pison an umpire allows you to use to make certain parts of your code also much more efficient so let's see what some more details of broadcasting in the next video