Gradient Checking Implementation Notes (C2W1L14)

Key Takeaways

in the last video you learned about gradient checking in this video I want to share you some practical tips or some notes on how to actually go about implementing this for your neural network first don't use graduating training or me to debug so what I mean is that computing D theta or procs eyes all the values of ID is a very slow computation so to implement gradient descent use backprop to compute D theta and just use back prop to compute the derivative and as only when you're debugging that you would compute this to Mitchell as close to D theta but once you've done that then you would turn off the grant check and don't run this during every iteration being a sentence it's just much too slow second if the never fails brag check look at the components look at the individual components to try to identify the bug so what I mean by that is is the faith in aprox is very far from DSA so what I would do is look at the different values of I to see which are the values of D theta aprox they're really very different than the values of D theta so for example um if you find that the values of theta or D theta they're very far off all corresponding to D BL for some layer or for some layers but the components for DW are quite close right remember different components of theta correspond to different components of B P and W but you find this is the case then maybe you find that some the bug is in how you're computing DP the derivative respect to parameters B and then similarly vice versa we find that the values they're very far you know the values from D theta aprox that are very far from D theta and you find that all those components came from GW or from GW and certain layer then that might help you hone in on the location of the bug there doesn't always let you identify the bug right away but sometimes it helps you give you some guesses about other where they track down the bug next um when doing grad check remember your regularization term if you're using regularization so if your cost function is J of theta equals 1 over m sum of your losses um and then plus this regularization term right some of the hell of wll Frobenius norm squared then this is the definition of J and you should have that D theta is gradients of J or respect to theta including the regularization term so just remember to include that term next Grouch egg doesn't work with dropouts because in every iteration dropout is randomly eliminating different subsets or the fit in humans there isn't a easy to compute cost function J the dropout is doing gradient descent on it turns out that dropout can be viewed as optimizing some cost function J but its cost function J is defined by summing over all exponentially large subsets of nodes they could eliminate in any iteration so the cost function J is very difficult to compute menu just sampling the cost function every time you live in a different random subset and military use gravel so it's difficult to use grad chair to double-check your computation with dropouts so what I usually do is implement grad check without dropout so you if you want in set key prop in dropout to be equal to 1.0 and then turn on dropout and hope that my implementation of dropout was correct there are some other things you could do like fix the pattern of nose dropped and verify that grad check for that a pattern of unis killed off is correct but in practice I don't usually do that so my recommendation is turn off dropout use drag check to double-check that your algorithm is at least correct without dropout and then turn on dropout so finally this is the subtlety it is not impossible rarely happens with not impossible that your implementation of gradient descent is correct when W and B are close to zero so at random initialization but that as you run grain descent and W and B become bigger maybe your implementation of back prop is correct only when W and B is close to 0 but it gives more inaccurate when W and B become large so one thing you could do I don't do this very often but one thing you could do is run drag check your randomness elevation and then train the network for a while so the wmb had some time to wonder away from zero from the small random initial values and then run drat check again after you've trained for some number of innovations so that's it so gradient checking and congratulations are coming to the end of this week's materials in this week you learned about how to set up your trained jab intersect how to analyze bias and variance and what things to do if you have high bias and Siberians versus maybe high by 9 high variance you also saw how to apply different forms of regularization like l2 regularization and drop on your neural network so some tricks for speeding up the training video network and then finally gradient checking so I think you've seen a lot in this week and you get to exercise all these ideas in this week's program exercise so best of luck exact and I look forward to seeing you in the week 2 materials