Batch Norm At Test Time (C2W3L07)

Key Takeaways

bachelor on processes or data one me me batch at the time but at times you might need to process the examples one at a time let's see how you can adapt your network to do that recall that during training here are the equations you just implement national within a single mini-batches sum over that mini batch of the zi values to compute the mean so here you're just summing over the examples in one mini batch I'm using M to denote the number of examples in the mini batch not not in the whole training set then you compute the variance and then you compute the norm by scaling by the mean and standard deviation what that's on added for numerical stability and then V tilde is taking Z norm and rescaling by gamma and beta so notice that mu and Sigma squared which you need for this scaling calculation are computed on the entire mini value but at times you might not have a mini batch of 64 128 alternative Pacific examples to process at the same time so you need some different way of coming up with mu and Sigma squared and if just one example taking the mean and variance of that one example doesn't make sense so what's actually done in order to apply your neural network at test time is to come up with some separate estimate of mu and Sigma squared and in typical implementations of national what you do is estimate this using a exponentially weighted average where the average is across the mini batches so to be very concrete here's what I mean let's pick some layer L and let's say you're going through mini batches x1 x2 together with the corresponding values of Y and so on so when training on x1 for that layer L you get some new L and in fact I'm going to write this as new for the first mini batch and that lane and then when you train on the second mini batch for that layer and that mean about you and there was some second value of you and then for the third mini batch in this hidden layer you end up with some third value for MU so just as means for how to use the exponentially weighted average to compute the mean of theta1 theta2 theta3 when you are trying to compute a exponentially weighted average of the current temperature you will do that to keep track of so what's the latest average value of this mean vector your seat so that exponentially weighted average becomes your estimate for what the mean of the B's is for that hidden layer and similarly you'd use an exponentially weighted average to keep track of these values of Sigma squared that you see on the first mini batch in that layer Sigma squared then you see on a second mini batch and so on so you keep a running average of the MU and the Sigma square that you're seeing for each layer as you train the neural network across different mini batches then finally at test time what you do is in place of this equation you would just compute Z norm using whatever value you see you have and using your exponentially weighted average of the MU and Sigma squared whatever was the latest value you have to do the scaling here and then you would compute each other on your one test example using that Z norm that we just computed on the left and using the beta and gamma parameters then you'll you have learned during your neural network training process so the takeaway from this is that during training time mu and Sigma squared are computed on an entire mini batch of you know say 64 and June 28 or some number of examples but at test time you might need to process a single example at a time so the way to do that is to estimate mu and Sigma squared from your training and there many ways to do that you couldn't clearly run your whole training set through your final network to get mu and Sigma squared but in practice what people usually do is implement an exponentially weighted average where you just keep track of the new and Sigma squared values you've seen during training and use an exponentially weighted average also sometimes called a running average to just get a rough estimate of mu and Sigma squared and then you use those values of MU and Sigma square that test time to do the scaling you need of the hidden unit values z in practice this process is pretty robust to the exact way you use to estimate mu and Sigma squared so I wouldn't worry too much about exactly how you do this and if you're using a deep learning framework they'll usually have some default way to estimate mu and Sigma squared tension work reasonably well as well but in practice any reasonable way to estimate the mean and variance of your hidden unit values of Z should work fine and test so that's it - dome and using it I think you'll be able to train much deeper networks and get your learning album to run much more quickly before we wrap up for this video I want to share you some thoughts on deep learning frameworks as well let's start to talk about that in the next video