Tuning Process (C2W3L01)

Key Takeaways

hi and welcome back you've seen by now that changin your net can involve setting a lot of different hyper parameters now how do you go about finding a good setting for these hyper parameters in this video I want to share with you some guidelines some tips how to systematically organize your hyper parameter tuning process which hopefully will make it more efficient for you to converge on a good setting of the hyper parameters one of the painful things about training deep nets is the sheer number of hyper parameters you have to deal with ranging from the learning rate alpha to the momentum term beta the using momentum or the hyper parameters for the atom optimization algorithm which were beta 1 beta 2 and epsilon maybe have to pick the number of layers maybe have to pick the number of hidden units for the different layers and maybe you want to use learning rate decay so you don't just use a single learning rate alpha and then of course you might need to choose the mini batch size so it turns out some of these parameters are more important than others for most learning applications I would say alpha the learning rate is the most important hyper parameter to tune other than alpha a few other high performances I tend to but maybe to Nick's would be maybe the momentum term I say is 0.9 is a good default and also tune the mini batch size to make sure that the optimization algorithm is running efficiently often also fit around the hidden units of the ones I've circled in orange these are really the three that would consider second in importance to the learning rate alpha and then third in importance you know after sitting around the others the number of layers can sometimes make a huge difference and so can learning rate decay and then when using the atom algorithm I actually pretty much never tune beta-1 beta-2 an epsilon pretty much always used point nine point nine nine nine and ten to the minus eight although you can try tuning those as well if you wish but hopefully does give you some rough sense of what type of parameters might be more important than others alpha most important for sure follow maybe by the ones I've circled in orange follow maybe by the ones I circled in purple but this isn't a hard and fast rule and I think other deep learning practitioners may well disagree with you all have different intuitions on these now if you're trying to tune some set of high preferences how do you select the set of values to explore in earlier generations of machine learning algorithms if you had to hyper parameters which I'm calling how to prime to one and have a ground to to here it was common practice to sample the points you know in a grid like so and systematically explore these values here I'm placing down a five by five grid in practice it could be more or less than five five grid but you try out in this example or twenty five points and then you know pick whichever hyper parameter works best and this practice works okay when the number of hyperparameters was relatively small indeed learning what we tend to do and what I recommend you do instead is choose the points at random so go ahead and you know choose maybe your same number of points all right 25 points and then try out the hyper parameters on this randomly chosen set of points and the reason you do that is that it's difficult to know in advance which hyper parameters are going to be the most important for your problem and as you saw in the previous slide some hyper parameters are actually much more important than others so to take an example let's say hyper parameter one turns out to be alpha the learning rate and to take an extreme example let's say that hyper parameter two was that value epsilon that you have in the denominator of the atom algorithm so your choice of alpha matters a lot in your choice of epsilon hardly matters so if you sample in a grid then you've really tried out five values of alpha and you might find that all of the different values of epsilon gives you essentially the same answer so you've now trained 25 models and only got them to try out five values for the learning rate alpha which is the thing that's really important whereas in contrast if you were to sample a random then you know you all have tried out twenty-five distinct values of the learning rate alpha and therefore you'd be more likely to find a value that works really well I've explained this example using just two hyper parameters in practice you might be searching over many more hyper parameters than this so if you have safety hyper parameters I guess instead searching over a square you're searching over a cube where this third dimension is hyper parameter three and then by sampling within this you know three dimensional tube you get to try out a lot more values of each of your three high parameters and in practice you might be searching over even more hyper parameters than three and sometimes it's just hard to know in advance which ones turn out to be the really important high parameters for your application and something random rather than in a grid and it shows that you're more richly exploring the pot set of possible values for the most important hyper parameters whether they turn out to be when you stand for hyper parameters another common practice is to use a course to find something scheme so let's say in this two-dimensional example that you've sampled these points and maybe you found that this point work the best it may be a few other points around it tended to work really well then in the course defined scheme what you might do is zoom in to a smaller region of the hyper parameters and then sample more densely within this space or maybe a gain at random but to then focus more resources on searching within this blue square if you're suspecting that the best setting of the hyper parameters may be in this region so after doing a core sample of this entire square that tells you to then focus on on a smaller square you can then stop pull more densely in this smallest square so this type of a course to find search is also frequently use and by trying out these different values of the high parameters you can then pick whatever value allows you to do best on your training set objective or does best on your development sets or whatever you're trying to optimize in your hyper parameter search process so hope this gives you a way to more systematically organize your hyper parameter search process the two key takeaways are used random something not a grid search and consider optionally but consider implementing a course defined search process but there's even more to hyper parameter search than this let's talk more in the next video about how to choose the right scale on which to sample your hyper parameters