Hyper parameters - EXPLAINED!

Key Takeaways

greetings fellow Learners now before we get into this hypnotic world of hyperparameters I have a thought-provoking question for you what are some of the factors that are outside of you that can brighten your day now could it be something as simple as just good weather outside or would it have to be something that comes from another person like somebody giving you chocolates or flowers I ask this so that we can see if we can relate hyperparameters to real life so for example if your body or mind is the neural network set to optimize happiness then can these external factors in some way be considered as hyperparameters since your body depends on them after watching this video please do comment your thoughts below and let me know what you think cuz I would love to hear them this video now is going to be divided into three passes in the first pass we'll go through an overview the second pass some details and the third pass code so let's get to it what are hyperparameters so let's talk about that in this phase this is a feedforward neural network to classify an image as a dog or not during the training phase we can pass image label pairs to the network during the inference phase we can then pass some unseen image and determine if it is a dog or not now during the training phase the neural network would learn some parameters these Edge weights now the performance of the model depends on the parameters of this model which are the edge weights and the parameters are in turn dependent on two things the first is the data during training of the model and the second is that they depend on another set of parameters called hyperparameters the prefix hyper is used to distinguish them from the model parameters and it indicates the top level parameters on which the model depends quiz time have you been paying attention let's quiz you to find out what makes these hyper prameters hyper a they are top level parameters that depend on the model parameters B they are the top level parameters on which the model parameters depend C they are top level parameters that represent the edge weights of the network and D they are the top level parameters that represent the activations in the network comment your answer down below and let's have a discussion and if you think I do deserve it at this point in the video please do give this video a like because it will mean a lot now that's going to do it for past one of this explanation but keep paying attention because I will be back to quiz you in pass one we covered what hyperparameters are and so in this pass let's look at some hyperparameters and talk some details so this is a non-exhaustive list of some hyperparameters and each of these have like tens to even thousands of configurations so the main question here is how does one set the value for these hyperparameters so first you need to find some ballpark value of how to set these hyper parameters and then you can find two hyperparameters to suit your own needs now to get some ballpark estimate I see two main ways to do this the first is practical experience that is like looking at other machine learning Engineers code and building some intuition of your own and then the second way is through theoretical best practices and this includes like seeing the configuration and research papers related to the problems and models similar to the one that you are trying to solve now once you have this ballpark starting point estimate for a hyperparameter you can then manually set these hyperparameters which involves like starting with some value for the hyper parameters training the model see the performance and then manually change the hyper parameter drain the model see the performance until you are satisfied or you can also automatically set hyperparameters and this is most commonly done with the technique known as grid search where we would set a range of values that different hyperparameters can take and then a Cartesian product of all of these hyperparameters is created and then we would train a model on each of these configuration settings and then we will choose the hyperparameter settings that yielded the best model performance let's say now that you are a machine learning engineer and you're trying to build a neural network that is going to predict the price of a house given some information about set house and let's walk through some hyperparameters and see how we would set them for this problem let's assume here that we are going to build a feed forward neural network and the number of features is 10 so the number of layers in this case we can start out super simple so something like one to two hidden layers and then we can evolve from there the number of neurons per layer we can keep it something small if we have an input of 10 maybe some hidden layer Dimension the first one could be like 64 the second one could be like 32 I like keeping it as powers of two but it's not really necessarily fact or youi to do this next is the type of layers we're going to use Simple dense fully connected layers as this is a feed forward neural network connection type we only need fully connected layers now skip connections we don't quite need it because this is quite a small Network itself now we current Connections in this case we are are dealing with nonsequential data and so we don't really need these recurrent connections maybe for sequential data like language or time series analysis some recurrent connections could be useful now the type of activation function we can use relu for our hidden layers and just simply use a linear activation function for the output neuron this is after all a regression problem so the output doesn't need to be constrained now REO can be used to pick up on the complexities of your problem without also excessively leading to Vanishing gradient problems now the Dropout positions in the network could be after every hidden layer with the dropout rate being set to initially like 0.5 to start out that is that there is like a 50/50 chance that a given neuron can turn off or can turn on now the batch normalization positions in the network could be after every single layer except for the final layer now if we go into the training and evaluation related hyperparameters we can say that the number of epoch during training could be set to somewhere starting at 5 and then increased to 10 potentially then training and validation test split ratio in this case if we have an entire data set we could probably split it up into 8020 data set if we are just training and then testing but if you want to do evaluation you can do 80 10 10 or like a 702 10 split now the loss function here well in this case we are doing a regression problem to to predict the price of a house and so a mean squared error would be appropriate but if you had like a classification problem maybe a cross entropy loss would be more appropriate here the metric for model evaluation could simply be like again a mean squared err that is like the difference between the price of a house and the difference between its actual price or it could be a mean absolute error to to make it more human interpretable and these other evaluation metrics could be used in case of like classification problems now cross validation settings here we can use like kfold cross validation in which case we would take the entire training data and split it up into K chunks the value of K could start out to be like five or 10 in this case would be like a reasonable starting value the learning rate is how fast you want this network to learn initially now with the optimizer Choice typically we can do like atom for example as it does provide some benefits over the stochastic gradient and descent and for more information on optimizers I highly recommend you check out this video right over here neural networks take data in parallel but how many samples can they take in parallel at a time and we could set this to be like 8 or 16 or 32 now this is a good example of me using my practical experience to set some of these hyperparameters but there are some more fun and complex models that we could refer some research papers to build so for example in the paper that introduced Alex there are some pretty good guidelines on how to set various parameters for convolution neural networks and these are networks that deal with spatial data like images or audio so overall I hope you can see how practical experience and research papers can help you choose appropriate hyperparameter values quiz time it's that time of video again have you been paying attention let's quiz you to find out how can I set the number of convolution layers for a digit classification problem a use my past experience as a starting point and manually tune hyperparameters B use my past experience as a starting point and perform some grid search C use research papers that solve similar problems as a starting point and manually tune hyperparameters or D use the research papers that solve similar problems as a starting point and perform a grid search no note here that multiple options may be correct so comment your answer down below and let's have a discussion now that's going to do it for quiz time and pass two of the explanation but keep paying attention because I will be back to quiz you so let's run through a collab notebook to tune hyperparameters of a neural network using grid search so in this case first I need to install Scorch Scorch is a library that allows you to use Psych kit learn libraries along with pytorch neural networks and so we install Scorch and we are also going to import it and specifically we're going to import the neural net regressor because in this case we are going to determine the price of a house which is going to be a regression based problem next we also import grid search CV to actually perform grid search and this is the whole reason we need Scorch because it is a part of psychic learn now we can scale our data with the standard scaler we can split our data into train and test we can use this fetch California Housing to get California Housing data set and then other torch related parameters along with numpy so we'll load the California Housing data set and we can see that we have 20,000 examples with eight features right over here we then perform scaling and transform the input data we can then split the train and test data into an 8020 split where 80% is training and 20% is for test next we will convert the numpy arrays to Pi torch float tensors we then create a neural network regression model where we have an input layer with a hidden layer and an output layer that consists of only one neuron and that one neuron is going to be just the price of the house and along the way we're going to also add some activation functions and specifically we're adding reu next we will take our regression model and then package it with in a Scorch neural network regressor and we're going to use the mean squared eror loss because again this is a regression problem and then we will use the atom Optimizer next we actually perform grid search and specifically we want to set some possible values of hyper parameters like Max epox the learning rate and the batch size in my case I think the max Epoch should be around one Epoch or two epochs the learning rate should either be 0.01 or 0.1 and the batch size could be 8 or 32 now we are going to feed the grid search CV with the Scorch model along with the set of grid hyperparameters over here and then we have the CV which is cross validation parameter is three which means that we're going to basically train all of the configuration ation S three times and looking at this you can see that the best score was this value right here and the hyperparameters set that corresponded to this best score was a batch size of eight learning rate of 0.01 and Max Epoch of two now if you want to see kind of down into more details of like what's going on in here you can actually print grid. CV results and you'll get like this wonderful blurb over here now what's important to here is that this based on the hyperparameters and their values it came up with this cartisian product of all possible models so there's about like eight possible models to train over here and all of these eight models are trained three times so we have the first time split zero second time this was the performance and the third time this is split two which is the performance now each of these numbers this number this number and this number we take the average of for the first model right here which is badge size 8 learning rate 0.01 Max epox one and you get this first value here as a metric of performance and similarly you do it for all the other eight right here now the one with the highest score is going to be this this one right over here which is yeah so the rank test over here so the first rank is going to be the second one right over here which basically means that it corresponds to badge size 8 learning rate 0.01 and Max epox 2 which is exactly what we saw returned by this function and so this configuration of hyper parameters would be the best for this current model now we can change the pram grid parameter to include other hyperparameters like activation functions Dropout rates and anything in the list of hyperparameters that I mentioned before quiz time ooh this is going to be a fun one we cannot not perform grid search on a activation functions B dropout rate C end model Edge weights or D optimization algorithms now comment your answer down below and let's have a discussion and if you think I do deserve it at this point and you love learning please do consider giving this video a like because it will help me out a lot now that's going to do for quiz time and pass three of this explanation but before we go let's generate a summary [Music] the performance of a model depends on the model parameters now these are like Edge weights of a neural network the model's parameters in turn depend on the training data and a set of other parameters called hyperparameters and here is an example of such hyperparameters to determine these hyperparameters we determine a ballpark starting point and then tune these parameters and we can get this ballpark starting point of these parameters through practical experience or from theoretical knowledge of reading research papers then you can tune them manually or automatically using techniques like grid search and that's all we have for today now to learn more about optimization algorithms that I mentioned within this video do check out this video on optimizers right over here but thank you all so much for watching and if you think I do deserve it please do give this video a like and I will see you in the next one bye-bye