How to find the best model parameters in scikit-learn

Key Takeaways

welcome back to my video series on scikit learn for machine learning in the previous video we learned about kfold cross validation a very popular model evaluation technique and then applied it to three different types of problems in this video I'll be covering the following how can kfold cross validation be used to search for an optimal tuning parameter how can this process be made made more efficient how do you search for multiple tuning parameters at once what do you do with those tuning parameters before making real predictions and how can the computational expense of this process be reduced let's just briefly review the steps of kfold cross validation as well as its benefits and drawbacks first we we choose a number for K often 10 and split the data set into K partitions or folds of equal size the model is trained on all of the folds except one then tested on the remaining fold and evaluated using the chosen evaluation metric that process is repeated K-1 more times such that each fold is the testing set set once and the training set all other times the average testing performance also known as the cross validated performance is used as the estimate of the model's performance on out of sample data that performance estimate is more reliable than the estimate provided by train test split because cross validation reduces the variance associated with a single trial of train test split and as we saw in the last video cross validation can be used for selecting tuning parameters choosing between models and selecting features however cross validation can be computationally expensive especially when the data set is very large or the model is slow to train let's now review how we use the cross Val score function in the last video to select the best tuning parameters it's important to understand this code because it will help you to understand the role of grid search CV which I'll be covering in the next section of this video in this case we were trying to select the best value for k for the KNN model when predicting species on on the iris data set we'll start by importing the relevant functions and classes and then creating the feature Matrix X and response Vector y next we'll instantiate the K neighbors classifier with a K value of five represented by the N neighbors parameter the model object which we called KNN is used as the first parameter in Cross fou score the other four parameters are the feature Matrix X the response Vector y the number of cross validation folds to use and the evaluation metric of our choice we're choosing to use 10-fold cross validation with classification accuracy as the evaluation metric cross valal score returned 10 scores namely the testing accuracy for each of the 10 folds used during cross validation remember that cross Val score takes care of the entire cross validation process including splitting X and Y into the 10 folds which is why we passed it the entirety of X and Y and not X train and YT train the scores object is a nump array and so we can use its mean method in order to calculate the mean which we use as the estimated accuracy for the k equal 5 model on the iris data set we want to repeat this process for a variety of K values to choose the K which will most likely generalize to out of sample data and thus using a for Loop is a Natural Choice pay close attention here because this is the code that we're going to replace with grid search CV we start by creating two lists K range contains the integers 1 through 30 and and indicates the values of K that we want to try and K scores is an empty list that will be used to store the cross validated accuracy for each of the 30 iterations the four Loop iterates through K range and within the loop we instantiate K neighbors classifier with the chosen K value use cross Val score to perform 10-fold cross validation and then append the mean of the 10 scores to the K scores list finally we'll print the 30 cross validated scores as you would expect the fifth score in this list is 966 which we knew from above would be the cross validated score when n neighbors equals 5 finally we plot the K values against the accuracy we see that the values 13 18 and 20 for K appear to be best even though the code above is not particularly difficult to write you could imagine that this is something you might want to do very often and thus it would be nice if there was a function that could automate this process for you that is exactly why grid search CV was created grid search CV allows you to define a set of parameters that you want to try with a given model and it will automatically run cross validation you using each of those parameters keeping track of the resulting scores essentially it replaces the four Loop above as well as providing some additional functionality to get started with grid search CV we first import the class from SK learn. grid search exactly like above we create a python list called K range that specifies the K values that we would like to search next we create what is known as a parameter Grid it's simply a python dictionary in which the key is the parameter name and the value is a list of values that should be searched for that parameter as you can see this dictionary has a single key value pair in which the key is the string n neighbors and the value is a list of the numbers 1 through 30 next we'll instantiate the grid you'll notice that it has the same parameters as cross Val score except that it doesn't include x or y but it does include param grid you can think of the grid object as follows it's an object that is ready to do 10-fold cross validation on a KNN model using classification accuracy as the evaluation metric but in addition it has been given this parameter grid so that it knows that it should repeat the 10-fold cross validation process 30 times and each time the N neighbors parameter should be given a different value from the list hopefully this helps you to understand why the parameter grid is specified using key value pairs we can't just give grid search CV a list of the numbers 1 through 30 because it won't know what to do with those numbers instead we need to specify which model parameter in this case and neighbors should take on the values 1 through 30 one final note about instantiating the grid if your computer and operating system support parallel processing you can set the N jobs parameter to -1 to instruct scikit learn to use all available processors finally we fit the grid with data by just passing it the X and Y objects this step may take quite a while depending upon the model and the data and the number of parameters being searched remember that this is running 10-fold cross validation 30 times and thus the KNN model is being fit and predictions are being made 300 times now that grid search is done let's take a look at the results which are stored in the grid scores attribute this is actually a list of 30 named tupes the first tupal indicates that when the N neighbors parameter was set to one the mean cross validated accuracy was 0.96 and the standard deviation of the accuracy scores was 0.05 while the mean is usually what we pay attention to the standard deviation can be useful to keep in mind because if the standard deviation is high that means that the cross validated estimate of the accuracy might not be as reliable anyway you can see that there is one tupal for each of the 30 Trials of cross validation next I'll just show you how you can examine the individual tupal just in case you need to do so in the future I'm going to slice the list to select the first tupal using the bracket zero notation because it's a name tupal I can select its elements using dot notation parameters is simply a dictionary containing the parameters that were used CV validation scores is an array of the 10 accuracy scores that were generated during 10-fold cross validation using that parameter and mean validation score is of course the mean of those 10 scores it's easy to collect the mean scores across the 30 runs and plot them like we did above so let's just quickly do that we'll use a list comprehension to Loop through grid. grid scores pulling out only the mean score and when we plot the results we can see that the plot is identical to the one we generated above now you might be thinking that writing a list comprehension and then making a plot can't possibly be the most efficient way to view the results of the grid search you're exactly right once a grid has been fit with data it exposes three attributes that are quite useful best score is the single best score achieved across all of the parameters best params is a DI AR containing the parameters used to generate that score and finally best estimator is the actual model object fit with those best parameters which conveniently shows you all of the default parameters for that model that you did not specify as you might have noticed in the plot there are two other values for K which also produced a score of 98 I don't know for sure how grid search decided to choose k equal 13 as the best but some limited testing indicates that it probably just picks the first parameter it tested that achieved that score it might have occurred to you already that sometimes you'll want to search multiple different parameters simultaneously for the same model for example let's pretend you're using a decision tree classifier which is a model we haven't yet covered in the series two important tuning parameters are max depth and Min samples Leaf you could tune those parameters independently meaning that you try different values for max depth while leaving Min samples Leaf at its default value and then you try different values from in samples Leaf while leaving max depth at its default value the problem with that approach is that the best model performance might be achieved when neither of those parameters are at their default values thus you need to search those two parameters simultaneously which is exactly what we're about to do with grid search CV in the case of C NN another parameter that might be worth tuning other than K is the weights parameter which controls how the K nearest neighbors are weighted when making a prediction the default option is uniform which means that all points in the neighborhood are weighted equally but another option is distance which weights closer neighbors more heavily than further neighbors thus I'm going to create a list of those options called weight options in addition to the 30 integer options in K range again I create a parameter grid and in this case the dictionary has two key value pairs one pair for each parameter to be searched now I'll instantiate the gra grid and fit it with data just like before this is known as exhaustive grid search because grid search CV is trying every possible combination of the N neighbors parameter and the weights parameter because there are 30 options for n neighbors and two options for weights 10-fold cross validation is being performed 60 times let's take a quick look at the results you can see that there is a tupal for every possible combination of n neighbors and weights once again we'll examine the best score and the best parameters it turns out that the best score did not improve and thus using n neighbors equals 13 with the default value for weights is still the best set of parameters for this model next I want to briefly remind you of what to do with these optimal tuning parameters once CE you found them before you make predictions on out of sample data it is critical that you train the model with the best known parameters using all of your data as you can see I instantiate K neighbors classifier with the best parameters I found above and then fit it with X and Y not X train and Y train in other words e even if you use train test split earlier in the model building process you should train your model on X and Y before making predictions on new data otherwise you will be throwing away potentially valuable data that your model can learn from here I've just made an example prediction on a madeup data point however grid search CV gives you a shortcut to this process by default grid search CV will refit the model using the entire data set and the best parameters it found that fitted model is stored within the grid object and it exposes a predict method to allow you to make predictions using the fitted model in other words the code in this cell accomplishes the same thing as the code in this cell obviously this is only useful if you're running grid search CV right before making predictions but it can be quite handy at times finally let's take a look at randomized search CV which is a close cousin of grid search CV the problem that randomized search CV aims to solve is that when you're performing an exhaustive search of many different parameters at once the search can quickly become computationally infeasible for example searching 10 different parameter values for each of four parameters will require 10,000 Trials of cross validation which would equate to 100,000 model fits and 100,000 sets of predictions if 10-fold cross validation is being used randomize search CV solves this problem by searching only a random subset of the provided parameters and allowing you to explicitly control the number of different parameters combinations that are attempted as such you can effectively decide how long you want it to run for depending on the computational time you have available as with grid search CV we import the randomized search CV class from SK learn. grid search with randomized search CV you specify parameter distrib distributions rather than a parameter grid for discrete parameters meaning parameters for which you provide a list of discrete values like integers or strings this specification looks exactly the same as before in this case both of my parameters are discret and so paramis is no different from Pam grid however if one of your tuning parameters is continuous such as a regularization parameter for a regression problem it's important to specify a continuous distribution rather than a list of possible values so that randomized search CV can perform a more fine grained search I don't have an example to show you here but some of the resources I'll mention below include examples of this anyway we can now instantiate the randomized search fit it with data and take a look at the scores you'll see that Pam disc is used instead of Pam grid and there are two new parameters n it which controls the number of random combinations it will try and random state which I've set for the purpose of reproducibility from the grid scores you can see that 10 random combinations of parameters were tried and once again we can look at the best score and the best parameters we can see that even though it only tried 10 combinations of n neighbors and weights it still managed to find a combination that has just as high a score as the combination found by grid search CV it's certainly possible that randomized search CV will not find as good a result as grid search CV but you might be surprised how often it finds the best result or something very close in a fraction of the time that grid search CV would have taken I've set up a quick experiment here in which I run randomized search CV with n it equals 10 and I repeat this process 20 times recording the best score each time as you can see most of the time it does result in a score of 98 and when it does doesn't find that score it's still close to 98 in terms of practical recommendations I would usually recommend starting with grid search CV but then switching to randomize search CV if grid search CV is taking longer than the time you have available when using randomized search CV start with a very small value of n itter time how long that takes and then do the math for how large you can set the NIT parameter without exceeding the amount of time you have available as always the psychic learn documentation is helpful if you want to learn more the first link is their user guide covering the topic of grid search in general followed by links to the detailed documentation for grid search CV and randomized search CV next is a comparison of randomized search and grid search also from the psychic learn documentation which provides some code that demonstrates how much time a randomized search can save you over an exhaustive grid search next is a video segment in I python Notebook on randomized search by Andreas Mueller who is one of the core contributors to psych it learn the relevant portion of the video which I've linked to is just 3 minutes long but the rest of the video is worth watching if you want to learn about the more advanced features of psychic learn the final resource is a paper by Yoshua benio famous for his research on deep learning which argues in favor of using a random search process for parameter tuning as always I appreciate you watching and look forward to your comments and questions please subscribe on YouTube if you'd like to be notified when my next video is released thanks again and I'll see you soon