Selecting the best model in scikit-learn using cross-validation

Key Takeaways

welcome back to my video series on machine learning in pych learn in the previous video we went through the entire data science pipeline including reading data using pandas visualization using Seaborn and training and interpreting a linear regression model using scikit learn we also covered evaluation metrics for regression as well as feature selection using the train test split procedure in this video I'll be covering the following what's the drawback of using the train test split procedure for model evaluation how does kfold cross validation overcome this limitation how can cross validation be used for selecting tuning parameters choosing between models and selecting features and what are some possible improvements to cross validation let's start by reviewing what we've learned so far about model evaluation procedures one of the primary reasons we evaluate machine learning models is so that we can choose the best available model the goal of supervised learning is to build a model that generalizes to out of sample data and thus we want a model evaluation procedure that allows us to estimate how well a given model is likely to perform on out of sample data we can then use that performance estimate to choose between the available models our first idea was to train each model on the entire data set and then evaluate each model by testing how well it performs on that same data this produces an evaluation metric known as training accuracy unfortunately training accuracy rewards overly complex models that are unlikely to generalized to Future data which is known as overfitting the training data the alternative procedure we came up with is called train test split in which we split the data set into two pieces known as the training and testing sets we train the model on the training set and we evaluate the model by testing its performance on the testing set the resulting evaluation metric is known as testing accuracy which is a better estimate of out of sample performance than training accuracy because we trained and tested the model on different sets of data as well testing accuracy does not reward overly complex models and thus it helps us to avoid overfitting however there is a drawback to the train test split procedure it turns out the testing accuracy is a high variance estimate of out of sample accuracy meaning that testing accuracy can change a lot depending on which observations happen to be in the testing set let's see an example of this in pyit learn using the iris data set which we've worked with previously first we'll import the relevant functions classes and modules then we'll read in the iris data set and Define our feature Matrix X and our response Vector y in this next cell we'll use the train test split function to split X and Y into four pieces this splits a random but I've set the random State parameter so that if you run this code at home using the same random State value your data will be split in the exact same way as my data we'll also instantiate the K neighbors classifier model with the parameter n neighbors equals 5 fit the model with the training data make predictions on the testing data and evaluate the accuracy of our predictions the reported testing accuracy of our model is 97% but what if we reran this code and the only thing we changed was which observations were assigned to the training and testing sets if we change the random State parameter to three and rerun the cell we get a testing accuracy of 95% if we change the random state to two we get a testing accuracy of 100% this is why testing accuracy is known as a high variance estimate naturally you might think that we could solve this problem by creating a bunch of different train test splits calculating the testing accuracy each time and then averaging the results together in order to reduce the variance in fact that is the essence of how cross validation Works let's walk through the steps for kfold cross validation which is the most common type of cross validation first we choose a number for K and split the entire data set into K partitions of equal size these partitions are known as folds so if K was equal to five and the data set had 150 observations each of the five folds would contain 30 observations second we designate the observations in fold one as the testing set and the union of all other folds as the training set in the example I just mentioned the testing set would contain the 30 observations from fold one and the training set would contain the 120 observations from folds 2 through 5 third we train the model on the training set we make predictions on the testing set and we calculate the testing accuracy then we would repeat steps 2 and three k times using a different fold as the the testing set each time since k equals 5 in our example we would be repeating this process a total of five times during the second iteration fold two would be the testing set and the union of folds 1 3 4 and five would be the training set during the third iteration fold three would be the testing set and the Union of folds 1 2 4 and 5 would be the training set and so on finally the average testing accuracy also known as the cross validated accuracy is used as the estimate of out of sample accuracy here's a diagram of five-fold cross validation that may be helpful to you as you can see each fold acts as the testing set for one iteration and is part of the training set for the other four iterations one thing I want to make clear is that we are dividing the observations into fivefolds we are not dividing the features into fivefolds that might be confusing from the diagram since normally we represent observation as rows whereas this diagram represents observations as columns I've written some simple code to demonstrate the process of splitting a data set into folds it's not important that you understand this code rather I just want to focus on the output pretend that you have a data set with 25 observations numbered 0- 24 and you want to use fivefold cross validation this is an example of how that data set might be split into the five folds each line represents one iteration of cross validation in which the data set has been split into training and testing sets for each iteration we can see the ID numbers of the 20 observations in the training set and the ID numbers of the five observations in the testing set you'll notice that for a given iteration every observation is either in the training set or the testing set but not both you'll also notice that every observation is in the testing set exactly once let's briefly compare cross validation to train test split to clarify the advantages of each the main reason we prefer cross validation to train test split is that cross validation generates a more accurate estimate of out of sample accuracy which which is what we need in order to choose the best model as you've seen it also uses the data more efficiently than train test split since every observation is used for both training and testing the model however there are two primary advantages to using train test split instead of cross validation first train test split runs K time times faster than kfold cross validation since kfold cross validation essentially repeats the train test split process K times this is an important consideration for larger data sets as well as models that take a long time to train second it's much easier to examine the detailed results of the testing process from train test split as you'll see below psyit learn makes cross validation very easy to implement but all you get back are the resulting scores this makes it difficult to inspect the results using a confusion Matrix or Roc curve which are tools for model evaluation whereas train test split makes it easy to examine those results before we walk through some cross validation code I want to present two recommendations for the use of cross validation first we've been using the value k equals 5 for our examples and in fact any number can be used for K however k equal 10 is generally recommended because it has been shown experimentally to produce the most reliable estimates of out of sample accuracy second when you use cross validation with classification problems it is recommended that you use stratified sampling to create the folds this means that each response class should be represented with a approximately equal proportions in each of the folds for example if your data set has two response classes ham and spam and 20% of your observations were ham then each of your cross validation folds should consist of approximately 20% ham thankfully psyit learn uses stratified sampling by default when using the cross Val score function which is what we'll use below and thus you don't need to worry about implementing stratified sampling yourself let's now go through an example for how cross validation can be used in pyit learn to help us with parameter tuning we're again using the iris data set and our goal in this case is to select the best tuning parameters also known as hyperparameters for the K nearest neighbors classification model in other words we want to select the tuning parameters for KN andn which will produce a model that best generalizes to out of sample data we'll focus on tuning the K in K nearest Neighbors which represents the number of nearest neighbors that are taken into account when making a prediction note that this K has nothing to do with the K in kfold Cross validation our primary function for cross validation in pyit learn will be cross Val SC score which will import from the sklearn doc crossvalidation module we're going to try out the value k equals 5 so we instantiate a k neighbors classifier model with the N neighbors parameter set to that value and save the model as an object called K&N note that we didn't have to import K neighbors classifier because because we already imported it previously we'll now use the cross FAL score function giving this function five parameters the first parameter is the model object KN andn the second and third parameters are X and Y meaning our feature Matrix and response Vector it's very important to note that we are passing the entirety of X and Y to cross fou score not X train and Y train as we'll discuss below cross Val score takes care of splitting the data into folds and thus we do not need to split the data ourselves using train test split the fourth parameter is CV equals 10 which means that we want it to use 10-fold cross validation the final parameter is scoring equals accuracy which means that we want to use classification accuracy as the evaluation metric there are many possible evaluation metrics so I recommend that you always explicitly specify which one you want to use you can see the complete list of metrics in psyit learns model evaluation documentation which I've linked to in the resources section below before we run this code let's discuss what the cross fou score function actually does and what it returns basically cross fou score executes the first four steps of kfold cross validation it will split X and Y into 10 equal folds it will train the KNN model on the union of folds 2 through 10 test the model on fold one and calculate the testing accuracy then it will train the KNN model on the union of fold 1 and fold 3 through 10 test the model on fold two and calculate the testing accuracy and it will do that eight more times when it's finished it will return the 10 accuracy scores as a numpy array in our code we're going to save that numpy array as an object called scores and then print it out we can see that during the first iteration the model achieved a testing accuracy of 100% in the second iteration the accuracy was 93% and so on as mentioned above we'll usually average the testing accuracy across all 10 iterations and use that as our estimate of out of sample accuracy it happens that numpy arrays have a method called mean so we can simply print scores. mean in order to see the mean accuracy score it turns out to be about 97% because we use cross validation to arrive at this result we're more confident that it's an accurate estimate of out of sample accuracy than we would be if we had used train test split our goal here is to find an optimal value of K for K and N which we set using the N neighbors parameter thus we will Loop through a range of reasonable values for K and for each value use 10-fold cross validation to estimate the out of sample accuracy we first create a list of the integers one through 30 which are the values we'll try for k then we create K scores which is an empty list in which we'll store the 30 scores we use a for Loop to Loop through the values 1 through 30 during each iteration of the loop we instantiate K neighbors classifier with n neighbors equal to the selected K value we run 10-fold cross validation with that model and finally we append the mean accuracy to the case scores list let's run this code and print the results these numbers are a bit difficult to scan through visually so we'll use a line plot with Matt plot lib in order to visualize how the accuracy changes Es as we vary the number for K the maximum cross validated accuracy occurs at k = 13 through K = 20 the general shape of the curve is an upside down U which is quite typical when examining the relationship between a model complexity parameter and the model accuracy as I mentioned briefly in a previous video this is an example of the bias variance tradeoff in which low values of K produce a model with low bias and high variance and high values of K produce a model with high bias and low variance the best model is found in the middle because it appropriately balances bias and variance and thus is most likely to generalize to out of sample data when deciding which exact value of K to call the best it is generally recommended to choose the value which produces the simplest model in the case of KN andn higher values of K produce lower complexity models and thus we'll choose K = 20 as our single best KNN model so far we've used cross validation to help us with parameter tuning let's look at a brief example to demonstrate how cross validation can help us to choose between different types of models specifically we want to compare the best KNN model on the iris data set with a logistic regression model which is a popular model for classification first let's run 10-fold cross validation with the best KNN model to see our accuracy as we saw above the accuracy is 98% note that instead of saving the 10 scores in an object called scores and then calculating the mean of that object I'm just running the mean method directly on the results we'll compare this with logistic regression by importing and instantiating a logistic regression model and then again running 10-fold cross validation this gives us an accuracy of 95% and so we would conclude that KNN is likely a better choice than logistic regression for this particular task finally let's check out how cross validation can help us with feature selection if you remember the advertising data set from the last video you may Rec call that we were using linear regression to predict sales one question we had was whether to include the newspaper feature in that model we start by importing pandas numpy and linear regression and then reading the data into pandas we'll first try a model with all three features to build our feature Matrix X we create a python list of feature names and then use that list to select just those columns from the data frame we also select the sales column as our response Vector y we instantiate the linear regression model and we'll again use cross file score for cross validation since it works with both both classification and regression models we can't use accuracy as our evaluation metric since that's only relevant for classification problems we want to use root mean squared error but that is not directly available via the scoring parameter so we'll instead ask for mean squared error and then later take the square root anyway let's run 10-fold cross validation on our model and examine the scores this doesn't seem right if you remember the formula for mean squar error from the last video you'd agree that the results should be positive numbers and all of these numbers are negative what happened here it's complicated to explain but it boils down to the fact that classification accuracy is a reward function meaning something you want to maximize whereas mean squ error is a loss function meaning something you want to minimize there are other psychic learn functions that depend on the results of cross Val score and those functions select the best model by looking for the highest value of cross file score finding the highest value of reward function makes sense for choosing the best model but finding the highest value for a loss function would select the worst model thus a design decision was made for cross valou score to negate the output for all loss functions so that when other s kit learn functions call cross Val score those functions can always assume that higher results in indicate better models if you found that confusing you're not alone there is a long discussion of this issue in pyit learns GitHub repository that has been going on since 2013 which I'll link to in the resources below if you want to read further this behavior is likely to change in a future version of psyit learn but for now we just need to work around this issue the workaround is simply to take the negative of scores and store it in msse scores we'll convert mean squared error to root mean squared error by taking the square root finally we'll calculate the average root means squared error across the 10 cross validation folds thus we would estimate that the out of sample root mean squared error for this model would be 1.69 our goal here was to compare the model including newspaper with a model excluding newspaper so let's create a new X that only includes the TV and radio features we'll compute the average root mean squared error in a single line This Time by using cross Val score taking the negative taking the square root and then taking the mean the resulting estimate is 1.68 since this is a lower number than the model that included newspaper and root means squared error is something we want to minimize we would conclude that the model excluding newspaper is a better model to wrap up I want to briefly go through some common variations to cross validation that are likely to make it an even better procedure the first is repeated cross validation in which kfold cross validation is repeated multiple times with different random splits of the data into the K folds and the results are averaged this provides a more reliable estimate of out of sample performance by reducing the variance associated with a single Tri file of cross validation the second Improvement is to create a holdout set instead of running cross validation on the entire data set a portion of the data is held out and not touched during the model building process the best model is located in tuned using cross validation on the remaining data at the end of this process the holdout set is then used to test the best model their performance on this holdout set is considered to be a more reliable estimate of out of sample performance than the cross validated performance since the hold out set is out of sample for the entire process the final Improvement is for all feature engineering and selection to take place within in each cross validation iteration performing these tasks before cross validation does not fully mimic the application of the model to out of sample data since those processes will have unfair knowledge of the entire data set and thus the cross validated estimate of out of sample performance will be biased upward as such a more reli viable performance estimate is generated when these tasks only take place within the cross validation iterations obviously all of these procedures add some amount of complexity to your modeling process or your modeling code and may also be computationally more expensive depending upon the particular problem the benefits of using these improved procedur procures may not be worth the costs so ultimately it's up to you whether or not simple kfold cross validation is good enough for your application the proper application of cross validation is a deep topic and so ultimately there's no way for me to offer Universal advice that applies to all situations I've got a lot of great resources for you this week first is pyit learns documentation on Cross validation which goes into many different cross validation strategies including how to build your own cross validation iterator there's also the documentation on model evaluation which covers many different evaluation metrics including advice on when and how to use use each metric if you're interested in the discussion behind why cross valou score returns negative values for mean squar eror check out this issue in scikit learns GitHub repository for a bit more depth on Cross validation at a conceptual level read section 5.1 of an introduction to statistical learning and watch the two related videos Scott fort row author of The excellent article on the bias variance tradeoff that I mentioned in a previous video has another good article which Compares adjusted r squared AIC and Bic train test split and cross validation as estimates of model Performance Machine learning Mastery has a friendly introduction to feature selection which provides some good general advice and touches on the issue of doing feature selection within each cross validation iteration there's also a great IPython notebook from Harvard's data science class which demonstrates how feature selection within cross validation iterations is important when the number of features is significantly larger than the number of observations in your data set finally I've linked to a readable paper that examines many different variations of cross validation in detail and argues for repeated cross validation among other recommendations thanks so much for your feedback on the last video in which I asked whether I should cover more Panda functionality or just focus exclusively on pyit learn the majority of you ask that I focus on psyit learn but introduce useful pandas functionality as needed so that's what I'll do in the next video we'll discuss how to search for optimal tuning parameters in a more automatic fashion until then I look forward to your comments and questions thanks as always for watching and please subscribe on YouTube if you'd like to be notified when my new videos are released