Python Tutorial: Introduction to model validation

Key Takeaways

hello my name is Casey Jones and welcome to this course on model validation let's get started so what is model validation well model validation consists of various steps and processes that ensure your model performs as expected on new data the most common way to do this is to test your models accuracy on data it has never seen before called a holdout set if your models accuracy is similar for the data it was trained on and the holdout data you can claim that your model is validated however model validation can also consist of choosing the right model the best parameters and even the best accuracy metric the ultimate goal of model validation is to end up with the best performing model possible that achieves high accuracy on new data before we begin exploring model validation let's review some basic modeling steps using scikit-learn modeling in Python follows a simple procedure regardless of the type of model you are constructing whether you are a seasoned scikit-learn veteran are new to building models with this module let's take a quick look at these steps first we create a model by specifying the model type and its parameters in this case we are creating a random forest regression model with random forests regressor second we fit the model using the dot fit method this method has two main arguments X an array of data used in the model as training data and Y an array of response values matching the size of the X array when dot fit is used the model parameters will be printed in the console to assess model accuracy we generate predictions for data using the dot predict method and lastly we look at the accuracy metrics here we are comparing the models predictions the variable predictions and the actual responses Y test future lessons and exercises will be devoted to accuracy metrics as they are a vital component to model validation for this current example though we are looking at the mean absolute error this function takes two arrays as arguments the true values y true and the predicted values y pred and returns the mean absolute error between them this process of generating a model fitting predicting and then reviewing model accuracy will be repeated throughout this course if you are unfamiliar with these steps you should consider taking the prerequisite courses they will go into more detail about using Python and performing these modeling steps throughout this course we will use 538 ultimate Halloween candy power ranking data sets several times this data set contains 85 different candies data on their various characteristics and a column specifying how often that candy selected in a head-to-head matchup with other candies this column is a win percentage and contains values between 0 and 100 model validations main goal is to ensure that a predictive model will perform as expected on new data obtaining predictions for training data or seen data and testing data or unseen data is coded in the same way and uses the dot predict method generally models perform a lot better on data they have seen before as unseen data may have features or characteristics that were not exposed in the model if your training and testing errors are vastly different it may be a sign that your model is over fitted we will use model validation to make sure we get the best testing error possible let's see why model validation is so important by looking at an example of training and testing act