R Tutorial: Out-of-sample error measures

Want to learn more? Take the full course at https://learn.datacamp.com/courses/machine-learning-with-caret-in-r at your own pace. More than a video, you'll learn hands-on coding & quickly apply skills to your daily work. --- Hi! I'm Zach Deane Mayer and I'm one of the co-authors of the caret package. I have a passion for data science and spend most of my time working on and thinking about problems in machine learning. This course focuses on predictive rather than explanatory modeling. We want models that do not overfit the training data and generalize well. In other words, our primary concern when modeling is "do the models perform well on new data?" The best way to answer this question is to test the models on new data. This simulates real world experience, in which you fit on one dataset, and then predict on new data, where you do not actually know the outcome. Simulating this experience with a train/test split helps you make an honest assessment of yourself as a modeler. This is one of the key insights of machine learning: error metrics should be computed on new data because in-sample validation, or predicting on your training data, essentially guarantees overfitting. Out-of-sample validation helps you choose models that will continue to perform well in the future. This is the primary goal of the caret package in general and this course specifically: don’t overfit. Pick models that perform well on new data. Let's walk through a simple example of out-of-sample validation. We start with a linear regression model, fit on the first 20 rows of the mtcars dataset. Next, we make predictions with this model on a new dataset: the last twelve observations of the mtcars dataset. The twelve cars in this test set will not be used to determine the coefficients of the linear regression model and are therefore a good test of how well we can predict on new data. In practice, rather than manually splitting the dataset, we'd actually use the createResamples or createFol