R Tutorial: Cross-validation

Want to learn more? Take the full course at https://learn.datacamp.com/courses/machine-learning-with-caret-in-r at your own pace. More than a video, you'll learn hands-on coding & quickly apply skills to your daily work. --- In the last video, we manually split our data into a single test set and evaluated out-of-sample error once. However, this process is a little fragile, the presence or absence of a single outlier can vastly change our out-of-sample RMSE. A better approach than a simple train/test split is using multiple test sets and averaging out-of-sample error, which gives us a more precise estimate of the true out-of-sample error. One of the most common approaches for multiple test sets is known as cross-validation, in which we split our data into ten "folds" or train/test splits. We create these folds in such a way that each point in our dataset occurs in exactly one test set. This gives us ten test sets, and better yet, means that every single point in our dataset occurs exactly once. In other words, we get a test set that is the same size as our training set, but is composed of out-of-sample predictions! We assign each row to its single test set randomly, to avoid any kind of systemic biases in our data. This is one of the best ways to estimate out-of-sample error for predictive models. One important note, after doing cross-validation, you throw all resampled models away and start over! Cross-validation is only used to estimate the out-of-sample error for your model. Once you know this, you re-fit your model on the full training dataset, so as to fully exploit the information in that dataset. This, by definition, makes cross-validation very expensive. It inherently takes eleven times as long as fitting a single model: ten cross-validation models plus the final model. The train function in caret does a different kind of resampling known as bootstrap validation, but is also capable of doing cross-validation, and the two methods, in practice, yield si