R Tutorial: Model Validation, Model Fit, and Prediction

Want to learn more? Take the full course at https://learn.datacamp.com/courses/machine-learning-for-marketing-analytics-in-r at your own pace. More than a video, you'll learn hands-on coding & quickly apply skills to your daily work. --- There are several goodnesses of fit measures used to judge a model's fit. One is the so-called coefficient of determination or the Multiple R-squared. The value of Multiple R-squared provides the proportion of the dependent variable's variance that is explained by the regression model, adjusted for the number of variables in the model. Hence, if R^2 equals 0, none of the variations is explained. An R^2 equal to 1 corresponds to a model that explains 100% of the dependent variable's variation. In general, I want my R^2 to be as high as possible, but values above 0.9 are rarely reached. The F-test is a test for the overall fit of the model. It tests whether or not R^2 is equal to 0. That is to say, at least one regressor (or a set of regressors) has significant explanatory power. In our model, the p-value of the F-test is smaller than 0.05, hence, the hypothesis of an R^2 of zero is rejected. The variables included in the model explain some variation of the margin in year 2. So far I have considered only in-sample goodness of fit measures, that is to say, the model is evaluated on the same data that it was fitted on. This bears the risk of overfitting. Overfitting occurs when not only the relation between the variables - shown in blue - is modeled, but also the relation between the errors - shown in red. Then the model performs great when predicting the dataset it has been fitted on, but the prediction results on new data are poor. The linear model, at first glance, looks like it does not fit well, but for predictions, it will be superior to the more complicated model shown in red. There are several ways to avoid overfitting. One is to keep your model lean. Some measures for the goodness of fit (for example, the AIC) penalize ev