CFA® Level II Quantitative Methods - Overfitting in Machine Learning

foreign [Music] we've learned in the last lesson that under supervised learning a set of training samples is used to train a machine learning model and a separate test sample is used to evaluate the model's ability to predict well on new data in some cases data scientists add in a separate validation sample to validate and tune the model before the final testing the training sample is often referred to as being in Sample while the validation and test samples are commonly referred to as being out of sample we can look at it this way when given a bunch of training data a machine learning algorithm tries its best to explain the variation in the data and builds this explanation into its model if this model still explains the variations well when given out of sample data we say that the model generalizes well such a model should have a high r squared for both in Sample and out of sample data underfitting occurs when the algorithm fails to identify the actual relationship often due to an oversimplified model such a model will have a poor fit for both in Sample and out-of-sample data the bigger problem for machine learning algorithms however is that of overfitting overfitting is an issue when the machine learning algorithm tries too hard to explain the variation in the training data such that the model becomes overly complex usually using too many features in the model while such a model will have a very high r squared meaning that it explains the variation in the training data very well the model does not generalize very well this is because some of the variation in both in Sample and out of sample data are just noise and randomness the r squared when using out of sample data is likely to be low to measure how well a model generalizes data scientists study three types of errors bias error is the in-sample error resulting from models with a poor fit variance error is the out of sample error resulting from overfitted models that do not generalize well and base errors are residual errors due to random noise one way to understand the types of Errors is to study the accuracy rate the accuracy rate is 1 minus the error rate if we plot the in-sample accuracy rate against the sample size used in the training we would expect a downward sloping curve this is because the more samples there are it gets harder to fit all the samples in a model that fits well conversely out of sample accuracy rate should be an upward sloping curve as the number of samples increases the accuracy converges to the true accuracy of the model a learning curve plots the accuracy rate of the model for out-of-sample data against the accuracy rate for in-sample data if the model is robust the two curves should converge toward the base accuracy of the model however in the case of an underfitted model both the in-sample and outer sample accuracies are far from the desired accuracy rate of the model because the high bias error for an overfitted model the buyer's error is low so the in-sample accuracy may approach the desired accuracy however the out-of-sample accuracy is low as there is high variance error only with a robust model will both accuracy rates converge towards the desired rate when the number of samples increase another way to study error rates is to make them a function of model complexity as complexity increases in the trained model bias error shrinks as the model gets better at explaining the in-sample variations however variance errors increase as the increasingly complex models get poorer at explaining the variations in the out of sample data typically linear functions are more susceptible to bias error and underfitting while non-linear functions are more prone to variance error and overfitting finding the optimal complexity for the model is a core part of the machine learning process and the key to successful generalization so as you can see one way to prevent the problem of overfitting is to reduce the complexity of the model in complexity reduction a penalty value is imposed for every feature used by the model this forces the model to only include features that reduce the out of sample data the second strategy comes from the principle of avoiding sampling bias when the sample size is too small sampling bias increases this is a problem as the already limited sample size has to be divided into training sample validation sample and test sample as the three groups of samples have to be mutually exclusive the training size may be reduced too much to mitigate this problem a technique called cross-validation can be used in the k-fold cross-validation the sample is randomly divided equally into K parts the training is done on K minus 1 Parts with one part left for validation this process is then repeated K times using different combination of the parts this helps reduce bias in the training sample by ensuring that each data point is used in the training set K minus 1 times the average of the K validation errors is then taken as a reasonable estimate of the models out of sample error you're watching an excerpt from our comprehensive animation library for more videos like these head on down to prep nuggets.com at prep nuggets let us do the hard work for you