The Purpose of Train-Test Split in Machine Learning | How to Correctly Split Data?

Key Takeaways

in this video we will cover one of the most important Concepts in machine learning called train test split let's say you have a data set of sold houses collected during the past 5 years consisting of their features and corresponding prices you decide to use all the data for training now when the model is ready you have to understand how good are its predictions how the goal of machine learning models is to learn patterns from a set of data and generalize well to new unseen data since you used all your data for training you don't have observations to test your model on for that reason in most machine learning projects the data set is firstly split into train and test sets the ratio of splitting depends on the size of the available data set and a specific scenario you always wish to have more training data on the other hand you should always allocate enough data for testing your model ideally you want your trained data distribution to be the same as the test data distribution this means that if the the model adequately learns the train patterns it would provide good results for the test set also when you have enough data you can achieve this by randomly splitting the data into train and test sets which ensures that both subsets are representative of the overall data set for little data train test split can be problematic because we may not have enough samples in the splits covering all the necessary examples of the problem domain for instance you have five Villas four huts and one apartment it it does not matter how you split this data you are going to face a problem with the one apartment you have note that you should never look into the distribution of the test set this means that the train test split should be done at the very beginning of any analysis you never want to leak information from the test set to the model you should configure all the hyperparameters train your final model and only at that stage use the test data for evaluation now you may ask how to configure the hyper parameters if you don't know how it performs Ms on unseen data this is done using the validation set a topic we will talk about in the upcoming videos if you want to learn more about artificial intelligence subscribe to our channel to be aware of the new videos press the like button and let's discuss AI in the comments section