Python Tutorial: Class distribution

Want to learn more? Take the full course at https://learn.datacamp.com/courses/ob... at your own pace. More than a video, you'll learn hands-on coding & quickly apply skills to your daily work. --- One of the most necessary steps for preprocessing, which you should be familiar with if you've taken other courses on Python and machine learning, is splitting up your data into training and test sets. We do this to avoid the issue of overfitting. If we train a model on our entire set of data, we won't have any way to test and validate our model because the model will essentially know the dataset by heart. Holding out a test set allows us to preserve some data the model hasn't seen yet. Just to review, this is how you split up your dataset in scikit learn using the train_test_split function. This should look familiar to you. The function shuffles up your dataset and then randomly splits it. By default, the function will split 75% of the data into the training set and 25% into the test set. In many scenarios, the default splitting parameters will work well. However, if your labels have an uneven distribution, your test and training sets might not be representative samples of your dataset and could bias the model you're trying to train. For example, if you look at the example training and test datasets on this slide, you can see that the training set has only samples labeled n, while there is a y label in the test set. A good technique for sampling more accurately when you have imbalanced classes is stratified sampling, which is a way of sampling that takes into account the distribution of classes or features in your dataset. So for example, let's say we had a dataset with 100 samples, 80 of which are class 1 and 20 of which are class 2. We want the class distribution in both our training set and our test set to reflect this, so in both our training and test sets, we'd want 80% of our sample to be class 1 and 20% to be class 2, which means we'd want 60 class 1 samples