Python Tutorial: Data distributions and transformations

Key Takeaways

welcome back in the previous exercises you practiced finding and handling missing values in this video we're going to discuss what it means to have different distributions between the data used to train a model and test data or future data and how and when to transform your data there is a chance that the way in which we split a data set before we send it into machine learning models can cause the training and test sets to have different distributions which introduces bias into the machine learning framework it's obvious in the illustration shown that the distributions have both different means and different variances and this will likely contribute to poor model performance you'll get to practice this concept in the exercises that follow and in Chapter two we'll introduce the best technique to avoid this problem to prepare two distributions for visualization in this lesson we'll use the two-way multiple assignment seen here as trained comma test since we aren't building models as of yet once we get into building machine learning models in the next chapter we're going to use the four-way split where we'll create a feature matrix capital X and target array Y split into train and test named here as X train X test Y train and so on notice however that the keyword arguments are exactly the same with the data set as the first argument test underscore size equal to 0.3 in this example means that the Train test split of the data will be 70% training and 30% test to create the plots you'll use the pair plot function from the Seabourn module which is always imported and aliased as SNS and returns a plot matrix of distributions and scatter plots so that takes care of what we need to cover for distributions let's move on to data transformations the left-hand side of this image depicts data that is right skewed outliers are one cause of non normality or non-gaussian behavior and transformation can reduce the influence of existing outliers the right side shows the same data after it has been transformed with the log function we'll talk about what to do if outliers still remain after any transformations in the next lesson for now our goal is to transform data from a non normal distribution to the most approximately normal one that we can as we need that for our models to perform better than they would otherwise in the exercises you'll use the box-cox function from the side pi dat stats module to perform power transformations a power transformation raises each value in the data set to a power P the table you see here this common values that can be passed to the transformation parameter lambda note that it's spelled L M B D a 0.0 is used for taking the log of X and 0.5 for the square root of x now it's finally time for you to go check out some distributions and