Python Tutorial: Data outliers and scaling

Want to learn more? Take the full course at https://learn.datacamp.com/courses/practicing-machine-learning-interview-questions-in-python at your own pace. More than a video, you'll learn hands-on coding & quickly apply skills to your daily work. --- In the last lesson, we discussed data distributions and transformations. In this video, we'll cover two additional preprocessing steps, finding and handling outliers and how and when to scale your data. Outliers are defined as one or more observations that are distant from the rest of the observations in a given feature. When looking at a histogram of a feature, outliers tend to show up in the tails as you see in this image. The inter-quartile range or IQR is defined as the difference of the values at the 1st and 3rd quartiles, which are at 25% and 75%, respectively, with the median exactly between at 50%. In general, those points above and/or below 1.5 times the IQR should be suspected as possible outliers, which corresponds to the shaded regions seen here. Individual points carry less weight overall in a large dataset than the same datapoint in a smaller dataset. And, a point that is only twice as large as your upper boundary is less concerning than one that is ten times as large. Looking at a simple linear regression model of a dataset with and without outliers reveals just how influential the extreme points are for this particular data. The slope and intercept coefficients are vastly different between the two. A thorough investigation should be undertaken to justify why to remove them or not. And, it's totally possible these anomalies are considered crucial when designing a ML model whose purpose is to detect such anomalous behavior. Some of the functions you'll encounter in the exercises are from the seaborn module where the boxplot function used on our target variable Loan Status supplied to y gives conditioned boxplots, distplot gives a histogram with a kde. Numpy's abs function returns an absol