Python Tutorial : Risk with missing data in loan data

Want to learn more? Take the full course at https://learn.datacamp.com/courses/credit-risk-modeling-in-python at your own pace. More than a video, you'll learn hands-on coding & quickly apply skills to your daily work. --- With the outliers now removed from our data set, we can now focus on another problem with credit data and that is when data is missing. Normally, you might think of missing data as when an entire row is missing, but that is not the only way data can be missing. Data can be missing when there are null values in place of actual values. It can also be an empty string instead of a real string. For this course, we will refer to missing data as when specific values are not present, not when entire rows of data are missing. Any of the columns within our data can contain missing values. If we see a row of data with missing values in a Pandas dataframe, it will look something like this. Notice for employment length we see NAN, or not a number, instead of a value. One issue with missing data is similar to problems caused with outliers in that it negatively impacts predictive model performance. It can bias our model in unanticipated ways, which can affect how we predict defaults. This could result in us predicting a large number of defaults that are not actually defaults because the model is biased towards defaults. Also, many machine learning models in Python do not automatically ignore missing values, and will often throw errors and cease training. Here are some examples of missing data and possible results. If there are null values in numeric or string columns, the model will throw an error. So, how do we handle missing data? Most often, it is handled in one of three ways. Sometimes we need to replace missing values. This could be replacing a null with the average value of that column. Other times we remove the row with missing data altogether. For example, if there are nulls in the loan amount, we should drop those rows entirely. We sometime