Python Tutorial: Handling missing values

Key Takeaways

in the previous lesson you were introduced to the two null value types that you encountered in Python in this lesson you will assign null values to the missing values in the data set missing values in a data set aren't usually left unfilled they are filled with dummy values like n/a - or period etc in this lesson you will learn to detect such missing values as well as replace them with nan let's use the college data set which contains various details of college students as an example we'll load the data using PT dot read CSV of college dot CSV the first step in analyzing the data set is to read and print a snippet of the data set will print the head of the College data frame find that all columns have float values if you observe clearly you can see that a few data points are filled with a period this suggests that missing values might be represented by a period however we can confirm this only through further analysis we'll use the info method to get a gist of the data set hey something's odd here all the columns except private or of object type although they are supposed to be float we can further explore and confirm by finding the unique values in one of the columns this way we can find any non numerical values let's apply the unique method on the column C SAT and sort them using MP dot sort from the output you can clearly observe that period is the only string value present hence we need to replace it with nan this can be simply achieved while loading the data set to a data frame you can use the argument na underscore values and be read or read CSV to specify the values for missing data if you again check the info of college you'll find that all the columns are now float64 type this is great now let's consider another data set to detect hidden missing values we will use the Pima Indians tab it is data set which contains various clinical diagnostic information of the patients from the Pima community while loading the data set we can observe non values for missing data when you bring the head of the data frame as before let's print the info of the college data frame they are all float or in type as expected further we can analyze using the described method on the damages data frame observe closely something very odd here is that the PMI column has a minimum value of 0 but we are aware that BMI cannot be 0 hence the zeros must rather be missing values in disguise to confirm the same we can filter all the rows where BMI is 0 there are 11 rows which have BMI as 0 they must be missing values these types of missing values can be tricky as they require some level of domain knowledge will replace these 11 rows of PMI column with nan and check again using NP dot is NaN of tab it is dot PMI great now that we have successfully removed the hidden missing values and replace them with Nance let's summarize what we learned in this lesson we learn to detect missing value characters like period detect the inherent missing values within the data like 0 and replace them with Nan's in the next lesson you'll dig deeper into analyzing the missed