Python Tutorial : Clean and Validate

Key Takeaways

in the previous lesson we read data from the National Survey of family growth and selected a column from a data frame in this lesson we'll check for errors and prepare the data for analysis we'll use the same data frame we used in the previous lesson NS f G which contains one row for each pregnancy in the survey I'll select the variable birth weight l b1 which contains the pound part of birth weight and assign it to pounds and birth weight oszi one contains the ounce part of birth weight so I'll assign that to ounces before we do anything with this data we have to validate it one part of validation is confirming that we are interpreting the data correctly we can use value counts to see what values appear in pounds and how many times each value appears by default the results are sorted with the most frequent value first so I use sort index to sort them by value instead with the lightest babies first and heaviest babies last as we'd expect the most frequent values are 6 to 8 pounds but there are some very light babies a few very heavy babies and two values 98 and 99 that indicate missing data we can validate the results by comparing them to the codebook which lists the values and their frequencies the results here agree with the codebook so we have some confidence that we are reading and interpreting the data correctly another way to validate the data is with describe which computes summary statistics like the mean standard deviation and min and Max here are the results for pounds count is the number of values the minimum and maximum values are 0 and 99 and the fiftieth percentile which is the median is 7 the mean is about 8.0 5 but that doesn't mean much because it includes the special values 98 and 99 before we can really compute the mean we have to replace those values with n a n to represent missing data the replacement that does what we want it takes a list of values we want to replace and the value we want to replace them with and P dot n means we're getting the special value nan from the numpy library which is imported as n P the result from replace is a new series which I assign back to pounds remember that the mean of the original series was about 8.0 5 pounds the mean of the new series is about 6 point 7 pounds it makes a big difference when you remove a few 99 pound babies instead of making a new series you can call replace with in place equals true which modifies the existing series in place that is without making a copy here's what that looks like four ounces since we didn't make a new series we don't have to assign it back to ounces now we want to combine pounds and ounces into a single series that contains total birth weight arithmetic operators work with series objects so to convert from ounces to pounds we can divide by 16 there are 16 ounces in a pound then we can add the two series objects to get the total here are the results the mean is about 7.1 which is a little more than what we got before we added in the ounces part now we're close to answer in our original question the average birth weight for babies in the US but as we'll see in the next lesson we're not there yet first let's practice what we learned in this video