Python Tutorial: Handling errors and missing data

Key Takeaways

so far we've imported flat files with minor tweaks to set column names and manage the amount of data loaded this is enough if the data is already in great shape but what if there are issues with the data or the import common issues include incorrect column data types which can hinder analysis missing values indicated with custom designators or records that pandas cannot read luckily read CSV offers ways to address these issues during import reducing the wrangling needed later on when importing data pandas infers each columns data type sometimes a guess is wrong checking the data types in the tax data we see that pandas interpreted zip codes as integers they're more accurately modeled as strings though zip codes are not quantities and include meaningful leading zeros instead of letting pandas guess we can set the data type of any or all columns with read CSV SD type keyword argument D type takes a dictionary where each key is a column name and each value is the data type that column should be note that non-standard data types like pandas categories must be passed in quotations here we specify that the zip code column should contain strings leaving pandas to infer the other columns printing the new data frames D types we see that zip codes D type is object which is the pandas counterpart to Python strings missing data is another common issue pandas automatically recognizes some values like n/a or null as missing data enabling the use of handy data cleaning functions but sometimes missing values are represented in ways that pandas won't catch such as with dummy codes in the tax data records were sorted so that the first few have the zip code zero which is not a valid code and should be treated as missing we can tell pandas to consider these missing data with the n/a values keyword argument na values accepts either a single value a list of values or a dictionary of columns and values in that column to treat as missing let's pass a dictionary specifying that any zeros in zip code should be treated as missing data then we filter the data using the is n/a method on the zip code column to view rows with missing postal codes one last issue you may face are lines that pandas just can't parse for example a record could have more values than there are columns like the second record in this corrupted version of the tax data let's try reading it by default trying to load this file results in the long error and no data is imported luckily we can change this behavior with two arguments error bad lines and warned bad lines both take boolean or true false values setting error bad lines to false makes pandas skip bad lines and load the rest of the data instead of throwing an error warned bad lines tells pandas whether to display messages when unpossible lines are skipped let's try importing the corrupted file again this time with error bad lines set to false and warn bad lines equal to true success a word of caution if lines were skipped it's worth investigating what was left out to see if there are underlying issues that should be addressed in this video you've learned about common issues encountered importing data from flat files and how to handle those issues now it's your turn to practice