Python Tutorial : Filtering DataFrames

Key Takeaways

we've previously seen how indexing and slicing works with pandas dataframes and series filtering is a more general tool for selecting part of the data not based on labels or positions but based on properties of interest of the data in itself data frames have several special methods for filtering underlying filtering is the idea of a boolean series here we ask where in the salt series is the number of sales recorded greater than 60 the series returned has true or false values according to whether the salt sold in that particular month was more than 60 units in itself a boolean series isn't that useful but when it is used to select from a series or data frame or array it is very powerful when used to make a selection a boolean series is called a filter a filter may be used directly between brackets as a logical expression or it may be assigned to another variable in this case the explain a variable enough salt salt filters can be combined using standard lock of logical operators like and or and not here we combine the filter salt bigger than or equal to 50 with the filter eggs strictly less than 200 first using and so both have to be true second using or so either can be true notice the use of parentheses to enforce order of logical operations missing values are commonly encountered in data analysis these missing values are sometimes represented by zeros at other times by the special name not a number value here let's copy our data frame and add a column bacon with some zero elements we'll use this new data frame df2 to explore pandas methods for selecting or excluding data slices with zeros or gnats for instance we might ask which columns have all nonzero values with the method all this selection excludes the bacon column because it has zero entries we might also ask which call have any nonzero entries in this case there are no columns of all zeros so all of DF 2 is returned a related question is which rows or columns contain man we can ask that with is null and not now combining any and is null returns any columns that have an an value conversely we might want all columns where all data is present that is there are known and values we can combine all and not now to get this from the original data frame DF we can also remove rows with missing data using drop na using how equals any the row may is dropped from DF because it does have an an entry by contrast how equals all would keep this row filtering can be powerful when used to assign values we might select the egg sales for those months in which the salt sales are high explicitly we extract the eggs column from DF and filter the indices where the corresponding entry of the salt column is larger than 55 the result is a series sub selected from the eggs column this filtering allows us to calculate one column based on another in whole or in part based on a filter on one column we can modify the value in another column for example perhaps with every large salt sale a few eggs are thrown in for free it's your turn now to practice using filters with pandas series and data frames in the exercises