R Tutorial : How to summarise missing values

Key Takeaways

now that you understand what missing values are how to count them and how they operate let's scale these up to more detailed summaries of missingness we need to summarize missing data to identify variables cases or patterns of missingness as these can bias our data analysis there are two main summaries basic and data frame summaries basic summaries are returned a single number like the number of missing or complete values using n miss or n complete however you will need more detailed missing the summaries to help you on your journey through a data analysis this lesson introduces you to missing data summaries gnarnia provides a family functions all starting with miss underscore which each provide different summaries of missingness and return a data frame this allows us to see features they can be difficult to articulate or time-consuming to calculate for example ms VAR summary and miss case summary returned the number and percentage of missings in each variable or case these summaries work with deep layers group by so you can fluidly explore missingness by age group use miss VAR summary to summarize the number of missing x' in each variable this returns a data frame where each row is a variable it also includes summaries of the number and percentage of missings for each variable in the data set and is sorted by the number of missing x' for example ozone has 37 missing values and is about 24.2% missing similar to miss var summary miss case summary returns a summary data frame where each case represents a row in the data set here case five the fifth row on the data set has two missing values which means 33% of that case is missing tabulation of missingness counts the number of times there are 0 1 2 3 and so on missings they're very useful compact summaries that reveal interesting structure miss var table returns a data frame with the number of missing x' in a variable and the number and percentage of variables affected for example there are 4 variables with no missings detected which corresponds to 66.7% of the variables and there was one variable with 7 missings and one variable with 37 missings similarly miss case table returns the same information but for cases we can also look at missing this over a given span or run for a given variable using miss VAR span and miss var run these can be really useful for data with many regular measurements like time series data miss VAR span calculates the number of missings in a variable for a repeating span this is really useful in time series data to look for weekly that is seven-day patterns of missingness miss var span returns a data frame with columns span counter which identifies the span the first the second and so on and also includes the number and proportion of missing and complete values for example in span 10 there are 432 missings and 3568 complete values note also that an miss plus n complete equals the span here this is a span of 4,000 and out of these 4,000 values in this span 0.108 are missing and point 8 down to our complete given by prop miss and prop complete miss VAR run returns the runs or streaks of missingness this is useful to try and find unusual patterns of missingness it returns the length of the run of complete and missing data this is particularly useful for finding repeating patterns of missingness sometimes you are interested in missing us for groups in the data each missing this summary function can actually be calculated by group using group by from d player for example we can look at the missing this by month in the air quality data set here we see that month 5 for ozone has 5 missings but month six rows own has 21 missings now