R Tutorial: Binning encoding: data driven

Key Takeaways

the encoding procedures we have discussed work well on categorical data with a manageable number of categories however using one hot encoding for a variable with thousands of categories will create a thousand or more new columns and will be complicated even if you combine similar categories based on contextual information let's discuss data driven approaches to reducing the space of categorical variables and creating meaningful features let's take a look at the education level variable from the adult underscore incomes data set there are sixteen distinct categories we want to incorporate into our model that predicts income levels above or below fifty thousand dollars we want to reduce these categories in a meaningful way leveraging the outcomes associated with these levels one approach is to look at the proportions of each category with respect to the income which is the outcome variable in this example we can combine prop table with the table function the prop table function takes a table which sells and divides each sell value by the sum of all the cells if you add a 1 after the table you get the value of each cell divided by the sum of the row cells in our example we want the proportion of income within each grade which is the gross sum these proportions give us insights into possible relationships the categories have with the outcome for example we can deduce that lower grade levels are associated with making less than $50,000 in in calendar year for example of the folks that only completed the 10th grade about 93 percent of those individuals make less than 50k a year we order the proportions I correspond to making over $50,000 a year using the arrange function and passing a table that contains the education span income and the corresponding proportions we can leverage this information to create meaningful categories for example we can group categories with similar proportions of making over $50,000 a year in 2/3 order ranges with low education from zero to ten percent medium education from ten to thirty percent and high education containing the rest from 30 to 100 percent we can attach this ad hoc information to our existing income data by using inner underscore joint and attaching the proportions table results with the proportion mappings for each grade level category an inner join takes two data frames and only combines records that have the same link discarding records with no links from either table the link is specified using the buy statement in our example we are linking education from the adult income stable with edy underscore span from a proportions table we now have the proportions associated with our education levels to map on our desired low medium and high education range categories we create a new column continued the new mappings where the low education category contains eight categories from preschool to twelfth grade the medium category contains four categories after graduating high school and the high education level contains a bachelor's degree and more now it's your turn let's