R Tutorial: Background on modeling for prediction

Key Takeaways

previously you were introduced to an example of modeling for explanation understanding what factors might explain teacher evaluation scores as given by students at the University of Texas Austin let's now consider an example of modeling for prediction the data set I'll use is the house sales in King County USA available at cackle comm it consists of homes sold near Seattle Washington in 2014 and 2015 I'll predict the sale price of houses based on their features such as size as measured by square feet of living space where one square foot is approximately 1/10 of a square meter condition number of bedrooms year built and whether it had a view of the waterfront just as with the evals data set I've included this data in the modern dive package which I preview using the glimpse function observe there are 21 thousand rows representing houses and 21 variables as before let's perform an exploratory data analysis or EDA of the outcome variable price recall the three approaches to EDA looking at the data visualizations and summary statistics since I've just done the first approach let's now visualize our data just as with the outcome variable score from our explanatory modeling example let's get a sense of the distribution of our new numerical outcome variable price using a histogram first let's look at the x axis tick marks since E Plus 0-6 means 10 to the 6 or 1 million we see that a majority of houses are less than 2 million dollars but why does the x axis stretch so far to the right it's because there are very number of houses with price closer to 8 million you say that the variable price is right skewed as exhibited by the long right tail this skew makes it difficult to compare prices of the less expensive houses recall that you saw something similar in the intro to the tidy verse course when visualizing the variable country population from the Gapminder dataset you visualize the relationship between countries life expectancies and populations using a scatterplot similar to this one because the populations of the two green dots corresponding to India and China were so large it was hard to study the relationship for less populated countries to remedy this you rescale the x axis to be on a log 10 scale now you can better distinguish points for countries with smaller populations furthermore horizontal intervals on the x-axis now correspond to multiplicative differences instead of additive ones for example distances between successive vertical white grid lines correspond to multiplicative increases by a factor of 10 now I'll do something similar where I log 10 transformed price using the mutate function to create a new variable log 10 price let's view the effects of this transformation on these two variables observe in particular the house in the sixth row with price 1.2 to 5 million since 10 to the 6 is 1 million it's log 10 price is six point zero 9 contrast this with all other houses with log 10 price less than 6 I'll treat log 10 price as our new outcome variable I can do this since log transformations are monotonic meaning they preserve orderings so if house aides price is lower than house B's then house a z' log10 price will also be lower than house bees log10 price let's take the earlier code to plot a histogram of the original outcome variable copy-paste and tweak the code to plot a histogram of the new log10 transformed outcome variable observe that after the transformation the distribution is much less skewed and in this case more symmetric and bell-shaped although this isn't always necessarily the case you can now better discriminate between houses at the lower end of the price scale now that you've seen a log transformation was warranted for the outcome variable price let's see if the predictor variable size as measured by the square feet of living space warrants a similar transformation