Basic Statistical Functions in R | Complete Tutorial

Key Takeaways

Welcome. Let's say you're a data analyst at a school and you're trying to analyze and visualize different student performance metrics and how they affect their assessment scores or in this case test score. Let's first work with Tidyverse and load this library. Remember, if you see an error when you load the library, it might be that the library is not installed. Remember to use install.packages for this. Let's go ahead and install the library. We'll also load our data set. In this case, student_performance.csv. Let's take a look and visualize our data set. We have some columns such as student ID, the school type, tutoring as a boolean, gender, absence groups, and test scores. Let's first visualize our data using str and summary or structure and summary. As you can see, structure says a overview of what type of columns we have and some of the values that we can see. Summary gives us as you can imagine summary statistics for some of the metrics that we have that are numerical such as student ID as well as test score. And then it also gives you a small description of the non-numeric ones or in this case characterbased categories. Now let's go over a new function s apply. In this case you can see it over here. We'll make two different vectors. One for numerical variables and the other one for categorical variables. Let's go ahead and clean this. We'll apply the names function on the data to get the names of the variables that we are interested in. We'll then do s apply to apply our data. The function for s apply and search for is numeric. In this case, this function will find all the numeric columns in the data. And as you can imagine, the categorical variables are the characterbased columns that we have. Let's run this and see what we get. Okay, we've called them in. Let's invoke the variables. As you can imagine, the numerical variables that we have are student ID and obviously test score. Let's go ahead and do the same for categorical just to be sure. Small typo here. Not too bad. All right. As we can imagine, school type, whether they're being tutored, their gender, and what type of absence group they fall onto is there. Excellent. Now, let's move into some of the statistics that you can use for these variables. In this case, test scores is the numeric variable that we're interested in analyzing. We'll combine the mean and median functions that we can see with summarize where we'll be able to summarize based on these metrics and combine it with group by. In this case, we're going to try to understand how each tutoring group has the mean test score and the median test score. Let's run this. Excellent. Now, we have some grouping happening based on whether they're being tutored or not. As you can see, the mean score for the non-tutor group is 67.4 and the median score is also the same. Now, for the group that is being tutored, we have a mean score of 75.4 and a median score of 75.3. So, as you can see, there is an effect on tutoring and how it increases our test scores. Now let's do the same exercise but for standard deviation and variance. And in this case now we're going to be doing at the school type to see if there's a difference in the standard deviation metrics and the variance for the school types. Let's run this. As you can see this is much smaller standard deviation for private school types as opposed to public school. Almost twice as much standard deviation. All right. That means that we can expect much less variable scores in private school students compared to public schools. Now, we've calculated a lot of these metrics, but let's go ahead and visualize them. We'll do a simple ggplot where we combine a box plot where we'll plot the tutoring group versus the test scores and color them based on tutoring. We'll also add some titles such as test score by tutoring as well as the x and yaxis labels. Let's run this and see how it looks. All right, beautiful. We have our box plot with our scores and some coloring. Let's also note that in a box plot, this long line is the median score. We can also see that we have a couple outliers denoted by these points as well as this point which helps us identify any extreme both low and high outliers. This can help us identify any entries or students that might be at risk. In this case, these very low test scores as well as this very high test score that's 100%. Which in other words might be a true value or it could be indicative of someone who might have cheated. Now let's try and look at a different type of plot, a bar plot. We'll be plotting the absence group types and try to see a histogram based on these. As you can see, we have pretty much the majority of the school close to this number 100 students are in low absence group, which is great. This is what we want. We obviously want students to be in school. This allows us to try to identify if we have a problem with absence categories. In this case, there's a decent amount of moderate absence groups. Might be worth interesting what might be happening or why students are not attending school. Now, let's recap what we did. Obviously, we loaded our data set and we try to visualize some of the summary statistics using STR and summary. We've also learned about s apply where you can apply a specific function such as is numeric and try to retrieve that based on all the entries in a data set. In this case to fish out any numerical variables and any categorical variables. Then we use the group by summarized combination of functions to try to retrieve mean and median for a numerical variable and try to visualize it in a nice table. Remember, you can always group by different columns to try to understand and see if there's anything interested in. We've also done the same that we did for mean and median, but for standard deviation and variance using SD and VAR functions. Finally, we've recaped on how to use these metrics and how to plot and visualize them on a box plot and a bar plot. Congratulations, you're doing great. Keep up the good work.