StatQuest: Random Forests Part 2: Missing data and clustering

Key Takeaways

Random forest, part two. Hip hip hooray, it's true. StatQuest. Hello, I'm Josh Starmer and welcome to StatQuest. Today we're doing random forest part two and we're going to focus on missing data in sample clustering. To be honest, the sample clustering aspect of random forest is my favorite part, so I'm really excited we're going to cover it. Here's our data set. We've got data for four separate patients. However, for patient number four, we've got some missing data. Random forests consider two types of missing data. One, missing data in the original data set used to create the random forest, and two, missing data in a new sample that we want to categorize. We'll start with this one. So we want to create a random forest from this data. However, we don't know if this patient has blocked arteries or their weight. The general idea for dealing with missing data in this context is to make an initial guess that could be bad and then gradually refine the guess until it is hopefully a good guess. Because this person did not have heart disease, the initial and possibly bad guess for the blocked arteries value is just the most common value for blocked arteries found in the other samples that do not have heart disease. Among the people that do not have heart disease, no is the most common value for blocked arteries. It occurs in two out of two samples. So no is our initial guess. Since weight is numeric, our initial guess will be the median value of the patients that did not have heart disease. In this case, the median value is 167.5. Here's our new data set with the filled-in missing values. Now we want to refine these guesses. We do this by first determining which samples are similar to the one with missing data. So let's talk about how to determine similarity. Step one, build a random forest. Step two, run all of the data down all of the trees. We'll start by running all of the data down the first tree. Dooby dooby dooby, dooby dooby dooby, dooby dooby dooby dooby dooby dooby dooby dooby dooby dooby boo. Notice that sample three and sample four both ended up at the same leaf node. That means they're similar. At least, that's how similarity is defined in random forests. We keep track of similar samples using a proximity matrix. The proximity matrix has a row for each sample, and it has a column for each sample. Because sample three and sample four ended up in the same leaf node, we put a one here. We also put a one here, since this position also represents samples three and four. Because no other pair of samples ended in the same leaf node, our proximity matrix looks like this after running the samples down the first tree. Now we run all of the data down the second tree. Badab boo bap badada boo bap badab boo bap badada boo badab boo bap badada boo bap badada boo bap boo. Note, samples two, three, and four all ended up in the same leaf node. This is what the proximity matrix looked like after running the data down the first tree. And after the second tree, we add one to any pair of samples that ended up in the same leaf node. Samples three and four ended up in the same node together again. And sample two also ended up in that same node. Now we run all of the data down the third tree. And here's the updated proximity matrix. Only samples three and four ended up in the same leaf node. Ultimately, we run the data down all the trees and the proximity matrix fills in. Then we divide each proximity value by the total number of trees. In this example, assume we had 10 trees. Now we use the proximity values for sample four to make better guesses about the missing data. For blocked arteries, we compare the average proximities for yes to the average proximities to no and select whichever is greater. In this example, we only have one proximity value for yes, 0.1. So the average proximity for yes is 0.1. For no, we have two proximity values, 0.1 and 0.8. So the average proximity for no is 0.45. No has a way higher average proximity, so we'll go with it. For weight, we use the proximities to calculate a weighted average. In this case, the weighted average equals sample one's weight sample one's weighted average weight. Sorry if there's any confusion between a patient's weight or a sample's weight and the weight used in the weighted average. To calculate that weight, we start with the proximity for sample one divided by the sum of the proximities. So sample one's weighted average weight is 0.1. Here's the weighted value for sample number two, who weighs 180. Here's the weighted average value for sample number three, who weighs 210. Ultimately, the weighted average of weight is 198.5. And remember, the weights that we used in the weighted average were based on proximities. Now that we've revised our guesses a little bit, we do the whole thing over again. We build a random forest, run the data through the trees, recalculate the proximities, and recalculate the missing values. We do this six or seven times until the missing values converge, i.e., no longer change each time we recalculate. Bam. Now it's time for an interlude of awesomeness. Let me show you something super cool we can do with the proximity matrix. This is the proximity matrix before we divided each value by 10, the number of trees in the pretend random forest. Just for the sake of easy math, imagine if samples three and four ended up in the same leaf node in all 10 trees. Now we have a 10 here and here. After dividing by 10, the number of trees in the forest, we see that the largest number in the proximity matrix is one. One in the proximity matrix means the samples are as close as close can be. That means one minus the proximity values equals distance. Closest can be equals no distance between. And not close equals far away. This is a distance matrix, and that means we can draw a heat map with it. If you don't know what a heat map is, check out the StatQuest. And we can also draw an MDS plot with it. And if you don't know what an MDS plot is, well, check out the StatQuest. I think this is super cool because it means that no matter what the data are, ranks, multiple choice, numeric, etc., if we can use it to make a tree, we can draw a heat map or an MDS plot to show how the samples are related to each other. This is awesome. Triple bam. Okay, enough fun stuff. Let's get back to the missing data problem. At long last, we'll talk about the second method. This is when we have missing data in a new sample that we want to categorize. Imagine we had already built a random forest with existing data and wanted to classify this new patient. So we want to know if they have heart disease or not. But we don't know if they have blocked arteries. So we need to make a guess about blocked arteries so we can run the patient down all the trees in the forest. The first thing we do is create two copies of the data, one that has heart disease and one that doesn't have heart disease. Then we use the iterative method we just talked about to make a good guess about the missing values. These are the guesses that we came up with. Then we run the two samples down the trees in the forest. And we see which of the two is correctly labeled by the random forest the the times. This option was correctly labeled yes in all three trees. This option was only correctly labeled no in one tree. This option wins because it was correctly labeled more than the other option. Bam. We filled in the missing data and we've classified our sample. Hooray! We've made it to the end of another exciting StackQuest. If you like this StackQuest and want to see more, please subscribe. And if you want to support StackQuest, consider contributing to my Patreon campaign, becoming a channel member, buying one or two of my original songs or a t-shirt or a hoodie, or just donate. The links are in the description below. All right, until next time, quest on.