Decision Trees | Data Science with Marco

Key Takeaways

hi everyone and welcome to data science with Marco today we are covering decision trees this is a very exciting subject because decision trees can be used for both classification and regression problems we will cover the basic decision tree and also we'll dive deeper in the more complex methods such as baggy boosting and random forests so let's kick off this lesson with a theoretical overview of decision trees tree based methods can be used for both classification and regression they involve dividing the prediction space into a number of regions the set of splitting rules can be summarized in a tree hence the name decision tree a single decision tree is often not better than linear regression logistic regression or Lda that's why we introduced bagging random forests and boosting to dramatically improve our trees before we move on we need to get familiar with a bit of terminology trees are drawn upside down the final regions are called leaves the point where split occurs is called a node and finally the segments that connect the nodes are called branches here is an example of a basic decision tree with the leaves at the bottom and the branches connecting the nodes now let's see how a regression tree works to create a regression tree we divide the predictor space into j distinct and non overlapping regions then for each observation that falls in region we predict the mean response value of that region each region is split to minimize the RSS it uses a top-down greedy approach also called recursive binary splitting why top down because all observations are in a single region before the split and we're greedy well because the best plate occurs at a particular step to make the best prediction at that step instead of looking ahead and making a split that will give a better prediction later on mathematically we define the pair of half-planes with the formula that you see on top and we seek J and s to minimize the RSS of both planes which is the formula you see at the bottom however this may lead to overfitting as you can see on the right that's why we need to sometimes prune the trees and use cross-validation to prevent overfitting now let's see how a classification tree works it is very similar to a regression tree but instead we predict the most commonly occurring class in a region also we cannot use the RSS since we are dealing with categories so we must minimize the classification error rate the classification error rate is simply the fraction of training observations in a region that do not belong to the most common class however this is not sensitive enough for tree growing instead we use the Gini index which is a measure of total variance across all classes the Gini index will be close to zero if the proportion is close to 0 or 1 which makes it a good measure of no purity a similar rationale is used for cross entropy which is also used for tree growing now that the basics are covered let's move on to more advanced topics on decision trees bagging stands for bootstrap aggregation we know that bootstrap can compute the standard deviation of any quantity we also know that variance is high in decision trees since they are prone to overfitting so bagging is a method to reduce variance and improve the performance of our algorithm bagging involves repeatedly drawing samples from the data set generating be different bootstrap training sets once all sets are trained we get a prediction for each set and we average those predictions to get a final prediction mathematically we express the final prediction like this and you recognize that this is simply the mean of all B predictions this means that we can construct a high number of trees that overfit but by averaging their predictions we can effectively reduce the variance and improve the performance let's see how random forests can also improve the quality of our predictions random forest provides a improvement over bagging by making a small tweak that the correlates the trees again multiple trees are grown but at each split only a random sample of M dictor's is chosen from all P predictors and the split is only allowed to use one of the M predictors typically M is the square root of P now how is that a good thing well in bagging if there is a strong predictor it will likely be the top split and all trees will be similar so variance will not be reduced in random forests because we force a random sample of predictors for each split we avoid the situation also realize that if M equals P then it's just like bagging finally let's cover boosting it works in a similar way to bagging but trees are grown sequentially they use the information from previously grown trees this means that the algorithm learned slowly also the trees fit the residuals instead of the target so the trees will be small and will slowly improve the predictions there are three tuning parameters for boosting the first one is the number of trees B where if it is too large then it will over fit so use cross-validation to find the right number of trees the shrinkage parameter alpha which is a small positive number that controls the learning rate typically it is point zero one or point zero zero one and finally the number of splits which controls the complexity of the boosted ensamble typically a single split works best and we also call that the interaction depth all right so I just get some code down I have a notebook open here also feel free to grab the data set in the description and put it in a folder called data and the data set here is about breast cancer so we are trying to identify patients with breast cancer from a simple blood test so we start off by importing our usual libraries so numpy as NP we are going to import pandas as PD then matplotlib dot by plot as p LT we will also need C born and today we are going to use the function plot confusion matrix as well this is going to be useful to evaluate our decision trees later on finally you use some Jupiter magic to display your plots in the notebook so now let's treat our dataset I will simply define my data path in this case what is in the folder data and the data set is called breast cancer dot CSV then I will use pandas to read the data so data is going to be equal to PD I'll read that CSV and I pass in my data path feel free to use tab at any time to autocomplete enemy will display the first five rows of the data set and there you go as you can see the first eye rows of our data set perfect now we are going to check if our data set is balanced because we have we are in a classification problem so I want to make sure that we don't have too much of healthy patients or patients with breast cancer in the data set that would make it imbalance so I will use the count plot and as you can see the classes are fairly balanced here so we do not need to do some crazy manipulations in this case now it will be interesting to define a function to make violent plots that will allow us to see the distribution of each feature for both classes so it can give us some intuition about the data so for example maybe you will see that most of the healthy controls are younger so defining the function we will need XY and data as parameters and then I will enumerate each Y so we will define a figure then I will set some parameters in this case I will I am simply setting the figure size I want it to be fairly large for you guys to see so I will set it to eleven point seven and eight point 27 I know I am very precise and then you will simply do a violin plot for each one so SN s dot violin plot X is equal to X Y is the colon and data is equal to data that is perfect so now in this case the Y is actually gonna be data columns everything but the last column so I want so in this case the features are actually going to be Y and X is gonna be the target variable because I want to get the distribution of each feature for each class now we can run the function actually passing in our X Y and data and you get the following plots so feel free to study those plots a little bit longer and get an intuition about the data set we are working with now we are going to check for null values to make sure that nothing is missing so for column in data a dot columns I will print the name of the columns or curly brackets call and then we will print the sum of null values so that is data column that is null that some running this cell and you see that we have no null values in this data set that is amazing now we will start some pre-processing first I would like to do some label encoding on the target variable so that we bring it to 1 or 0 so from s calendar pre-processing import label encoder we will initialize the label encoder and then we will fit transform that on the row classification so data classification is le dot fit transform and you pass in data classification now to make sure that everything is right we'll display the first five rows and everything is right now the healthy control is zero and someone with breast cancer will have a label of 1 now we will split our dataset into a training and test set so from a scaler and model selection we are going to import train test split so our target is of course the classification the values dar reshape minus 1 1 and our features is gonna be everything but classification so I am simply going to drop the classification column and I'm going to specify the axis as well access is equal to 1 awesome now actually splitting our dataset note that our dataset is fairly small in this case so I will use a smaller test size than we are used to in this case I will use only 10% of it as a test sighs so you pass in XY test size is equal to 0.1 and a random state of 42 so that we make sure that we get the same results so now let's build our baseline model so it's been our baseline model it will be a simple decision tree classifier so from SK learn doctrine we are going to import decision tree classifier we will initialize the classifier so then the brackets then we will fit the model so call fit and then you pass in X train and y train and finally we will plot the confusion matrix so the confusion matrix will show us how many instances were misclassified so you pass in your classifier X test Y test and I will specify that I want blue colors in this case so it's gonna be a gradient of blue I do not want the grid and I want to show the plot so as you can see we get this confusion matrix and you see that only 3 instances were misclassified in this case now I would like to show you a cool trick because you can visualize your decision with the function plot tree so from a nut tree you can import plot tree and then let lets see what it looks like so you pass in the classifier and I'm gonna specify the max that to five so we will only see five different splits and there you see it so you can see the top split you can see which picture was used what what's the value of this plate and you can also check for the Gini index of each region so that is pretty cool feel free to you know not even pass in Max that so you can visualize the entire decision tree if you want to so now let's try and improve on our baseline model and we will use bagging first so from SK learn dot ensemble we are going to import bagging classifier we initialize the model as always so bagging COF is bagging classifier then we fit the model so without fit pass in your X train and Y trained in this case you need to do dot Ravel and now we will plot the confusion matrix you pass in your classifier pass in X test Y test and again I will specify gradient of Blues to give the pause consistent in the entire notebook I will remove the grid and show the plot and as you can see when we have one misclassified instance in this case so bagging is an improvement over our baseline now let's see how we can implement random forests at this point feel free to pause the video and try it on your own as the process will be very similar to bagging so to what we've done above so from a scalar non example we're going to import random forest classifier we initialize the model so random CLF is going to be equal to random first classifier and in this case I will specify the number of trees I want a hundred trees then we fit the model we pass in our train ties our train set sorry extraneous and then we will plot the confusion matrix so I'll just grab this code right here copy paste it down and then all I have to do is replace bagging CLF with random forest CLF and I forgot to to Ravel the right train sorry about that so why try not Ravel and there you have it we have actually a perfect classifier with no instances that were misclassified that is pretty great however keep in mind this is a small data set it doesn't mean that our model is necessarily very good at this point and finally we're going to implement boosting so from SK learn dot and sample we are going to import gradient boosting classifier gradient boosting classifier so as we have done before we initialize the model then we fit it so boost yellow dot fit X train and Y train and then we will plot the confusion matrix so grabbing the code again copy paste it below and remove a random CLS and paste boost CLF instead and again I forgot the Ravel sorry about that guys my train unravel and then you get this following confusion matrix where only one instance is misclassified which is not better than random forest but better than our base line so that's it for this tutorial I hope that you enjoyed it in the next one we will cover support vector machines so c2