How To Do A Classification Tree in Python | Machine Learning in Python | Databytes

Key Takeaways

hi welcome to this supervised machine learning tutorial on how to do a classification tree in python click the link in the video description to follow along in data camp workspace first of all you might wonder when should i use a classification tree and the answer is that you want the response variables the thing you're trying to predict to be logical values so trues and falses or categorical variable now classification trees have pretty weak predictive power in general so more on that in a moment but it means they have fairly niche use cases so the first case is that you need easy interpretability of the results so for example if you're working in a heavily regulated environment like finance or healthcare you might have to explain your model uh to a regulator and so uh classification trees are great because very easy to interpret what's going on secondly they're really computationally cheap to run so if you've got a very large data set and you're trying to train your model on something low powered like a laptop then classification trees are pretty ideal they also have a very niche but kind of cutting edge use case that rather than try to make predictions with the uh classification tree use them as a feature engineering step so you can use the classification tree to convert continuous features into categorical features as this is feature engineering and you then feed the results into something more computationally intensive but with better predictive power like a deep learning model and as i mentioned because these classification trees have weak predictive power you actually rather than just creating one of them uh you can create an ensemble of classification trees so that just means lots of trees and there are two broad classes of technique for doing this you have bagging methods which basically means you create lots and lots of different trees and then average them somehow or get them voting on like which parts of the tree are best so this is how random forests work there are also boosting methods where you do one tree at a time so you start off with one classification tree and then you sort of iteratively improving the tree so it gets better and better and better and that's how something like gradient boosting works however both those things are out of scope for this tutorial we're going to focus on single trees today in terms of python packages we're going to be using psychic learn today you can also use pie carrot if you prefer let's move on to a case study so we'll be looking at determining the variety of raisin and this is an automated food detection problem so automated food detection means you have images of food and you have to try and work out what food is in that image so this is really useful for things like uh automated production lines in food production uh things like food safety trying to work out what allergens are present in food things like dietary monitoring so apps where you take a photo of your lunch and it tells you how many calories you've eaten and this process traditionally has two steps so the first step is you take your image data and you convert it into numeric features so things like dimension shape and color of your food and then second step is to run a classification model on those features so uh we're gonna be doing step two today so i've got a racing data set from the uci machine learning archive and uh again this is the output of step one so it's got numeric features and we're going to be doing the uh classification part and we're looking at turkish reasons today so we're going to be using pandas for importing the data circuit level modeling and matplotlib for a little bit of work on the plot so let's write some code so we start off importing uh pandas as a pd and then from scikit learn so from sk learn we're going to import several functions so i'm going to do this three times copy and paste paste so from the model selection submodule we are going to import train test split [Music] from the tree sub module we're going to import uh so if i start decision uh so uh in terms of um classification trees that are actually it's it's a type of decision tree there are two types of decision trees you've got classification uh trees classification decision trees when uh the response is logical or categorical and then you've got regression trees or regression decision trees which are used when the response is numeric so in this case we're going to be focusing on the classification trees which are called decision tree classifier and psychic learn all right and then finally uh we're looking in the metrics submodule and we'll take a look at uh the accuracy score and uh the confusion matrix um oh so one other thing in order to plot the results we also want a function called uh plot tree and that is in the uh the tree sub module of second learn and finally we're going to do import map plot lib uh dot py plot uh as plt uh go to repeat pi plot okay so let's run this and we have everything we want now so we're ready to import data set uh so we're going to do uh pd dot read underscore csv and the file is called raisins.csv so csv and let's just assign this to a variable called uh raisins i'm gonna oops i'm gonna copy and paste the name of this you can see the output and here you can see you've got 900 rows so each row represents one raisin or at least one image of raising the features that we're taking out of that and i have to say at first i got a little bit confused about what some of these columns meant so major axis length minor isis length what's that exoticity what's that mean um so if we have a look at the data dictionary it turns out this all makes sense if you think of a raisin as being an ellipse so the major axis length that's the the longest part across the ellipse minor axis length shortest part along the ellipse and xcentricity is how close to circular is it so an eccentricity of one means it's perfectly circular eccentricity with zero means like completely flat it's just a line and then if we have a look on the right hand side of the data set they've got variety and you've got choices of keseman uh brazen and bezny raisin and the other features is other kind of properties of this ellipse so first data manipulation task we need to split the data into a response column and explanatory columns so the response column is the one called variety so let's call it's variable response in there so i'm going to start with the raisins and we're going to get the variety column and then the explanatory variables uh that's going to be we're going to start with raisins again and we want every column except variety so we're going to do drop [Music] columns equals variety so this is going to be a series because it's just one column of um a data frame and explanatory is going to be another data frame which is one less column than raisins next step is to split the data into training testing sets we're going to be doing uh training the model on the training data and filling uh making predictions on the testing date so uh let's call train test split and we're gonna pass in uh the explanatory um variables and the response variable and train test split returns four different things so it's going to return explanatory train uh explanatory test response train and response test let's run that we'll see how we split this up so next step we're going to fit the model to the training set this takes two steps so the first thing is you need to create a decision tree classifier object this is going to be our model object also called mdl shutter model and so this is going to be decision tree classifier and just to make things easier when we come to look at the output i'm going to set um one of the parents is to uh call max depth to three so that's it we're gonna have three different uh layers of depth within our tree um this will become clear when we take a look at the output and something you don't need to do in general just makes it easier to see what's going on now we're ready to fit the model to the training sets we use mdl and then the dot fit method so i'm going to fit this to explanatory train and response train [Music] the output from this isn't very exciting just tells you what kind of model you use what is more exciting is doing predictions though so uh now we're going to fit the uh so we're going to sorry we're going to make predictions on the testing set using model 2 model dot predict and we'll get this time we're going to pass in explanatory test let's run this you see the result uh is a numpy array and for each reason in the uh testing set it's giving you its best prediction of whether it's a keseman raisin or a bezny race so all very exciting however it's not that useful on its own we don't know whether those are right or wrong so we have to do some model evaluation so let's save this uh to a variable i'm going to call it predicted responses let's just run that and so we're going to assess the model performance now there are four possible outcomes either um it was a bezny raisin and we correctly did that so that's a good outcome or it was um a kesem and raisin and we also correctly predicted that that's a good outcome and then there are two possibilities where we did wrong where it was actually cast member we predicted best knee or was actually best but we predicted keseman and we can see how many times each of these things occurred with confusion matrix and in this case we pass in the actual and predictive responses so the actual responses uh that is called uh response test and the predicted responses are it's called predictive responses so we've got a two by two matrix same form as this so in this case you can see 94 times it was actually bezny were predicted basically 95 times it was actually kes member protected guesstimates those are pretty good and then there were 12 and 24 cases where we got it wrong whether that's um good enough accuracy unclear and actually just from looking at this 2x2 matrix it's really difficult to get a sense of like quite how good things are also a better sort of way of um figuring this out is we can use accuracy score which is pretty common metric about the performance of a model so i'll show you the code and i'll explain what this means so let's do accuracy score and this takes the same arguments as confusion matrix so i'm going to copy and paste this so we want the actual responses and predicted responses so let's run this score 0.84 so in this case uh what that does is the sum of the diagonals so the sum of the correct responses 94.95 divided by the sum of all the cases this is how many rows there are on the testing set so 94 plus 24 plus 12 plus 95. so uh you can see uh it guessed right uh 0.84 or 84 of the time so that seems reasonable to me but i don't have a strong sense of like whether that's good enough all right so let's visualize the tree and you can see how the decisions are made so we're going to call plot tree and this takes a few arguments so first of all it takes the model and then we want to set the feature names argument and this is going to be the names of the columns in the explanatory data set which is uh explanatory dot columns takes an argument called class names uh which is basically the the names the different values in the response so bezny and kessiman um we could type this manually but it's sort of better to do this uh programmatically so we can say um it's the unique responses uh so so response dot unique uh so the unity values and responses that's that that's basically estimate i'm gonna you sorted to make sure they are in alphabetical order uh finally the plot is a little bit easier to interpret if you use an argument called filled uh make that true um i will explain what that means when you see the plot let's have a look at this now this actually dumps a load of annoying output there however you can see we have um a plot and it looks like a tree upside down uh so this is the root and then uh these are branches and then you have leaves at the end let's just go back and tweak our code a bit to make sure the output is a bit more readable and i'll go through explaining this in a little bit more detail so first thing we're going to do some a bit of matplotlib codings we're going to use plot dot figure and let's make this a little bit bigger using fixed size so we're going to make it [Music] 10 inches by 15 inches and then if you do plot show at the end it's not going to dump all this text output uh one other thing uh let's change the font size to 11 turn it up to 11 all right boom that's a little bit more readable okay so what happens is you start off with um a row in your data set and then you check the major axis length and if the value is less than 422.423 you go left if not you go right spoiler a classification tree is basically just a flowchart and then you uh if you went left you would check the perimeter see if the value's uh less than 1125 if you went right you check the major axis length that says less than 466 and you just work through each of these decisions into the name decision tree until you get to the end and you're going to end up in one of these boxes and it will tell you uh is this a bezny raisin those are the red ones or is it the kesman raisin those the blue ones so in this case the color is the strength of belief about uh how right it thinks these uh decisions are so in this case dark blue um it's um it's pretty confident that these are all kissman this one is a very pale red so it thinks they're busy but it's really not sure about it anyway that's how the model works underneath when you're explaining to your regulators how your model works you can go through this okay so we've classified some reasons if you want to learn more classification trees are covered um in this uh psychic learn tutorial which goes through uh understanding the decision tree structure so what this plot looks like in a lot more detail um classification trees are covered in so many places in data camp many different courses cover it i would start with the machine learning the tree based models in python course and then they're also some industry specific um machine learning courses like machine learning for finance and python machine learning for marketing and python those also cover um different types of decision trees including classification trees and then there is a decision tree classification workspace template if you want to apply your skills i hope this has been helpful please do subscribe please check out future tutorials this has been wonderful thank you [Music] you