Machine Learning Tutorial Python - 9 Decision Tree

Key Takeaways

We are going to solve a classification problem using decision tree algorithm today. When you have a data set like this, it's easier to draw a decision boundary using logistic regression. But if your data set is little complex like this, you cannot just draw a single line. You might have to split your data set again and again to come up with the decision boundaries. And this is what decision tree algorithm does for you. We will use this particular data set where you try to predict if person's salary is more than $100,000 based on the company his job title and the degree that he has. Now when you look at the data set and when you give it to any human being to solve this problem you will naturally try to build a decision tree in your brain. So first you will split the data set using the company and here you can see what happened is if your company is Facebook no matter what your degree or job title is your answer is always yes you are always getting $100,000 peranom I mean they have a lot of money right now and their stock is going up revenue is going up so they don't mind paying such a high salary but in other two cases uh you have mixed samples So you need to ask further question. For example, for Google I will ask what is the job position and based on that I have further conclusions such as if it's a business manager the answer is always yes. Sales executive answer is no. Computer programmer again I need to split my decision tree. And you can do this iteratively to come up with a tree like this. Now this sounded very simple but in real life you will not have three attributes. which will have probably 50 attributes and it matters in which order you split the three. Right now we chose company first then job title and then the degree in which order you select these attributes is going to impact the performance of your algorithm. So the question arises how do you exactly select the ordering of these features. So let's look at our example. So here we used company first. We might have used the degree instead of company in which case our data set would be split like this. Now observe carefully on the left hand side what's happening is we are getting little bit of a pure subset. What I mean by pure subset is in the case of Facebook all the samples are green. Okay. So this has a very low entropy. Now if you remember the definition of entropy from your school days, it is basically the measure of randomness in your sample. Here there is no randomness. Everything is green. So six green samples, zero red, hence low entropy. Here there is some entropy but still majority of the samples are red. Okay. Whereas on the right hand side for this case four red four green means there is total random randomness it is 50/50 hence my entropy is one okay here it's little better entropy is little low so overall I'm thinking if I use company as shown on the left hand side I will have a high information gain okay whereas on the right hand side I have low information gain Hence you should use an approach which gives you high information at every split. Hence we chose company as the first attribute and in the further split also you can use high information gain uh criteria to divide it further. There is another term that you hear often when you're dealing with decision tree which is genie impurity. Now this is nothing but an impurity in your data set. For example, when I split my sample like this at the bottom, most of the samples are red whereas one is green. So this is almost pure but there is little bit of impurity. All right, it is sort of similar to entropy. I'm not going to go into mathematics too much. You can read articles on it. We'll straight away jump into writing code. I launched my Jupyter notebook and loaded uh the same data set into my data frame. You can see I have the same CSV file that I'm loading into my data frame. Uh now the first step is once I have my data frame ready, I want to divide it it in between the target variable and the independent variable. So I will call the target v independent variable data frame inputs and I will just drop see this is my target column okay target variable so I'm just going to drop that and I will say access is equal to columns [Music] Okay. So once I execute this, what happening is my input looks like this. So it doesn't have the last column which is my answer and my target looks like this which is my last column. Now by this point you all know that machine learning algorithms can only work on numbers. They cannot understand labels. So what we have to do is we have to uh convert these particular columns these three columns into numbers. And one of the ways is to use the label encoder. So from sklearn.processing pre-processing I will all right I hit tab so it was autocomp completing but it was slow but see if you hit tab again it's not working this is sometime see now it worked it's funny all right once Once I uh import label encoder, I am going to create the object of this class and I have three columns. So I have to create like three different objects. Okay. So first is Ali company. The second one is Ali job and then degree. Once you have these three, what you do is in your inputs data frame, you are creating one more column. And this is how you create extra column in a data frame. You call fit and transform method on your company column. And you can do the same thing for your job and degree column also. So here you have job your degree. Once you do that and when then when you print head this is how your data frame going is going to look like. It has three extra column and we have label encoded your label columns into numbers. Next step is to drop those label columns. So I'm going to create new data frame here and just say drop. And you can drop multiple columns at the same time. Axis is equal to columns. And when you look at your inputs and data frame, what it did is dropped all the label columns. Now all you have is numbers. So Google it encoded as number two. Um the second one was ABC Pharma which was encoded as zero and Facebook is encoded as one and same thing for like job title and degree. It just assigns different numbers to different labels. Now we are ready to train our classifier. So as usual I am going to import some module. uh for decision tree you import uh the tree from your skarn uh library and then your model is nothing but tree dot decision tree decision tree classifier and then you can now train your model. So you can call fit here and I'm going to call inputs n and my target variable. So it train my model. Now I'm not using test train split here just to keep things simple but ideally you should split your data set into training set and test set 80 2030 whatever ratio you prefer. All right. But I'm just keeping it very simple. Here it use criteria as genie impurity by default. You can change it to entropy also. Again for math I'm not going to go into very much detail. You can Google it to know the difference between genie and entropy. Uh these details are abstracted by skarn library. So you're fine. Although knowing math always helps uh in terms of uh what kind of criteria you should choose for a given problem. So I I still suggest going through that. All right. Now my model is ready to predict. So the first thing I'm going to do is predict my score. All right. And the way you predict your score is you supply your input and target data set. Now pause this video for a moment and tell me what is going to be your score. the score is going to be one because I'm using the same data set uh which I use for training and my data set was also very simple. So I was expecting that it will do very okay. It will be very accurate with my prediction. Uh hence the score is one. In real life when you have complex data set your score will be less than one. Okay. So now let's do some prediction. So I'm going to do predict. All right. What are we going to predict? So let me predict a salary of person working in Google. Sales executive is his job and master's degree. Okay. So let's see. So that's number two row. Okay. So 2 to 1 2 1 All right. Is expecting 2D array usually you supply data frame. So I'm just going to do this and it says zero means the person who is working in Google sales executive is his job. Master degree his salary is not going to be more than $100,000. And by the way, just a disclaimer, I just made made up this data set. In reality, Google says executive might be getting much more than $100,000, but I just made it up. All right, so that's a little disclaimer. How about business manager? So business manager number label encoded number is zero. So his salary is one. All right. So we are doing perfectly. All right. Here you can uh have this model and you can do further prediction uh using uh the train model and by calling a predict method on that. All righty. Now the most important part which is the exercise. So I expect all of you to work on the exercise once you learn this concept because just by watching the tutorial you're not going to learn anything. So you must do an exercise on your own. I have a Titanic data set. So this is showing the survival rate of passenger uh in a Titanic crash. This is the real data set and you can get this CSV file uh by clicking on a link in the video description below. So that link contains the Jupyter notebook that was used in uh this tutorial and it has exercise subfolder. Within that you have Titanic. CSV here you should ignore all the red columns and use the remaining columns to predict the survival rate. So here the survived column is your target variable and you have to predict the survival of a passenger based on the class the sex age and the fair that he paid before on boarding Titanic ship. Okay. So that's what you have to do. come up with uh the score of your model and post your score as a comment in below and I will verify your answer and and we'll see uh how well you can do with it. All right, that's all I had for this tutorial. Thank you very much for watching. Bye.