Handle Categorical features using Python

hello today we'll be seeing that how we can handle catechol features and we will also try to implement it by using Python so whenever we get a machine learning use case and the data set that we see most of the features you will find this kind of category Jers so let us first understand what is catechol features and then how we can what are the different ways to handle it and then we'll also see that what are the various tools like scalar and pandas and how we can do that so to begin with guys category features like whenever you are you whenever you see a data set right some of the features some if I take an example of gender in gender you may have seen that you have different type of categories like male females not specified and all the other records will basically be repeated by this so here some of the properties are that you will be having some fixed set of values you know fixed set of values in all the records it will be either male female not specified okay now in this particular example what we are seeing is that suppose if I take an example of gender so first step always remember that machine will never be able to understand this string categories altogether you know it is very difficult from them to understand now if we take some other cats are the features of example in other features will be continuous random variables or discrete random variables in that suppose you will be having some major use suppose if you you will be having some amount values or salary values right this all values are basically understood by machine learning algorithm the reason they are understood is that because machine learning algorithm does some kind of internal mathematical calculations a lot of equations you know in the machine learning algorithms that are basically applied if you're considering regression kind of problem statements or a classification kind of problem statements in those but if we have a feature which will be having this many fixed set of categories and if you try to give this categories directly to a machine learning algorithm it will never be able to understand you know so this particular raw data when I say the terms of category Chur is not suitable for the machine learning algorithm to process so you we'll be trying to discuss how do we you know apply feature engineering on categorical features and how to handle it so the first step that once we see a catechol feature we basically have to apply of something called as label encoding now when I say label encoding what it is label encoding a sim is a simple concept in where where in the number of features that I have that are present in this particular feature column you know so suppose there is a feature called a there's a record where it has this particular feature has a value called as male now this male gets converted to a new level or gets assigned to a label like zero then female will get assigned to another label like one not specified will get assigned new level unless - now when we are considering this right now all this particular labels will get repeated so again not specified will be having to value female will be having one value now this particular label encoding has a very big problem now what is the problem let us discuss about it let us consider that I have a feature which is a binary category so when I say binary category feature let me say that I am having a feature f1 when my output is just like mean I have only two different types of values everywhere you know male and female male and female male female in all the records now usually when I try to apply a label encoding this will get converted to new labels like 0 & 1 now when I have just zeros and ones you know and then I try to convert this into a label encoding and pass it to a machine learning algorithm the machine learning algorithm will be able to distinguish it very easily because they are just two categories one is 0 and 1 is 1 so when the value is 0 it will consider that the value we are talking about means and when the value is basically 1 BF is talking about senior but in the case of multiple categories feature now you can see here the values like 0 1 2 now when the machine learning algorithm will receive this particular inputs like 0 1 2 what to consider is that 2 is greater than 1 or 1 is greater than 0 because altogether you can see that in machine different mathematical calculations will be happening so this basically says that not specified is greater than female and female is greater than me so this is not a correct scenario altogether you know so we should always remember that we should not stop only in labor encoding for multiple categories like this kind of example for a binary catechol features I think we should we should basically stop in the labeling pudding you know because here I'm actually getting the values as zeros or ones which is very very easy for a machine learning algorithm to distinguish but if you have multiple categories we should not stop in label encoding and we should begin a new step which is called as one not including now one hot encoding applies a simple mechanism wherein what it does is that based on the number of categories present over here it will divide those into that many number of columns so here I have male female not specified now here you can see that it has got converted into three columns that is male female and not specified now wherever the values mean you know that gets that particular column in that particular column wherever the male column is that that value gets one remaining all gets feed no similarly whenever there is a female female particular feature or column gets one remaining all gets cos now the machine learning algorithm will be able to distinguish clearly because the feature that is having the one value or the column that is having the one value is basically indicated with that particular so here whenever I have one over here it is basically male whenever I have 0 over here it is female oh sorry whenever I have one over in the female column that is basically indicating that this is faithfully representing this whole record is representing a female row you know female or basically it is all about the feeling know then after doing one hot encoding guys it is always remember that suppose after doing one hot encoding oh yeah male female not specified I'll just try to drop this column now see whether do we have any impact of a dropping this column here you can see that I am having one 0 right that basically indicates that this is the male mail row then I am having female as 1 over here and 0 over here that basically indicates this is a female row again if I have both the value as zeros what does this basically indicate this basically indicates my third column that is not specified now since it is representing this thing we can clearly say that this particular column is not required because whenever the values of male and female is 0 that is basically representing my third column and this particular condition is basically called as dummy variable trap dummy variable trap basically indicates that you don't have to use all the columns you just have to use n minus 1 column now what is N and basically indicates that how many number of unique categories I have inaudible speech by using this by doing this you will be able to solve this particular problem and this particular problem is basically a nice dummy variable trap right now going ahead guys I will be showing you how to code how you can code is by taking a wonderful example of a data set and then I'll be writing the code line by line so it will be very official for you and how we can actually do it now to go ahead what I will also suggest you is that we basically have two different ways you know one by using SQL on library and one by using and off slide now in this both the techniques what I felt is guys pandas library is quite easy it is very very easy whereas Escalon like the D it requires some lines of codes to write you basically have to use label encoding label encoder one hotting folder one hot encoder libraries to do this so I would suggest you to use pandas because there is a function which is called as get underscore dummies and this will help you to reduce to within a line of code you convert your whole category feature into dummy variables you know like this within the lines now let us go ahead and try to see how we can do it Factory to begin with just go and show you how to do it so initially I'll try to import on dowsers PD because I'm going to use pandas a lot then I'm going to use numpy as NP doing this I just read a data set which is like PD dot read underscore C is me don't worry about the cold guys I'll be providing you the code in my dieter Blaine so I have a data set which is for a start up dots expand underscore extends dot CSV if I go and see my DX dot head now here you can clearly see that I'm having some columns like R&D spend administration marketing spend state and profit on this this is basically a regression kind of problem where we need to determine what is the profit based on the other parameters like R&D administration marketing your state so here state you can see that it is clearly a category feature and it has only three unique features so what we'll do is that I'll try to convert this into owe me variables okay in order to do this I'll just use I just used ear PD dot get done let's put amis so I may get some error let's see I hope I may not get any errors because know we can quickly do this otherwise again we'll get some stuck in some error is done but error is actually good now you will be able to see what kind of errors you are making so that to make your profit map and there is also a parameter I just don't remember parameter okay let's go first Oh drop underscore first is quite true if I set it that basically means that it is going to convert this particular state columns you know now in this particular state column you can see that there are three unique categories that so that will get converted into three columns but from that three columns I don't want one column because to solve the dummy variable times so for that I'm setting up this drop and let's go first is but you can see that once I execute it I just have Florida and New York column I'll just let me write it so that I get just get the top five records you know so drop and let's go first is equal to true over here Florida and New York over here the California column is dropped and this works in an ascending order so let me just save this whole law dummy variable in a variable like DF one okay I'll just create a variable DF 1 now in this particular DF one I have DF p dDOT's tetanus condoms and now if you want to see the DF one is basically all my dummy variables for the state column now the next thing is that I will just do a concatenation operation in the concatenation operation I am just going to give my DF comma so I'll just put DF 1 former DF and my axis will be one you know I just need to append it append this new column that is getting created with my DF so it's this gets executed this will get executed fine so here it is it is got executed you can see that Florida New York has got appended in front of this but I still have a state column first of all let me replace it back to my DF so that my DF will now look like this and just remove this head part it so this is my DF but I still have the state column guys I've converted this into categories but I need to drop this also what I can do is that I can just write DF tour drop I can just give my state column and I can suggest my ax is equal to 1 and I can also do any place is equal to true you just don't directly try to put in place is equal to first check you know whether it is working or not and then you try to put it I can see that the state column is not there I can just write DF forehead perfect now I have got my now this all are my independent features what I see from here and profit is my dependency so let me just convert this into independent independent I think from here you will be able to apply your linear regression algorithm I'll just write : -1 and I'll write Y you have got a log colon comma I'll just give my as this is actually present 1 0 1 2 3 4 5 so my output is basically present in my insistence and the next thing is that I will just try to do a train to split from a scale on dot order selection is a selection in poor train test plate yeah now I'll say say X underscore train X hundreds of tests - foot crane - good test and I'll just write inter split I'll just give my X comma comma Y comma just my test and let's go size as 0.25 right so here it is it has got executed successfully I'll just apply my scale on dot linear regression models for that it is present in linear model I'm going to import linear regression here I'm going to create a object called as a regression which will be having a linear regression and then I'll just say a regression dot said here I have my extreme um white friend it it's got executed what I'll do is that I'll create my white bread that will be my variable and I'll just say regression dot predict and let me put my X and for test what I'll do I'll just check my R square value because I need to see what accuracy to say so I'll just write from a scalar and dot metrics the import I have to going to import art two squares here I'm just going to write score is equal to r2 square and here I'm going to compare that Wireless for test and this book that that's it the that's core value I can see that I'm getting 0.93 and it is very good score because it is very very nearer to one and this is basically my regression problem statement but the main fact is that I try to solve or what is a dummy variable and this is how you solve it army variable but there is again one more problem is it a dummy variable let me consider an example suppose in one of the feature I have like pink code now and suppose in pin code I have values like 2001-2002 this is just an example 2003-2004 and I have suppose 500 unique pin codes throughout know how to solve this kind of problems I cannot just turn word this into 499 columns as I said that n9 n minus 1 columns will get created but is 500 unique in modes right so there is another mechanism for this are very beautiful hack what I do is that I convert this category feature of pin code whenever I have so many into numerical feature shocked by hearing this how can I convert a caracal feature into numerical feature yes we can we can convert it so let me just show you suppose R here I have a feature where my values is 2001 2002 2003 2004 again 2001 this is again repeating again 2002 but I have unique 500 now with respect to 2001 suppose I have my output like 1 0 0 1 0 okay and one now suppose I want to convert this to 1 2001 category feature into unique integer or into a numerical feature so what I'll do I'll just write f1 - how we just need to find out for 2001 how many times it was 1 suppose if I had 2001 here also equals 1 away so from this I can see that 2 2001 is basically having one month so I'll increase I just make sure I will just count that suppose the count is 2 divided by a total number of times 2001 Smith you can see that total number of time 2001 is Penan but this 2000 you had zero so I did not increase the count from 2 to 3 so this will be like 2 2 divided by 3 and this will somewhere be like you know or 33.2 tb7 6.6 yeah so this feature for this 2001 you will get converted into a numerical feature like point six every way now similarly we do this and we do this for each and every feature each and every capital feature you know or the pin codes and this is just an example where in generate very good example because in most of the scenarios suppose is here having a state column area of 100 states and inside that you should basically try to use this technique and convert it into a numerical teacher trust me it will definitely work this because recently I I was doing a problem statement where initially I was getting 78 percent accuracy but by just implementing this technique it increased by plus seven percent you know and my overall equally she was eighty five percent which is of very great increased no very good increase and it is still in production it is working very very good know so apply this particular technique for multiple categories where you have many category features like pin code or number of state number of countries this particular technique where you can merge your future category into a numerical feature simple just count how many times one it is present divided by total number of this particular pin code and this is basically the end of auto handled category feature this I hope you liked this particular video or make sure please to subscribe the channel share with all your friends never give up keep on learning um data science is the thing that is happening I'll see y'all in the next video have a great day have a wonderful day guys thank you