Exploratory Data Analysis: Feature Selection and Data Imputation

Key Takeaways

hello everyone um welcome to the session again and in this video we will be continuing our project on predictive analysis and diabetes data if you remember in the last session we have done a great implementation with respect to understanding the data part and we have seen that uh one issue is there in this particular data set which we have figured out that although this data doesn't contain missing values or null values I would say still it contains some weird zeros which doesn't make sense uh because any human being cannot have BP value as zero or let's say skin thickness value as zero and at that point of time I told you that when we will move towards the Ed step we will figure it out how can we solve this problem and there are multiple things in this video you will be able to get the insights the only prerequisite is that to understand this video because this is going to be a long we you you should have a good understanding of stats Concepts you should have a good understanding of box plots outliers because this video is majorly focused on the implementation per se and if there is anyone who is not having the idea with respect to the concepts of what is distribution symmetric non- symmetric those people or those Learners might face some trouble I'll try to explain the concepts in a great detail but still if you're having any trouble with respect to the STS concept so at that point of time you can apply reverse engineering what does that meaning you can pause the video try to understand the concept behind the stats if I'm talking about somewhere then again come back to this video and check out the further part of the uh video okay so this is how maybe you can learn the EDF so let's get started uh please make sure you will watch this particular part of the uh video till the very end because this is one of the most important video in the entire project part okay so let's get started here now before going towards the data imputation part let's first of all understand the correlation uh heat map means how the two features are correlated what is the significant value which we will be able to get and the simple way is to get that is data frame which we have created do core if you'll just run this function by default it will give you that with respect to the two features that we have what is the correlation coefficient value which we will be able to get and it's really easy to determine that isn't it I just have used the statistical function to calculate the correlation value between the two features that we have now the question which you can ask from me is that Pria white is required why you are trying to calculate the correlation coefficient value among the two features that we have the only reason is uh those who are not aware about the significance of this correlation coefficient for them I just want to explain to you in a very brief way whenever we are talking about correlation coefficient even I'll try to create a heat map also further in this particular session itself so whenever we're talking about correlation coefficient first of all do remember that it's will you refer uh mathematically how we can calculate I'm not going into that part if you curious to learn that thing maybe you can just refer Wikipedia everything is available there the only thing which I want to explain you here is the internal intuition behind getting this particular values so what I told you that the range of this correlation coefficient will always lie from minus1 to + one this is what you will observe now when I'm saying that the two features are highly correlated it simply indicates that either their value is very approximately equals to negative of one or it is equals to approximately positive of one and both of them indicates that somehow the two features that are available to us in a given data are giving similar kind of information to give the prediction to the model whether the person is diabetic or not so why to take both of them if let's say I'm saying that there is feature F1 and then we have feature F2 and let's say they are very highly correlated maybe positively maybe negatively so let's say their value is let's say 0.97 means 97% they are giving or signifying a similar kind of an information to a model prediction to a model prediction so the question is that why we will take both of the features that's the biggest question so at that point of time any one of the feature what what we usually do we can remove and that's why it is also a part of a feature selection Technique we can say it's also a part of a feature selection technique because higher the number of features we have which we also called as Dimensions it will take more time to my model to train so we what we want is we want distinguish information to the model so that it will be able to generate something great to us so that we will be able to get a better prediction that's the only reason we are trying to create a correlation coefficient a when I'm saying that the range is either negative of minus1 to positive of minus1 so when you're saying the two features are negatively correlated means their value is negative of let's say minus one or minus of 97 so in that case it clearly indicates those two features are inversely proportional to each other means if let's say you will be having feature F1 and here you will be having feature F2 so as in when the feature F1 is increasing your value of feature F2 is decreasing this is how the flow is going like this kind of a curve is there so as in whenever your feature of F1 is increasing value of feature F1 is increasing FS2 is decreasing but this is something which you can say is neg meaning of negative correlation coefficient means they are correlated the impact is negative means one value of the feature is increasing other one is automatically decreasing so if FS1 is increasing FS2 is decreasing or it can be vice versa then another concept is a positive correlation coefficient what is the meaning of that simple meaning I think now everyone understand this point it it will be vice versa where you are saying that when F1 is increasing my value of FS2 is also increasing means somehow they're directly proportional to each other something like this right this is where you are saying when F1 is increasing my second feature is also increasing they are correlated and that to positive positive manner positively manner so this is the literal meaning of positive correlation coefficient positive correlation coefficient so this is the only importance of why we are doing this calculation in the implementation part per se so usually you will observe that the best practice will be that we will try to create a heat map of the same so what I can do is I can create a heat map where I'm saying pl. figure where the fix size I'm just defining that figure size should be 15 cross 15 so that it will look broader and good after that we are using the cbone package to create a visualization dataframe do core if you can see I'm just calling the same function which I have just defined in the above part so you all know that this will simply create a matrix where I will be having a numbers of correlation coefficient uh value after that I'm saying annotation to be true as of now let's not take anything and we will see what will happen by default annotation will be false we will have to make it true I'll show you the difference between these two after that I'm trying to save this figure so that if let say in future I'm trying to create a presentation and I want to showcase to the clients that how I try to decide the feature selection part I can showcase to them that's the only reason and then I'm using pl. show now let's wait for a while and you will observe that this is how my correlation coefficient heat map is coming up with a different different colors that we have I hope it is clearly visible to each and everyone right so now here if you can see the only problem is that I don't have the values available in my chart right so the values are not there because I have if you remember remove the annotate value as true so what you have to do just make it like that if you will make this parameter as true now you will observe the difference between the previous uh you know picture and this picture you will observe now I will be having numbers as well can you analyze this so for every particular feature you have the values available now right so we have uh for every two features the value of correlation coefficient and now if you will be able to see as far as I can see in this particular complete picture and I really want you all to also create the same with me so you will observe that no features no two features have high correlation coefficient at least in this particular data set it means for us every feature is important and we cannot remove any single feature for now so that is the uh real indication which we will be able to get it from here it means that when we will do the data imputation we have to take care for each and every feature that we have that's the only reason why I have done correlation coefficient part first before jumping towards the data imputation part right so this is the overall idea behind the correlation coefficient and I hope the literal intuition behind the same is also clear to everyone you will learn a lot of maths in this Eda that's for sure right now another important thing which you should know or you should be aware of is the descriptive statistics of the given data this is all so very important because this usually help us to understand what is the mean what is the standard deviation what is the median what is 75 percentile what is uh 25 percentile uh what is the count is there any missing Val there there or not minimum value maximum value everything you will be able to understand within one go so what you have to do just you have to mention data frame do describe function and this function will help you to understand the complete description of a given data where if you can see now I will be having a count I will be having a mean I will be having a standard deviation minimum maximum and all the quartiles that we have q1 Q2 and Q3 now again if you can see if you remember in the last video uh we have talked about that how we will be able to evaluate the missing values term where we have used the isal function which will return the either value as true and false and after that because I have used the sum function so it is giving me how many missing values are there another way to determine the same thing is by using this descriptive stats how basically you know that this number is there so if you remember in the last session we have talked about the shape and we have figured it out that the number of records that we have in this data is 768 so what you are analyzing here that in each and every column I have 768 number of Records so this count parameter will provide me the number of non null values so if let's say your data contains null values obviously for that particular feature the value will be less than 768 I hope it is clear so now because it doesn't contain any null values that's why you will observe 768 is common for each and every column or the feature that we have similarly you can evaluate the mean part standard deviation part minimum value that we have 25 percentile 50 percentile is nothing but the median 75 percentile which is Q3 and the maximum value so in this way we will be able to get the complete metric which we can analyze it very clearly for example let's say you want to determine that whether your data which is given to you is in a standard normal form or not what is the meaning of standard normal form the simple meaning is that the mean will be zero and standard deviation will be one right so any data any data if you will observe if it follows a gausian format or a bell-shaped curve it's in a normal form right it's in a normal form which is having some PDF probability density function I'm not going into that much depth but I'm just trying to Showcase you if there is anything you're not aware of please do your own research uh with respect to the statistical part right so this is something which we usually say is a normal normal form normal data or you can say it's a goian format right goian distribution or normal distribution it is it is very famous in uh statistical terms right now whenever we will be having this kind of a distribution or you can say it's also Famous by the normal distribution now let's say you want to determine that whether your data is in a normal form or not normal distribution form or not we will try to create a distribution plot and this is also called as a symmetric kind of a distribution symmetric kind of a distribution having some mean value and some standard division value now when I'm saying that is any data belongs to a standard normal distribution or not the simple thing which I want to indicate here is that in that case the mean will be equals to zero Sigma denotes a mean part mean is Mu which is zero and sigma is standard deviation which is equals to one that is the literal meaning of standard normal distribution we will discuss in future as well now any data is either symmetric or it can be non- symmetric as well for example here we will be having either right side inclination so right skewed distribution right skewed distribution because we will be having more data towards the right side or it can be vice versa left skew distribution like this like this this will be a LIF skewed distribution right so this statistical things will help us to determine the data imputation part now why is that so I will let you know so this is the overall idea so we will be able to get the descriptive stats now the next important concept is data imputation because although we understand ke missing values are not available but we are also aware that in this particular scenario zeros doesn't make sense and somehow we have to handle these zeros now the first way which we will discuss it in this particular session is that we can do the imputation Via any of the measures central tendency measures which can be mean median or mode now the question is how is that so and how will we determine that at what point of time which particular measure is the perfect one for this again the knowledge of this distribution plots is important either it will be a symmetric or it will be a non symmetric distribution non symmetric distribution when I'm saying non symmetric it will be either a left skewed or it will be a right skewed distribution that we have now the question is that at what point of time you will observe a non symmetric distribution will be there when there will be outliers available in your data what is the meaning of outliers I can give you a very simple example with the help of which you will be able to understand this point for example let's say we are doing one survey that in Bangalore in Bangalore the people who are working as a data scientist in Bangalore the people who are working as a data scientist I'm doing a survey that on an average how much salary they are earning and let's say I went to different different uh places and exploring the different different companies like product based service based and figure out some numbers so some people are earning maybe 50,000 per month some people are earning let's say uh 80,000 some people and I'm doing specifically for freshers let's say only for freshers I'm saying because the margin should be same we cannot compare the experienced people with the freshers definitely for sure so we are saying that some people are earning 50k some people are earning 80k some people are here earning 1 lakh as as well let's suppose as a fresher some people are let's say also earning maybe salary as two L and by the way I'm when I'm saying salary I'm saying inh hand salary after tax deduction let's say I'm assuming that part okay now I met a guy in Bangalore and maybe he's working in some tier one company product based company and he said that let's say I'm from tier one college maybe from IIT and I got a very good package and I'm as of now earning maybe let's say one CR I'm just taking some random number just to help you explain the concept please don't go in a serious manner I don't know whether this number exists or not with respect to frur part but by the way what I'm saying is ke this number is a outlier comparable to what other numbers we have right this is an outlier can I say so what is the meaning of outlier outlier simply indicates some exceptional value available within a given data that we have now when I will take the average or the mean of this outlier with the help of this outlier this mean shuffles a lot because now what you will do you will say what is the uh 50,000 plus you will say 880,000 plus you will say uh one lakh right one lakh plus you will say 2 lakh plus you will say one CR one CR divided by you will be having 1 2 3 4 five terms so this is how you usually calculate the average and you will see because of this outlier term available in your data your mean shifts a lot your mean shifts a lot it simply indicates that mean outlier when whenever outlier is there mean is not a great measure whereas if I will go for median in that scenario how we will evaluate a median first of all in doing the evaluation of a median if you remember we firstly sort the data in an ascending order from lower to higher part and after after that we will pick the middle value if let's say the number of values we have is old we will take the average of middle values if it is even we will take the sorry if it is old we will take the middle value if it is even we will take the average of the two numbers that we have in the middle so we will take the middle value now depending upon what or how many numbers we have in a given data now as it is a old number value number because the value of n here is five so you will be able to observe our median is one lakh simply so what you have observed from this explanation the simple thing which I want to indicate here is that median is mark my words it is usually asked in lot of interviews as well you will observe median is more robust to outliers so it simply indicates that whenever you are dealing with any numeric data at that point of time whenever you are doing data imputation so first thing when I'm saying let's say whenever you're dealing with a numeric data because we do have different kinds of data sets right specifically I'm talking about a numeric data and let's say you want to do a data imputation because you have figured out maybe null values are there or maybe the values are zeros anything anything can happen for example in our case zeros are there that's a Maj major issue now whenever you are doing the data imputation at that point of time you need to figure it out what is the distribution of your data now this distribution will help you to understand ke whether outliers are available in your data or not if let's say your distribution is symmetric in that case though it indicates there will be no outliers and if there are no outliers you can do the you can do the imputation Via mean it's a good Matrix but if let's say you are saying that the distribution that we got for a given feature for a given column that we have is non symmetric is nonn symmetric whether it may be left skewed or it may be right skewed means it do contains outliers the simple indication is outliers are there in that case never ever go for mean the always preferable choice will be the median so we will go for median and we will do the imputation via the median only so this is how the flow will go let's say you will be having a categorical data in that case maybe we can go for a imputation with a we can go for a data imputation with a mode part now how the mode will be evaluated basically it depends on the frequency so whatever be the value which is having a higher frequency with that value itself I will just try to do the imputation that's it this is the imputation via the mean median or mode but this is not the only way usually practically when we do projects there can be a way where people usually apply the algorithms machine learning algorithms only for example KNN is very widely used maybe we will discuss in the further sessions uh once we will be having an idea of how Canan algorithm works so what I'm saying is what usually people do this is approach number one which we are doing it today right in this particular project this is the approach number one which we are applying approach number one but always remember this is not the only approach that we have approach number could be which we will'll see in the upcoming project video where what we can do is we can apply a we can train a model now this model as of now is a blackbox for you but in future you will be able to understand for example there is one model name is Cann we are applying this kind of a model and this model will give me the predictions that which particular value should be the suitable value to do the data imputation right in that case we are not going for mean median mode we're just trying to train a model getting the predictions that what could be the optimal value in that place instead of zero but this approach we are not using as of now because we have covered up limited algorithms in our course part right so this is how the flow could be so now you will be able to understand that if we want to go for approach number one where we are doing a imputation via uh mean median or mode we have to understand that what kind of distribution my data holds and that is where our next discussion starts where what we will do is we will try try to plan and see the distribution of our data and it's very simple we will go for cbone again SNS do distplot is a function that we have disc means distribution so we are looking for distribution plot now in this complete Ed process uh see I'm am doing a recording of the videos so it is not feasible for me to do it each and everything from scratch and show you uh with respect to the code part and everything I can showcase the mindset to you and I can showcase few things with respect to how the things will work then I will give you a task that now do the similar thing in the entire data that we have this is how the flow will go because otherwise this recording will be of five hours 10 hours which is not something I want right so basically I will give you a idea intuition that this is how you can uh go ahead with respect to the further task the major thing is in this videos I want to explain the conceptual understanding plus the implementation part per se definitely after that I will give you a lead that okay now you you go ahead you do that do that part right so I will show you the distribution plot of few columns and we show you how the imputation will happen similar thing you have to do at your end as well okay so dis plot is the function to display the distribution plot now what you will do you will say data frame dot let's say whatever columns we have for example the very first column I think we have print a list of all the columns pregnancies right so let me just copy that and paste it over here so now if you will see the distribution plot of this particular data you can see that this is inclined almost towards I can say maybe we can say it will be uh right skewed or you can also say it is approaching towards the normal distribution the problem is because of these zeros which are here in outliers and if we can remove them or we can do the imputation to towards them obviously we are approaching towards a symmetric distribution only so you can either go with the respect to median part per se with respect to the uh imputation or you can go with the mean also but as of now because it is more showcasing me towards the right inclined so I will go for the medior so what we can do is we can go and do the imputation with respect to the median part same thing you have to check for all the columns that we have for example data frame dot let's say we will be having I think blood pressure also one of the column blood pressure so what we will do we will again check the particular value that we have again you can see zeros are there can you see in the graph itself you will be able to understand and once you will do the imputation of the zeros automatically this is properly a symmetric distribution which we are seeing here so definitely we can go here and do the imputation with respect to the this with respect to this BP per se we can go with the imputation of mean maybe with respect to the pregnancies per se you can go ahead and do the imputation with respect to the pregnancies per se we can go ahead and do the imputation with respect to the median similarly let's say I will go and check out the further columns that we have which is insulin so let's do it for that insulin so similarly what you have to do you have to check the distribution for the entire plots that you have now you can see it is right skewed it is perfectly visible to me so definitely in this particular feature also with respect to insulin we can go and do the uh we can say imputation with respect to the median part only right this is how the flow will go so with respect to every feature now you know that how basically we will be able to do the distribution plot right it's very simple we just have to use this dis plot function and we will be able to do that now the question is so this is how now your major task would be to check each and every column that we have for example glucose skin thickness BMI diabetes pedigree function and age outcome is a uh thing which we don't focus on because at any at after some stage we will just have to remove this target column also we will separate our input features and the target value so as of now just focus on this input features that we have until Age part right so next part is that how we can do a data imputation with respect to mean or median so for example we will be having insulin right so we will be having this data frame now what I will do is I will take this data frame of insulin and what I'm saying is replace so in Python we have this replace function so what we are saying is replace this zero with doing the calculation of data frame of insulin median value why because the only reason is this insulin is a insulin is a right skewed distribution so it contains a non symmetric data it means there exist in outliers and that is the main reason just now we have discussed mathematically that whenever the outliers are there mean is very dangerous metrix at that point of time we'll have to go for the median part only and now you will be able to see done if you will if you if you want to analyze data now you will see or you will observe there will be no zeros available with respect to the insulin you have just removed it right can you see in insulin we have some values available so there is no as such single value you will observe where I would say the zeros are there similar thing you have to do for other metrics as well now what you can do is you can just copy these lines I think we have eight features available with us uh 1 2 3 4 5 6 7 8 right so now maybe we can use blood pressure so let's see the columns again that we have data frame do columns and we will be able to have the list of entire columns so we can take it step by step pregnancies same code you have to use it's just either the mean value will be there or median value will be there depending upon uh whether the distribution plot is um you know symmetric or not so I'm just copying it up blood pressure then we have skin thickness skin thickness skin thickness then we have BMI so BMI BMI and BMI then we have diabetes prede function it would be great if you can just copy paste sometimes what happen is there will be a small spell spelling mistake which we usually done and because of that uh it shows an error So to avoid that it's a good way now what we can do because we already have Ren this file U maybe I can just separate this block with another block right so that it will not repeat again so the only thing is now it is not Medan for everyone right for example BP I think we have planned to go for mean so what you have to do it's just that in case of blood pressure just mention me similarly I think for pregnancies we have planned only uh the median only so you just have to refer median in this case similarly for glucose just uh take out some time and check the distribution plot for the pending part it's very easy right just have to mention glucose just write all the code blocks with respect to this part then we have BMI which is pending then we have I think uh glucose is done BP is done skin thickness is spending right so maybe you can just copy this part done and then I think diabetes pedigree function is pending so maybe we can just copy this part pedigree function and last thing is age which is pending so maybe just we can do this as well so let's copy this and refer the Age part this is also done so let's try to check out the distribution plots for every particular value that we have so again if you can see very clearly for glucose it's a kind of a normal distribution approaching towards normal so I will go ahead with the mean with respect to glucose so where is my glucose yeah this is a mean now let's check the BP is done skin thickness plot let's check that uh with respect to skin thickness again you will be able to see it's kind of if we can replace the zeros it's a normal distribution only so maybe you can go with either mean or median maybe let's go with the median only BMI BMI is something which let's go ahead it's a normal distribution so we can go ahead with the mean part here but yeah I'm doing it bit fast you can just check it out at your end maybe something if I have missed I have already told you the logic accordingly you have to now take care it's kind of a right skew so median is fine and age is also right sco so maybe median is fine so I think this is how the thing will look like but definitely at your end please check it very wisely you just have to understand if your plot is approaching towards normal distribution then replace it with the mean if it's a highly skewed one maybe left or right replace it bya median the reason I have already indicated you now you can run this T task is done now you will observe if let's say I will just show you data frame. head for the top 20 records and if I just run this you will be clearly able to understand now that that we are not having a single zero available in our data and that is what I want to I really want to achieve this is the first part of Eda where what we have done is we have done a bit of mathematics I would say specifically statistical part where I have shown you the descriptive stats how we will be able to calculate what's the significance of that I have shown you so basically if you will be able to observe we have talked about the descriptive statistics and its signif ific significance after that I have shown you correlation coefficient part and its significance very very important after that I have also talked about distributions types of distributions it will be skewed and non-sew distribution and again its significance then we have talked about that mean or I would say median is more robust to outliers and why right the reason I have stated so this thing until so far we have covered up so we have seen majorly with the help of this concept how we can do the imputation how we can do the data data imputation via mean and median what is the inference we have made is when the distribution is symmetric we can go for mean and when it is skewed whether it be a left or right we we can go for the median and let's say when the distribution is uh when the data is of categorical data in that case we can go for mode because it indicates the highest frequency so this thing will work only for the numeric data that we have analyzed so this is the overall idea about this session Ed is not finished yet I'll further discuss more concepts of Ed in the upcoming part of the session because I think I think the video is too long now so let's take it until this part take a note this is the complete wrapup of this video and I'll see you all in the upcoming video where we will do further things on the Eda part I hope that until so far everything is clear but still if you have any confusion any doubt you can let me know I'll see you all very soon in the upcoming video